Introduction

Can a local model build a bubble shooter game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a bubble shooter like Puzzle Bobble as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 17 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a bubble shooter on a hexagonal grid. The player turns a launcher with the arrow keys and shoots a colored bubble. The bubble flies, bounces off the side walls and sticks to the ceiling or to other bubbles. A group of three or more bubbles of the same color pops, and bubbles that no longer hang from the ceiling fall. The ceiling moves down after every eighth shot, and the game is over when a bubble reaches the bottom line.

The exact rules the hidden tests check include the grid of 13 rows with every odd row shifted by half a bubble, the six neighbors of a cell on even and odd rows, the speed of a shot, the distance at which a flying bubble stops, the cell it snaps to (with a rule for ties), the scores and two levels given as maps of characters. The model chooses how to build all of this.

Results

Qwen 3.6 27B in Thinking mode and Qwen 3.8 27B in Think: medium passed every result, Qwen 3.6 27B in Instruct mode lost one, Qwen 3.8 27B in Instruct mode lost eleven, and Qwen 3.6 35B built no working game. The chart below shows the score of each setup: the share of passed results, the build of the game and the 17 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a working game in both modes: all hidden tests passed in Thinking mode, and all but one in Instruct mode. In Instruct mode, the model copied one row of the second level with one empty cell too few (see below). Everything else worked, including the hexagonal neighbors, the wall bounces and the falling bubbles.

In Instruct mode, the model wrote the whole game at once, then fixed it in a few small edits. It wrote 24 tests, more than any other setup, but none of them checked the second level. In Thinking mode, it built the game in steps, from a short skeleton to the full rules, and found and fixed its own mistakes on the way (see "One run step by step").

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not build a working game in either mode. In Instruct mode, it broke its own file with an edit and could not fix it: the game did not compile. In Thinking mode, its first answer got stuck in a loop while it was writing the grid, and the runtime stopped the answer before a game file existed. Both problems are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium, but only 6 of the 17 hidden tests in Instruct mode. In Think: medium, the model wrote most of the game in one go, then fixed one compile error, a few logic details and its own tests. It needed the fewest edits of the setups that passed every test.

In Instruct mode, almost the whole game was correct: the geometry, the bounces, the stop distance, the snapping and the falling bubbles. But one missing rule broke nine hidden tests: the game removes every new bubble, also a single one (see below). Two more tests failed because the frame with the restart key also moved the launcher.

Qwen 3.6 27B and Qwen 3.8 27B both passed every hidden test in thinking mode. In Instruct mode, Qwen 3.6 27B came much closer. Both models ran in NVFP4 with the same settings. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.6 27B in Instruct mode wrote the most tests and still missed a wrong row of the level map. Qwen 3.8 27B in Instruct mode wrote 10 tests, all passing, but none of them checked that bubbles pop only in groups of three. Qwen 3.6 35B is not in the chart: neither of its runs produced a game that compiles.

Checks and edits

Qwen 3.8 27B in Instruct mode needed the most checks and edits and still had the lowest score of the setups that built a game. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.6 27B in Thinking mode made many edits because it built the game in steps. Qwen 3.6 35B made almost no edits: in Instruct mode it stopped after one broken edit, and in Thinking mode it never created a file.

Unwanted tool calls

Qwen 3.8 27B in Instruct mode spent 14 tool calls on nothing. Eight of them repeated an earlier edit after the file had changed, so they were refused. The others asked for a tool that does not exist, passed an unknown parameter or referred to text that the answer did not contain. Qwen 3.6 27B in Instruct mode had two refused edits, Qwen 3.8 27B in Think: medium one wrong call, and Qwen 3.6 27B in Thinking mode none. Qwen 3.6 35B in Instruct mode made no wrong calls, but it often answered without any tool call and needed five reminders.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls read far more tokens. Qwen 3.8 27B in Instruct mode read over two million tokens, most of them because of its many checks and edits. Thinking mode writes more tokens because the model reasons before each step. Qwen 3.6 35B in Instruct mode also wrote many tokens: most of its answers repeated the whole file as text instead of changing it with a tool.

One run step by step

Qwen 3.6 27B in Thinking mode built the game in small steps and found most of its mistakes itself.

1. It started with a skeleton: the variables, the two levels and two tests. One of its own tests expected a wrong position, and it fixed the test.
2. It added the flight of the shot, the snapping to a cell and the end of a shot.
3. The check found two compile errors: two variables declared in one line, which SC# does not allow, and a list method that SC# does not have. It fixed both.
4. Its tests showed that the shot did not move. The model found the reason itself: the code that moves the shot was never called from the game loop.
5. It changed the color of an effect, because the task reserves that color for the aiming line.
6. It added more tests, fixed a missing brace and then an extra brace, and all 18 of its tests passed.
7. It submitted the game, and all 17 hidden tests passed.

How the models fail

Every new bubble disappears

The game of Qwen 3.8 27B in Instruct mode removes every bubble it shoots, also a single one. The task says that only a group of three or more bubbles of the same color is removed. The model's code collects the group of the new bubble and removes it without checking its size:

One missing condition broke nine hidden tests. In most of them, the field was empty after the shot, because the new bubble had vanished. In the test for groups of two, the score was 20 instead of 0. The model's ten tests all passed: none of them shot a bubble next to fewer than two bubbles of its color.

The restart frame does more

In the same game, the frame with the restart key also turned the launcher. The task says that after the game ends, only Enter does something, and nothing else happens in that frame. The model's game loop restarts the game first and then aims the launcher in the same frame:

HandleMenuKeys();   // Enter starts a new game and clears gameOver
AimShot();          // checks gameOver again, which is now false
FireOrMoveShot();

The aiming code checks whether the game is over, but the flag is already cleared. A held arrow key therefore turns the launcher in the restart frame. Two hidden tests failed: after game over and after a cleared level.

A row with one empty cell too few

Qwen 3.6 27B in Instruct mode copied the last row of the second level with four dots instead of five. In the map, a dot is an empty cell, and an odd row has seven cells:

Only the hidden test that compares the second level failed. The model had made a similar slip in the first level, but its own tests found that one, and it fixed it. None of its tests checked the second level.

A broken file and a loop

Qwen 3.6 35B lost both runs before a working game existed.

• Instruct mode: the model answered twice with code as text before it created the file. Then it replaced a block of the game loop, but its new code already ended with the closing brace of the function, and the old closing brace stayed one line below. The file no longer compiled. The model ran the check three more times without a change, and then it answered with the file as text instead of calling a tool. In its last answer, it wrote that line 538 had an extra brace, but it never deleted it.
• Thinking mode: in its first answer, the model started to write the grid, a list of 104 cells, and repeated "-1, " until the runtime stopped the answer as broken output. In this test series, a broken answer ended the work, so no game file existed.

Conclusion

Qwen 3.6 27B in Thinking mode and Qwen 3.8 27B in Think: medium passed every result, and Qwen 3.6 27B in Instruct mode was close behind.

• Qwen 3.6 27B: all hidden tests passed in Thinking mode, and all but one in Instruct mode, which copied one row of a level map with a mistake.
• Qwen 3.8 27B: all hidden tests passed in Think: medium. In Instruct mode, a nearly correct game lost most hidden tests because of one missing rule.
• Qwen 3.6 35B: no working game in either mode, because of a broken edit and a loop.

For building a game like this as an agent, the thinking mode of Qwen 3.6 27B or Qwen 3.8 27B in NVFP4 is the safer choice. In Instruct mode, check that the rules with limits and the level data are tested.

References

Benchmark runs with NVFP4 quantization