Introduction

Can a local model build a maze bomber game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a maze bomber game like Bomberman as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 14 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is played on a grid of 15 x 13 cells with fixed walls and breakable bricks. The player walks through the maze and places bombs. A bomb explodes after 90 frames. Its fire forms a cross that reaches two cells in each direction, stops before a wall and destroys the first brick in its way. A bomb hit by fire explodes in the same frame. Three enemies walk through the maze, and fire removes them. The player has three lives and wins when no enemy is left. Destroyed bricks can leave power-ups for more bombs or longer fire.

The exact rules the hidden tests check include the start state, the movement delay, the bomb limit, the frame in which a bomb explodes, the shape of the fire and where it stops, the chain reaction, how long the fire lasts, the enemy steps every 12 frames, lives, power-ups, winning, the restart with Enter and the drawing of the scene. The task also says that the listed game variables are the whole game state: the tests read them and may set them. The model chooses how to build all of this.

Results

Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in Instruct mode passed every result, Qwen 3.6 27B in Thinking mode lost one, Qwen 3.6 35B in Instruct mode lost two, Qwen 3.6 35B in Thinking mode lost eight, and Qwen 3.8 27B in Instruct mode lost eleven. The chart below shows the score of each setup: the share of passed results, the build of the game and the 14 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a working game in both modes: all hidden tests passed in Instruct mode, and all but one in Thinking mode. In both modes, the model's own test found the same early mistake: a new bomb lost one frame in the frame it was placed (see below). The model fixed it at once.

In Instruct mode, the game was correct after a few edits. Then the model kept adding tests, 52 in the end, and tried the same edit of its own tests again and again. 23 repeated edits were refused. The run reached the limit of 50 model rounds before the model submitted the game, so the last version of the file was graded. It passed every hidden test.

In Thinking mode, the model fixed two compile errors and the bomb timing, then added the effects of part B and submitted. One hidden test failed: an enemy picks one random direction and stays when that cell is blocked (see below).

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B in Instruct mode came close with 13 of 15 results, while Thinking mode lost eight results. Both games lost the same two hidden tests: a bomb explodes one frame too early, because its timer runs already in the frame in which it is placed.

In Instruct mode, the model wrote the whole game at once and fixed one of its own tests. Then it answered three times without a tool call. In each of these answers it described the bomb timing bug correctly, but it printed the whole file as text instead of changing it. The run stopped after the third answer, and the last version of the file was graded. The game also lacks two effects of part B: the warning before a bomb explodes and the effect when the player loses a life. The hidden tests do not check part B, so the score does not show this.

In Thinking mode, the fire reached only one cell in each direction, because of a misplaced line (see below). This one line broke six hidden tests. The model also made 15 tool calls that were rejected.

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium, but only 3 of the 14 hidden tests in Instruct mode. In Think: medium, the model built the game from a short skeleton, fixed its own mistakes and submitted with 12 tool calls that did work, the fewest of the setups that passed every test (see "One run step by step").

In Instruct mode, three separate mistakes added up. The model kept a private list with the frame in which each bomb was placed. Bombs that the tests add have no entry in this list, so nine hidden tests stopped with an error. The fire reached only one cell, because of the same misplaced line as in Qwen 3.6 35B. And Enter started the new game one frame late. All 14 of the model's own tests passed: they set fire directly on a cell and never checked an explosion.

Qwen 3.6 35B in Instruct mode scored far more than Qwen 3.8 27B in Instruct mode in this test. Both models made mistakes in the bomb rules, but the mistakes of Qwen 3.8 27B broke many more hidden tests. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.6 27B in Instruct mode wrote 52 tests, and three of them still failed when the run ended. Qwen 3.8 27B in Instruct mode wrote 14 tests, all passing, and still had the lowest score. Qwen 3.6 35B wrote only three or four tests in each mode, and none of them placed a bomb.

Checks and edits

Qwen 3.6 27B in Instruct mode made the most checks and edits, most of them on its own tests. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.8 27B in Instruct mode made 13 edits and still submitted a game with three separate mistakes. Qwen 3.6 35B in Instruct mode made one check and one edit before it stopped calling tools.

Unwanted tool calls

Qwen 3.6 27B in Instruct mode spent 24 tool calls on nothing, and Qwen 3.6 35B in Thinking mode 15. In Instruct mode, Qwen 3.6 27B repeated an earlier edit 23 times after the file had changed, so these edits were refused; one more call had broken call syntax. Qwen 3.6 35B in Thinking mode had one more refused edit, and 14 of its calls referred to text that the answer did not contain. Qwen 3.8 27B had two wrong calls in Instruct mode and three in Think: medium. Qwen 3.6 27B in Thinking mode and Qwen 3.6 35B in Instruct mode made no wrong calls, but Qwen 3.6 35B in Instruct mode stopped calling tools at all.

Tokens

Every tool call sends the whole conversation to the model again, so long runs read far more tokens. Qwen 3.6 27B in Instruct mode read over three million tokens: it used all 50 rounds, and its file grew to more than 1,200 lines with its tests. Qwen 3.8 27B in Think: medium wrote the most tokens, because it reasons before each step. Qwen 3.6 35B in Instruct mode read the fewest tokens, because it stopped early.

One run step by step

Qwen 3.8 27B in Think: medium built the game from a short skeleton and found its mistakes before the hidden tests did.

1. It started with a skeleton of 128 lines: the variables, the board, empty drawing functions and two tests.
2. Its next three calls were rejected: one check had broken call syntax, and two edits referred to text that the answer did not contain. The fourth call added the whole game logic in one edit.
3. It noticed that its rule for bricks made only about 12 bricks, while the task asks for 30 to 90. It changed the rule.
4. It reset a counter that a new game did not reset.
5. It added five tests: an explosion, an enemy in the fire, a power-up, the pause and the restart. The check found a compile error in the test messages, and it fixed it.
6. It changed the colors of its own effects once more, and all 7 of its tests passed.
7. It submitted the game, and all 14 hidden tests passed.

How the models fail

A bomb placed in this frame

Four of the six runs first counted down a new bomb in the frame in which it was placed. The task says that every frame first handles the Space key and then decreases the timers of the bombs that were placed in an earlier frame. So a new bomb must keep its timer of 90 in its first frame and explode in the 90th frame after it is placed. Qwen 3.6 27B found this mistake with its own test in both modes. Qwen 3.6 35B kept it in both modes and lost two hidden tests.

Qwen 3.8 27B in Instruct mode noticed the rule but solved it in a way that breaks the tests. It added a private list with the frame of each bomb. The setups that passed only remembered how many bombs there were before the Space key was handled:

The private list broke nine hidden tests in Qwen 3.8 27B in Instruct mode. The task had warned about this: the listed variables are the whole game state, and the tests may set them.

Fire that stops after one cell

Qwen 3.6 35B in Thinking mode and Qwen 3.8 27B in Instruct mode wrote the same bug: one line that ends the fire after the first cell. The fire must go up to bombRange cells, 2 at the start, in each direction. After each cell, the code checks whether another bomb is there. Both models put a break after this check, outside of it:

In Qwen 3.6 35B in Thinking mode, this line broke six hidden tests: the fire cross, walls and bricks, the chain reaction, the fire duration, the enemy in the fire and the longer fire from a power-up. In Qwen 3.8 27B in Instruct mode, the same tests failed, most of them already earlier with the list error. Neither model had a test that placed a bomb and checked the whole cross.

Smaller mistakes

• The restart comes one frame late. In Qwen 3.8 27B in Instruct mode, Enter only marks that a new game should start, and the new game starts in the next frame. The task says that Enter starts a new game in its frame. The model's own restart test first failed, and the model changed the test to wait two frames instead of changing the game.
• An enemy that picks a blocked way stays. In Qwen 3.6 27B in Thinking mode, each enemy picks one random direction in a step and stays if that cell is not free. The task says that every enemy with a free neighbor cell moves to one of them. The hidden test failed in the first enemy step.

Conclusion

Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in Instruct mode passed every result, and Qwen 3.6 27B in Thinking mode was one test behind.

• Qwen 3.6 27B: all hidden tests passed in Instruct mode, but the run reached the round limit before the model submitted the game. In Thinking mode, one enemy rule failed.
• Qwen 3.8 27B: all hidden tests passed in Think: medium with few tool calls. In Instruct mode, a private list, one misplaced line and a late restart broke eleven results.
• Qwen 3.6 35B: 13 of 15 results in Instruct mode, but the model stopped calling tools and left out two effects of part B. In Thinking mode, the same misplaced line as in Qwen 3.8 27B broke the fire.

For building a game like this as an agent, the thinking mode of Qwen 3.8 27B or Qwen 3.6 27B in NVFP4 gave the most reliable results in this test. In any mode, check that the model's tests run the real game actions, like an explosion, and not only set the state by hand.

References

Benchmark runs with NVFP4 quantization