Introduction

Can a local model build a box pushing puzzle on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a box pushing puzzle like Sokoban as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 14 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The worker pushes boxes onto goals in three small levels. The arrow keys move the worker one cell. A box moves when the worker pushes it, but only into a free cell. U undoes the last move, R starts the level again, and Enter goes to the next level after a level is complete. The game counts moves and pushes and keeps the best result of each level for the session.

The exact rules the hidden tests check include the three level maps, the order of the boxes in the lists, walls and blocked pushes, one move per key press, undo move by move, restart, the states "level complete" and "all levels done", the best result with its tie rule and the drawing of the screen. The model chooses how to build all of this.

Results

Qwen 3.8 27B in Think: medium passed every result. Four other setups lost exactly one hidden test, and Qwen 3.6 35B in Instruct mode built no game. The chart below shows the score of each setup: the share of passed results, the build of the game and the 14 hidden tests.

All four lost tests are about the moments between levels: what happens after a level is complete, after the last level and when a new game starts. Moving and pushing worked in every game that was built.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B lost the same hidden test in both modes, for the same reason. After the last level, both games moved the level number one step too far, to a level that does not exist (see below). In Instruct mode, the drawing code then read past the end of a list, and the test stopped with an error. In Thinking mode, the model's own test found this error, but the model fixed only the drawing, not the level number.

The two modes worked very differently. In Thinking mode, the model built the game in steps and submitted it after 19 tool calls. In Instruct mode, it wrote 492 lines at once and had a working game with passing tests after ten calls. Then it got stuck: it repeated the same answer and the same edit again and again until it reached the limit of 50 rounds, and it never submitted the game. The last version of the file was graded.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B built a nearly complete game in Thinking mode, but no game in Instruct mode. In Thinking mode, the game lost one hidden test: undo still worked after a level was complete, although the task says that U does nothing then. The model even wrote its own test for this wrong behaviour. The way to this game was long: seven times the model answered without a tool call and needed a reminder.

In Instruct mode, the model wrote a first version of the game into its answers three times, but it never called the tool that creates the file. The run stopped after three answers in a row without a tool call.

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium, with the fewest tool calls and the fewest tokens of all setups that built a game. It needed only eight tool calls: a skeleton, one rejected call, three edits, two checks and the submission (see "One run step by step").

In Instruct mode, the game was correct except for one rule: when a new game started after the last level, the game also cleared the best results of the session, which the task says must stay (see below). The model's nine tests did not cover the end of the game.

Qwen 3.8 27B in Think: medium was the only setup that handled every transition between levels correctly. Qwen 3.6 27B lost one test in both modes. Both models ran in NVFP4 with the same settings. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The best game came with a test of the full end of the game. Qwen 3.8 27B in Think: medium wrote 11 tests, and one of them played through the last level and checked that a new game keeps the best results. Qwen 3.6 27B in Thinking mode wrote the most tests, also one for the end of the game, but that test did not check the level number. Qwen 3.6 35B is shown only for Thinking mode: in Instruct mode, it built no game.

Checks and edits

Qwen 3.8 27B in Think: medium needed two checks and three edits. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.6 35B in Thinking mode made 17 edits to a short, dense file of about 140 lines: many of them repaired the structure of the file. Qwen 3.8 27B in Instruct mode needed 12 checks, and five of them found compile errors.

Unwanted tool calls

Qwen 3.6 27B in Instruct mode spent 26 tool calls on nothing. Twenty-five of them repeated an edit that was already applied, so they were refused, and one referred to text that the answer did not contain. Qwen 3.8 27B in Instruct mode had three calls that referred to text that the answer did not contain. Qwen 3.6 35B in Thinking mode had one call with broken syntax and one refused attempt to rewrite the whole file. Qwen 3.8 27B in Think: medium had one call with broken syntax, and Qwen 3.6 27B in Thinking mode one refused repeated edit. Qwen 3.6 35B in Instruct mode made no tool calls at all.

Tokens

Every tool call sends the whole conversation to the model again, so long runs read far more tokens. Qwen 3.6 27B in Instruct mode read almost 2.7 million tokens, mostly in the loop at the end of its run. Qwen 3.6 35B in Thinking mode read over 1.6 million tokens and wrote the most, about 144,000 tokens, because it reasoned at length before its many small steps. Qwen 3.8 27B in Think: medium read the fewest tokens of the setups that built a game.

One run step by step

Qwen 3.8 27B in Think: medium built the game in three steps and found two of its own mistakes before the first check.

1. It started with a skeleton of 150 lines: the variables, the three level maps, the start of a game and of a level, the drawing and two tests.
2. Its next call had broken syntax and was rejected. The model sent the step again.
3. It added the main part: the undo history, the checks for walls and boxes, moving and pushing, the completed level and the full game loop. The file grew to 366 lines.
4. Before it ran any check, it read its new code again and found two problems itself: the new code used other names for the undo lists than the variables it had declared, and a new game would not reset the best results correctly. It fixed both in one edit.
5. The first check compiled the game, and both tests passed.
6. It added nine tests: walls, pushing, a blocked push, undo, restart, the best result, Enter after a level, the whole end of the game and the colors. All 11 tests passed.
7. It submitted the game, and all 14 hidden tests passed.

How the models fail

A new game forgets the best results

The game of Qwen 3.8 27B in Instruct mode used one function for the first start and for every new game. The task says that a new game after the last level keeps the best results, because they belong to the session. Only the very first start sets them to -1. The model's function always cleared them:

The same model wrote both versions, in different modes. In Think: medium, it also wrote a test that plays to the end of the game and checks the best results after a new game. In Instruct mode, none of its nine tests reached the end of the game.

One level too far

Both games of Qwen 3.6 27B moved to a fourth level that does not exist. The task says that Enter starts the next level, and after the last level it sets "all levels done" instead. So after the third level, the level number stays at the third level. Both games first added one to the level number and only then checked whether that level exists:

In Instruct mode, the drawing code then read the best result of level 4 from a list of three values, and the hidden test stopped with an error. In Thinking mode, the model's own test for the end of the game stopped with the same error. The model saw the message and added a range check to the drawing code. The test passed, but the level number was still wrong, and the hidden test found it.

Undo after the level is complete

The game of Qwen 3.6 35B in Thinking mode allowed undo after a level was complete. The task says that while a level is complete, only Enter and R do something, and the arrows and U do nothing. The model's code handles U in that state and takes back the last move, so the completed level becomes incomplete again:

if (levelComplete) {
    if (Input.WasPressed("Enter")) { ... }
    else if (Input.WasPressed("R")) LoadLevel(levelIndex);
    else if (Input.WasPressed("U")) { ... /* takes back the last move */ }
}

The model also wrote its own test that checks exactly this: complete level 1, press U and expect the level to be incomplete again. The test passed, so the check did not help. A test only helps when it checks the rule from the task.

A working game that was never handed in

Qwen 3.6 27B in Instruct mode had a working game after ten tool calls, but it never submitted it. All nine of its tests passed. Then it started to add the completion effect and changed the game loop several times. After that, it wrote the same answer again and again: "I see that all 9 tests pass. Let me review the current state of the app", followed by the same edit, which was refused each time. The run ended at the limit of 50 rounds. The last version of the file was graded and lost only the test for the end of the game.

Qwen 3.6 35B in Instruct mode never created the file. Its first answer contained a first version of the game as text. After a reminder, the second and third answers did the same. The run stopped after three answers in a row without a tool call, so there was no game to test.

Conclusion

Qwen 3.8 27B in Think: medium passed every result. Qwen 3.6 27B, Qwen 3.6 35B in Thinking mode and Qwen 3.8 27B in Instruct mode each lost one hidden test.

• Qwen 3.6 27B: one hidden test lost in each mode, both times because the level number went past the last level. In Instruct mode, the run also ended in a loop without a submission.
• Qwen 3.6 35B: one hidden test lost in Thinking mode, because undo still worked after a completed level. No game in Instruct mode.
• Qwen 3.8 27B: all hidden tests passed in Think: medium, with the fewest tool calls. In Instruct mode, a new game cleared the best results.

For a puzzle game like this, all three models in NVFP4 can build the moves and pushes correctly. The transitions between levels need their own tests: the last level, a new game and the keys that must do nothing.

References

Benchmark runs with NVFP4 quantization