Introduction

Can a local model build a digger puzzle game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a digger puzzle game like Supaplex as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 14 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a puzzle on a grid of 20 x 12 cells. The player moves through the level with the arrow keys. He digs through base, collects infotrons and pushes stones to the side. Stones and infotrons fall when the cell below them is empty and roll off other round objects. A falling stone kills the player, two enemies walk around the level, and the exit opens when all eight infotrons are collected.

The exact rules the hidden tests check include the start level, given as a map of characters, the move delay of the player, the order and timing of the falling and rolling objects, when a falling object kills the player or an enemy, the moves of the enemies, the restart after the game ends and the colors of the screen. The task also names the variables that hold the game state, because the tests read and set them. The model chooses how to build all of this.

Results

Qwen 3.8 27B in Think: medium passed every result. Qwen 3.8 27B in Instruct mode and Qwen 3.6 27B in Thinking mode lost three results each, all because of the start level. Qwen 3.6 27B in Instruct mode lost eight, and Qwen 3.6 35B built no game that compiles. The chart below shows the score of each setup: the share of passed results, the build of the game and the 14 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a playable game in both modes, but in both modes the start level was wrong. The task gives the level as a map of characters. In both modes, the model typed the map as numbers and turned every base cell into an empty cell (see below). In Thinking mode, this was the only problem: the movement, the falling and rolling objects, the enemies and the restart all passed.

In Instruct mode, two more rules broke five tests. The game kept the stones and infotrons in its own lists and not in the level, so stones that a test placed in the level never fell. And the player could not step onto the cell of an enemy, so the game did not end when he did. The model wrote six tests in Instruct mode and fourteen in Thinking mode. None of them checked the base cells of the level.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not build a game that compiles in either mode. In both modes, the first file used C# features that SC# does not have. In Instruct mode, the model fixed two of them, then explained the third in several answers but did not change the code. In Thinking mode, it never ran a check, and its fourth answer got stuck in a loop. Both runs are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium and lost three in Instruct mode, all because of the start level. In Think: medium, the model copied the map rows from the task as text and converted them with code. Every cell came out right. It then fixed two small compile errors, added the rules in a few steps and wrote 13 tests.

In Instruct mode, the model also typed the map as numbers. It lost most base cells, moved several stones and infotrons by one cell and added one hardware cell. Everything else in the game was correct. Its own test of the start state checked the infotron in the wrong cell, because it copied the same mistake from the code.

The rules of this game were not the hard part: the start level was. Three of the four 27B setups built correct movement and physics, but two of them lost three results each on the map alone. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.6 27B in Thinking mode wrote the most tests and still lost three hidden tests on the start level. Qwen 3.8 27B in Think: medium wrote one test fewer and passed everything. Qwen 3.6 35B is not in the chart: neither of its runs produced a game that compiles.

Checks and edits

Qwen 3.6 27B made the most checks and edits in both modes. A check compiles the game and runs the model's tests. An edit changes some lines of the file. In Thinking mode, Qwen 3.6 27B built the game in many small steps and fixed several missing and extra braces on the way. Qwen 3.6 35B made few edits: in Instruct mode it stopped calling tools, and in Thinking mode it never ran a check.

Unwanted tool calls

Qwen 3.8 27B in Instruct mode made the most unwanted tool calls. Two of them repeated an earlier edit, so they were refused. Two more referred to text that the answer did not contain, and one edit was rejected. Qwen 3.6 27B in Instruct mode had four refused repeated edits, and Qwen 3.6 35B in Instruct mode one rejected call. The thinking setups made no wrong calls. Qwen 3.6 35B in Instruct mode had another problem: it often answered without any tool call and needed eleven reminders.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls read far more tokens. Qwen 3.6 27B in Instruct mode read the most, about 1.4 million tokens, because of its many checks and edits. Qwen 3.6 35B in Instruct mode also read over a million tokens: it answered many times without a tool call, and every answer and reminder made the conversation longer. Thinking mode writes more tokens because the model reasons before each step. Qwen 3.6 35B wrote many tokens in both modes, in Thinking mode mostly in the answer that looped.

One run step by step

Qwen 3.8 27B in Think: medium started with a small game that compiled and added the rules in a few steps.

1. It started with a skeleton of about 200 lines: the variables from the task, the start level copied as text rows, the drawing and two tests.
2. The check found two compile errors: a text joined with a number by +, which SC# does not allow. The first fix swapped the two arguments of an assert, and the next check found that. The model fixed it, and both tests passed.
3. It added the movement of the player, the falling and rolling objects, the enemies and the collisions as helper functions.
4. Before the next check, it replaced fixed-size arrays, which SC# does not have, with lists and called the new functions from the game loop.
5. It added six tests for the rules, then five more for winning, the closed exit and the end of the game. All 13 tests passed.
6. It submitted the game, and all 14 hidden tests passed.

How the models fail

Base cells typed as empty cells

Three setups typed the start level as numbers and lost the base cells. In the task map, a dot is base, the soil that the player digs through, and a space is an empty cell. Qwen 3.6 27B in both modes wrote 0 for every dot, so the level had no base at all. Qwen 3.8 27B in Instruct mode kept only 25 of the 138 base cells. Qwen 3.8 27B in Think: medium copied the rows as text and converted each character with code:

The wrong map broke three hidden tests in each of these runs: the start state, the screen, which drew no base or one hardware cell too many, and the restart, which checks the start level again. Typing the map by hand also moved some objects. Qwen 3.8 27B in Instruct mode put five infotrons and three stones one cell to the side. Its own test of the start state checked one of these infotrons in the wrong cell, because the test copied the mistake from the code.

Qwen 3.6 27B was close to finding the mistake in both modes. Its test for digging needed a base cell next to the start position, where the task map has one. The model saw that the cell was empty in its level, and placed a base cell in the test by hand instead of fixing the map.

Stones outside the level

The game of Qwen 3.6 27B in Instruct mode did not keep its stones in the level. The task says that the level variable holds the stones and infotrons, and that the tests read and set it. The model moved all round objects into separate lists when the game started and set their cells in the level to empty:

objType.Add(v);
objX.Add(x);
objY.Add(y);
level[y * GRID_W + x] = 0;

The physics then moved only the objects in these lists. A stone that a test placed in the level never fell, never rolled and never killed the player. Four hidden tests failed because of this, and a fifth one, the restart test, could not reach the end of the game.

The player stops before an enemy

In the same game, the player could not step onto the cell of an enemy. The task says that the player moves onto an empty cell, a base cell or the cell of an enemy, and that the game ends when the player and an enemy share a cell. The model's code moves the player onto an empty cell only when no enemy is there, and has no rule for the enemy's cell. The player stays where he is, the game goes on, and the hidden test for this case failed.

C# features and a loop

Qwen 3.6 35B lost both runs before a game that compiles existed.

• Instruct mode: the model answered twice with a plan and code as text before it created the file. The file used three C# features that SC# does not have: two variables declared in one line, fixed-size arrays and a number with the suffix f. The model fixed the first two after its checks. For the third one, it wrote in several answers that SC# has no 5f suffix, but it did not change the code. After three answers in a row without a tool call, the run ended.
• Thinking mode: the model wrote a file of 558 lines and changed it twice, but it never ran a check. The file did not compile, because two variables were declared in one line. In its fourth answer, it repeated "OK, WRITING THE RESPONSE. FINAL ANSWER." and similar sentences for more than 200,000 characters, until the runtime stopped the answer as broken output. In this test series, a broken answer ended the work.

Conclusion

Qwen 3.8 27B in Think: medium passed every result. Qwen 3.8 27B in Instruct mode and Qwen 3.6 27B in Thinking mode built a correct game on a wrong start level.

• Qwen 3.8 27B: all hidden tests passed in Think: medium. In Instruct mode, the game was correct except for the start level, which lost three results.
• Qwen 3.6 27B: in Thinking mode, the same three results were lost on the start level. In Instruct mode, the stones outside the level and the blocked enemy cell cost five more.
• Qwen 3.6 35B: no game that compiles in either mode, because of C# features, missing tool calls and a loop.

For building a game like this as an agent, the thinking mode of Qwen 3.8 27B in NVFP4 did best in this test. With every model, check that data given in the task, such as a level map, ends up in the code exactly.

References

Benchmark runs with NVFP4 quantization