Introduction

Can a local model build a fast clicking game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a Diamond Dash-like game as an app in SC#, our C#-like scripting language, played with the mouse. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 14 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a board of 8 x 9 diamonds in 5 colors, and a round lasts 60 seconds. A click on a group of at least 3 diamonds of the same color, connected left, right, up or down, removes the group. The diamonds above fall down and new ones come from the top. Larger groups give more points, and quick hits in a row raise a combo that multiplies the score. A click on a smaller group is a miss and blocks clicks for one second.

The exact rules the hidden tests check include the start of a round, which cells form a group, the order in which diamonds fall and new ones fill the columns, the score for each size and the combo, the miss and its pause, clicks outside the board and exactly at its edges, the clock, a new game with Enter, and new colors when the prepared list is empty. On the screen, the tests check that every diamond's color is shown in its cell, that the mouse over a group of 3 or more highlights exactly that group, and that the effects of a hit or a miss end within a second. The model chooses how to build all of this.

Results

Qwen 3.8 27B in Think: medium passed every result, and Qwen 3.6 35B in Instruct mode handed in a game without clicks. The other four setups built a playable game and lost one to four hidden tests. The chart below shows the score of each setup: the share of passed results, the build of the game and the 14 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B lost two hidden tests in Instruct mode and three in Thinking mode. In both modes, a click just left of the board counted as a click on the first column. In Instruct mode, the game also took a click in the last frame of the pause after a miss. In Thinking mode, the new diamonds filled each column from the top instead of from its lowest empty cell, which broke both tests of falling and refilling (all three are described below).

The two modes worked differently. In Instruct mode, the model stopped four times before it handed in the game and had to be reminded each time. In Thinking mode, it needed only 14 answers and three checks. In FP8, the same model passed every hidden test in Thinking mode, but only 5 of 14 in Instruct mode, so in total FP8 passed fewer hidden tests here than NVFP4.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B passed 10 hidden tests in Thinking mode, but only 2 in Instruct mode. In Instruct mode, the model wrote a first version with an empty click handler. Then it answered three times in a row with a plan, but without a tool call, so the run stopped (see below). The graded game showed a board, but a click did nothing.

In Thinking mode, the game worked. It filled the columns from the top, like Qwen 3.6 27B in Thinking mode. The highlight under the mouse missed some cells of the group. And when the time ran out, the end screen showed the final score, but not the clock.

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium and all but one in Instruct mode. In Think: medium, the model wrote 17 tests, far more than any other setup, and changed more than 1,000 lines on the way. Its game checks the board area first and only then computes the cell, so the edge test passed. In Instruct mode, the model wrote a compact game in 13 answers. Its only mistake was the click just left of the board.

In this game, thinking mode helped the 35B model the most. Qwen 3.6 35B gained eight hidden tests in Thinking mode. Qwen 3.8 27B gained one, and Qwen 3.6 27B lost one. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

Qwen 3.8 27B in Think: medium wrote 17 tests and was the only setup without a mistake. Its tests checked the pause after a miss, the combo, the order of falling and refilling and the highlight one by one. The other setups wrote two to six tests. Qwen 3.8 27B in Instruct mode wrote only two tests and still lost just one hidden test.

Checks and edits

Qwen 3.6 27B in Instruct mode ran the most checks and, together with Qwen 3.6 35B in Thinking mode, made the most edits. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.6 35B in Instruct mode made one edit and one check before it stopped. The other setups needed three to five checks.

Unwanted tool calls

Qwen 3.8 27B in Instruct mode had three rejected calls, all before its first file. Each referred to text that the answer did not contain, and the fourth attempt created the file. Qwen 3.6 35B in Instruct mode had two rejected calls, one of them an attempt to rewrite the whole file, and then three answers without any tool call. Both modes of Qwen 3.6 27B made no wrong calls.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many answers read more tokens. Qwen 3.6 27B in Instruct mode read the most, because of its many answers and reminders. Qwen 3.6 35B in Instruct mode read the fewest, because it stopped early. Thinking setups write more tokens, because the model reasons before each step. Qwen 3.8 27B in Instruct mode wrote only about 8,000 tokens for a full game.

How the models fail

A click left of the board selects the first column

Three setups counted a click just left of the board as a click on the first column. The board starts at x = 40, so a click at x = 39 must do nothing. Qwen 3.6 27B in both modes and Qwen 3.8 27B in Instruct mode first computed the cell and then checked whether it is on the board:

The same rounding hides the mistake in normal play. Only the 39 pixels left of the board and above it are affected, and a player rarely clicks there. A hidden test that clicks exactly at the edges finds it at once.

The new diamonds come in from the top

Two Thinking setups filled the empty cells of a column from row 0 down. The task says that each column is filled from its lowest empty cell up to row 0, and each new diamond takes the next color from a prepared list. With two empty cells, the first color must go to the lower cell. Qwen 3.6 27B and Qwen 3.6 35B in Thinking mode put it into the top cell:

The falling itself was correct in both games. On the screen the mistake is invisible, because the new colors are random in normal play. But the order decides which colors a test or a saved game gets, and both games failed the two hidden tests that check the board after a hit.

The pause after a miss ends one frame early

Qwen 3.6 27B in Instruct mode took a click in the last frame of the pause. After a miss, clicks are ignored for 30 frames: in each of them, the counter goes down by 1 and a click in that frame is ignored. The model first lowered the counter and then accepted a click when it was 0:

if (lockFrames > 0) { lockFrames--; }

if (lockFrames == 0 && Input.WasPressed("MouseLeft"))
{
    HandleClick();
}

In the frame in which the counter goes from 1 to 0, the click is accepted, so the pause lasts only 29 frames. Qwen 3.8 27B in Instruct mode wrote the same step as an if … else: in a frame that lowers the counter, it does not look at the mouse at all.

Qwen 3.6 35B in Instruct mode stops calling tools

Qwen 3.6 35B in Instruct mode handed in a game whose click handler was empty. Its first file compiled and its three tests passed, but the function for a click had no code. The model then wrote three answers in a row with the same plan, "Replace ProcessClick with full game logic", without a single tool call. After three answers without a tool call, the run stops, and the last version of the file is graded. The same model in Thinking mode built a working game.

Conclusion

Five of six setups built a playable game, and only Qwen 3.8 27B in Think: medium passed every hidden test.

• Qwen 3.6 27B: a working game in both modes, with two lost tests in Instruct mode and three in Thinking mode: the click edge, the pause after a miss and the order of the new diamonds.
• Qwen 3.6 35B: a working game with four lost tests in Thinking mode. In Instruct mode, it stopped calling tools and handed in a game without clicks.
• Qwen 3.8 27B: all hidden tests passed in Think: medium, with the most own tests. In Instruct mode, only the click edge was wrong.

For a small mouse game like this, Qwen 3.8 27B and Qwen 3.6 27B in NVFP4 build a working version in one attempt in either mode. Qwen 3.6 35B needs Thinking mode. The rules that break are the exact ones: pixels at the edge of an area, the order of the new diamonds and the last frame of a pause.

References

Benchmark runs with NVFP4 quantization

Benchmark runs with FP8 quantization (side note)