Introduction

Can a local model build a tower jumper game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a tower jumper like Icy Tower as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 13 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a jump-and-run up a tower. The player runs left and right between two walls and jumps from floor to floor. A faster run gives a higher jump, and the player bounces off the walls. The floors are short platforms, one every 48 pixels, and every 25th floor is a full floor. When the player reaches floor 5, the tower starts to scroll down, and it gets faster every 30 seconds. Jumps over two or more floors build a combo that gives bonus points. The game is over when the player falls below the bottom of the screen, and Enter starts a new game.

The exact rules the hidden tests check include the start state, the rule for new floors, running and slowing down, the wall bounce, the jump speed, landing only from above, walking off a floor, the combo and its timer, the scrolling speed, the camera, game over and a new game, the growing list of floors, and what the screen shows. The model chooses how to build all of this.

Results

Qwen 3.6 27B in both modes and Qwen 3.6 35B in Thinking mode passed every result. Qwen 3.6 35B in Instruct mode and Qwen 3.8 27B in Instruct mode lost two, and Qwen 3.8 27B in Think: medium stopped after a first draft. The chart below shows the score of each setup: the share of passed results, the build of the game and the 13 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B passed every hidden test in both modes. In Thinking mode, it wrote the whole game in one answer, fixed one compile error and submitted the game after only five tool calls (see "One run step by step").

In Instruct mode, it also wrote the whole game at once, 550 lines, and fixed one compile error: a variable declared twice. At that point, the game was already correct: we checked this version, and it passes all 13 hidden tests. The model then spent 21 more tool calls on its own tests, because some of them expected wrong results. Eight of these calls repeated an edit that was already applied and were refused. In the end, all 11 of its tests passed, and it submitted the same game code.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B passed every hidden test in Thinking mode and lost two in Instruct mode. In Thinking mode, it started with a short first version and added the rules in a few large steps. The checks found a wrong range for a random number and a continue outside a loop, and its floor test showed that the game created too few floors. It wrote only two tests, for the start state and the floor rule, but in the end every rule of the game was right.

In Instruct mode, the rules of the game were right too: all hidden tests of the movement, the combo, the camera and the floors passed. But the game did not draw the floors and never showed "GAME OVER". The model noticed that the drawing was not finished, but it never changed it. Instead, it ran the check 43 times in a row without any change, until the run hit its limit of 50 rounds (see below).

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B lost two hidden tests in Instruct mode, and in Think: medium its run ended after a first draft. In Instruct mode, the rules of the movement and the combo were right. Two mistakes broke the other two tests: the game drew the wrong floors, and a new game kept the floors of the old game (both are described below).

In Think: medium, the model started as the task suggests, with a small first version: the start state, the floors, a simple drawing and two tests. In its next answer, while it planned the movement, its reasoning got stuck in a loop, and the runtime stopped the answer as broken output. In this test series, a broken answer ended the work, so the first draft was graded. It passed only the start state test.

In this game, thinking mode helped Qwen 3.6 35B, but not Qwen 3.8 27B. Qwen 3.6 27B passed in both modes. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.6 35B in Thinking mode wrote only two tests and passed every hidden test. Qwen 3.8 27B in Instruct mode wrote 12 tests and lost two hidden tests: none of its tests checked the screen or the floors after a new game. Qwen 3.8 27B in Think: medium wrote two tests for its first draft only.

Checks and edits

Qwen 3.6 35B in Instruct mode ran 45 checks, but 43 of them came in a row without a change to the file. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.8 27B in Instruct mode made the most edits: it needed four tries to get its floor rule right and many more for its own tests. Qwen 3.6 27B in Thinking mode needed only two checks and one edit.

Unwanted tool calls

Qwen 3.6 27B in Instruct mode had nine rejected tool calls. Eight of them repeated an edit that was already applied, so they were refused. One referred to text that the answer did not contain. Qwen 3.8 27B in Instruct mode had one call with broken call syntax. The other setups had no rejected calls. Qwen 3.6 35B in Instruct mode is a different case: none of its calls was rejected, but 43 of them were the same check of the same file.

Tokens

Every tool call sends the whole conversation to the model again, so long runs read far more tokens. Qwen 3.6 35B in Instruct mode read almost two million tokens, most of them for its repeated checks. Qwen 3.8 27B in Instruct mode read 1.7 million tokens over 39 tool calls. Qwen 3.6 27B in Thinking mode read the fewest tokens of the setups that built a full game. Thinking setups write more tokens because the model reasons before each step.

One run step by step

Qwen 3.6 27B in Thinking mode built the game with five tool calls and passed every hidden test.

1. In its first answer, it wrote the whole game in one file: 429 lines with eight tests. The tests covered the start state, a jump, landing and score, walking off a floor, a combo, game over, a new game and the texts on the screen.
2. The check found one compile error: two variables declared in one line, which SC# does not allow.
3. It replaced that line with two lines.
4. The check passed: all eight of its tests were green.
5. It submitted the game, and all 13 hidden tests passed.

How the models fail

The floors are drawn above the screen

The game of Qwen 3.8 27B in Instruct mode draws the wrong floors. Floor n has its top at world y = -48 * n, and the screen shows the world from cameraY down to cameraY + 240. So floor n is on the screen when -48 * n - cameraY is between 0 and 240. The model computed the range of floors with the wrong sign:

At the start of a game, the player stands on the ground, but the screen shows only one floor at the top. In the hidden test, floors 2 to 6 should be on the screen, but the game drew floors 6 to 11, so only one of the five floors was visible. When we fixed the range, this test passed.

A new game keeps the old floors

In the same game, a new game does not create new floors. The task says that a new game starts "with the floors by the floor rule". The model's new game resets every variable and then calls the function that adds floors. But this function only adds floors after the last one in the list, so the old floors stay:

void NewGame()
{
    playerX = 152;
    // ... every other variable is reset ...
    GenerateFloors(highestFloor + 20);   // adds floors only after the last one in the list
}

The hidden test changes floors 1 to 3 into full floors, ends the game and presses Enter. After the new game, these floors were still 288 pixels wide, but normal floors must be from 48 to 160 pixels wide. The games that passed this test clear both floor lists first. The model's own test of a new game checked the player, the score and the game over flag, but not the floors. With the two lists cleared and the drawing range fixed, the game passes all 13 hidden tests.

No floors and no "GAME OVER"

The game of Qwen 3.6 35B in Instruct mode never draws the floors. Its drawing code has the background, the walls, the player and the score, but no floors and no "GAME OVER" text. Its own review even listed the drawing of the floors as not finished. But after that, the model only ran the check again and again: 43 checks in a row with the same file and the same result, two passing tests. It never submitted the game, and the run ended at its limit of 50 rounds. When we added the floors and the text to the drawing, the game passed all 13 hidden tests.

A first draft and a loop

Qwen 3.8 27B in Think: medium stopped after its first version. The task suggests building the app in steps, and the model did this: its first version had the start state, the floor rule, a simple drawing and two tests. Each frame only drew the background and the player. In its fifth answer, it planned the movement, the landing and new tests. Then its reasoning started to repeat the same phrase, "Writing test code...", hundreds of times, until the runtime stopped the answer as broken output. In this test series, a broken answer ended the work, so the first draft was graded: the start state test passed, and the other 12 hidden tests failed because nothing in the game moved.

Conclusion

Qwen 3.6 27B passed every hidden test in both modes, and Qwen 3.6 35B in Thinking mode did the same.

• Qwen 3.6 27B: all hidden tests passed in both modes. In Thinking mode, it needed only five tool calls.
• Qwen 3.6 35B: all hidden tests passed in Thinking mode. In Instruct mode, the rules were right, but the game did not draw the floors and the model looped on checks.
• Qwen 3.8 27B: two hidden tests lost in Instruct mode, both because of floors: the drawing range and a new game. In Think: medium, a loop in its reasoning ended the run after a first draft.

For a game like this, Qwen 3.6 27B in NVFP4 was the most reliable choice in this test. In all setups, the rules of the movement were the easy part. The mistakes were in the parts that the models' own tests did not check: the screen and a new game.

References

Benchmark runs with NVFP4 quantization