Introduction

Can a local model build a side-scrolling run-and-gun game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a Contra-like run-and-gun game as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 15 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a side-scroller on a level of 120 x 15 tiles. The player runs and jumps over pits and platforms, aims in eight directions and shoots. Soldiers keep coming from the right edge, two turrets fire aimed bullets, and a boss waits at the end of the level. Two power-ups give a spread gun and a rapid gun. The camera follows the player but never moves back.

The exact rules the hidden tests check include the start state, running and the camera, the jump and gravity, platforms and pits, crouching, the muzzle and the speed of a shot, the fire rate and the bullet limit, the weapons, hits and scores, how soldiers walk and appear, when enemies may shoot, dying and the screen. The task also gives the order of the steps in a frame. Some parts are left to the model, for example when a soldier shoots, as long as it stays within the given limits.

Results

No setup passed every hidden test: Qwen 3.6 27B in Thinking mode came closest with two failed tests, and Qwen 3.6 35B built no working game. The chart below shows the score of each setup: the share of passed results, the build of the game and the 15 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a working game in both modes: 13 of 15 hidden tests passed in Thinking mode and 11 in Instruct mode. In both modes, a new player bullet moved already in the frame in which it was fired. This one rule broke two tests: the muzzle position and the fire rate (see below).

In Thinking mode, this was the only mistake. In Instruct mode, the enemies also broke two rules: a new soldier bullet moved in its first frame, and the turrets fired at random moments, so the time between two shots was often far below 60 frames. Instruct mode wrote 13 tests of its own and Thinking mode 14, but none of them checked where a new bullet starts.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not build a working game in either mode. In Instruct mode, it wrote a game of 728 lines in one go, but the file did not compile. In Thinking mode, it never created a file. Both problems are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B built a working game in both modes: 12 of 15 hidden tests passed in Instruct mode and 11 in Think: medium. In Instruct mode, the game had the same bullet mistake as Qwen 3.6 27B. It also did not let the player die in a pit while the player was invulnerable, which the task requires.

In Think: medium, the model was the only one that kept new player bullets in place in their first frame. Its game lost hidden tests for other reasons. It kept a private list with one shot timer per soldier, so the game failed whenever a test added a soldier, because the list was shorter than the soldier list. And a new soldier moved already in the frame in which it appeared, so a soldier that appeared over a pit dropped at once.

The best result came from Qwen 3.6 27B in Thinking mode, but all four working games were close. Each of them lost two to four hidden tests, mostly because of the order of steps in a frame. With one run each, the differences between them are a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.8 27B in Instruct mode wrote only 5 tests and still had the second-best score. Qwen 3.6 27B wrote the most tests in both modes, and none of them caught the bullet mistake. Qwen 3.6 35B is not in the chart: neither of its runs produced a game that compiles.

Checks and edits

Qwen 3.6 27B in Instruct mode needed the most checks and edits. A check compiles the game and runs the model's tests. An edit changes some lines of the file. The other three working games needed six checks each. Qwen 3.6 35B stopped early in Instruct mode and never reached the file in Thinking mode.

Unwanted tool calls

Qwen 3.8 27B in Think: medium had the most rejected calls: four of its 23. Three referred to text that the answer did not contain, and one had broken call syntax. Qwen 3.8 27B in Instruct mode had three rejected calls: two referred to missing text, and one edit was invalid. Qwen 3.6 27B in Instruct mode tried twice to rewrite the whole file instead of editing it, and both attempts were refused. Qwen 3.6 27B in Thinking mode made no wrong calls. Qwen 3.6 35B made no wrong calls either, but it often answered without any tool call: in Instruct mode it needed three reminders.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls read far more tokens. Qwen 3.6 27B in Instruct mode read the most, over 1.2 million tokens, because of its many checks and edits. Qwen 3.8 27B in Think: medium wrote by far the most tokens, almost 140,000, because it reasoned at length before its steps. Qwen 3.6 35B wrote about 59,000 tokens in each mode, much of it code as text in its answers instead of changes to the file.

One run step by step

Qwen 3.6 27B in Thinking mode wrote most of the game at once and then fixed it in a few small steps.

1. It wrote a first version of 681 lines: the level, the player, the enemies, the drawing and four tests.
2. Two checks found variables declared in one line, which SC# does not allow. The model split the declarations.
3. The next check found a brace problem in the code for bullet hits. The model fixed it with two small edits.
4. One of its tests expected the jump speed -7 after the first frame. The model saw that gravity already acts in the same frame and changed the test to -6.6, which is what the task says.
5. It added ten more tests in one step. One test expected the rapid gun's pause to be 4 after the frame, but the pause already drops by one at the end of the frame. The model fixed the test to 3.
6. All 14 of its tests passed, and it submitted the game.
7. 13 of 15 hidden tests passed. The two failed tests both came from the first frame of a new bullet.

The model fixed its tests, not the game, in steps 4 and 5, and both times this was right. Its test for shooting checked the number of bullets and their speed, but not where a new bullet starts, so the one wrong rule stayed.

How the models fail

The bullet moves in the frame it is fired

Three of the four working games moved a new player bullet already in the frame in which it was fired. The task fixes the order of steps in a frame: step 5 fires the bullet at the muzzle, and step 6 moves only "player bullets that existed before this frame". So after the first frame, a new bullet must still be at the muzzle, 8 pixels in front of the player.

With the player at x = 100, the first bullet must be at x = 108. In the three games, it was at 113. The same mistake broke two hidden tests in each game: the muzzle position in all eight directions, and the fire rate. Qwen 3.6 27B in Instruct mode made the same kind of mistake once more: a soldier's new bullet moved in its first frame, so the hidden test found a bullet that did not start at any enemy.

A new soldier moves at once

In Qwen 3.8 27B's game in Think: medium, a soldier moved already in the frame in which it appeared. The task puts the movement of soldiers first in step 8 and the new soldiers after it: "Then the enemies may act". The game added the new soldier first and then moved all soldiers:

// Step 8: soldiers, enemy bullets, enemy actions
if (cameraX < 1600 && soldierX.Count < 4)
{
    ...
    soldierX.Add(sx);
    soldierY.Add(208);
    ...
}
for (int si = soldierX.Count - 1; si >= 0; si--)
{
    // a soldier above a pit falls by 4, others walk
}

On solid ground, the new soldier only walked one pixel, which still fit the allowed area. But the place of a new soldier was random, and in the graded run one soldier appeared above a pit. It dropped to y = 212 in its first frame, while a new soldier must stand at y = 208. This is the same mistake as the bullet in the other games, only for another object.

A private list that the tests do not fill

The same game failed three more hidden tests because it kept part of the soldier state in its own list. Next to the soldier lists from the task, the model kept a list of shot timers, one per soldier:

for (int si = 0; si < soldierX.Count; si++)
{
    soldierShootTimer[si]--;
    ...
}

The task says that its variables are the whole game state, and that tests read and set them. A hidden test that adds a soldier fills soldierX, soldierY and soldierDir, but cannot know about soldierShootTimer. The timer list stays shorter, and the game stops with "List index 0 is out of range". The tests for dying, for walking soldiers and for hits failed this way.

No death in a pit while invulnerable

Qwen 3.8 27B's game in Instruct mode ended the damage check at once while the player was invulnerable. The task says: "If playerY > 240 (fell into a pit), the player dies even when invulnerable." The game returned before it reached the pit check:

if (invulnerableFrames > 0) { return; }
...
if (playerY > 240)
{
    KillPlayer();
}

After a death, the player is invulnerable for 120 frames. A player who fell into a pit in this time kept falling, and the life was not lost.

Turrets that fire at random

The turrets in Qwen 3.6 27B's game in Instruct mode fired with a small chance in every frame. The task leaves the timing to the model, but within limits: "The time between two shots of a turret is from 60 to 120 frames". A chance of 1.5% per frame gives no such limits, and the hidden test measured 30 frames between two shots. The task allows randomness, but only inside the given limits.

A file that does not compile and a file that never existed

Qwen 3.6 35B lost both runs before a working game existed.

• Instruct mode: the model wrote a game of 728 lines in one go. Several variables were declared in one line, which SC# does not allow. The model fixed the global ones, and the next check showed the same problem inside a function. It fixed one more place, but did not check again. Then it answered three times with the whole file as text and plans for more fixes, without a tool call, and the run stopped. The graded file still had a declaration of two variables in one line.
• Thinking mode: the first answer had the code only as text, so no file existed. After a reminder, the model wrote a long plan and many code blocks, among them dozens of tests, and wrote again and again that it would now create the file. It never did. The answer grew to about 176,000 characters, and the runtime stopped it as broken output. In this test series, a broken answer ended the work.

Conclusion

No setup passed every hidden test, but four of six built a working run-and-gun game.

• Qwen 3.6 27B: 13 of 15 hidden tests in Thinking mode, the best result, and 11 in Instruct mode. Both modes moved a new bullet in its first frame.
• Qwen 3.8 27B: 12 of 15 hidden tests in Instruct mode and 11 in Think: medium. Think: medium got the bullet rule right, but lost tests to a private list and a soldier that moved at once.
• Qwen 3.6 35B: no working game in either mode.

The main lesson of this task is the order of steps in a frame. Most failed tests came from objects that moved, appeared or were checked one step too early. For a game like this, the 27B models in NVFP4 give a good base, but the rules for the first frame of new objects need their own tests.

References

Benchmark runs with NVFP4 quantization