Introduction

Can a local model build a fixed shooter game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a fixed shooter like Space Invaders as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 16 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The player moves a cannon along the bottom of the screen and shoots at a formation of 30 aliens. The aliens stand in 5 rows of 6. The formation moves sideways, and at the edge of the field it steps down and turns. It gets faster as aliens die. The aliens drop bombs, and the player has three lives. The game is over when a bomb takes the last life or the aliens reach the bottom line. The player wins when no alien is left.

The exact rules the hidden tests check include the names and start values of the game variables, the order of the steps in each frame, the speed and limits of the cannon, the flight of the shot, the points for each row, the step and the turn of the formation, the bottom line, the start point of a bomb and the restart after the game ends. Two rules leave the design to the model but set limits: bombs must appear at least once every 90 frames, with at most 3 at once, and the formation must move at least every 10 frames once 15 or fewer aliens are alive. The model also had to add a pause, an explosion effect, a best score and a short help text.

Results

Qwen 3.6 27B passed every result in both modes, and so did Qwen 3.8 27B in Think: medium. Qwen 3.8 27B in Instruct mode lost one result, and Qwen 3.6 35B built no working game in either mode. The chart below shows the score of each setup: the share of passed results, the build of the game and the 16 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B passed all hidden tests in both modes. In Instruct mode, it wrote the whole game in one file of about 590 lines. The first check found one compile error: the code created a list with a start size, which SC# does not support. After that fix, one of its own tests expected a wrong position of the cannon, and the model fixed the test. It submitted the game after 9 tool calls, the fewest of all setups.

In Thinking mode, the model also wrote the whole game at once and hit the same compile error. Its own tests then found a bug in the explosion effect. Before it submitted, it changed the bomb code so that a new bomb does not fall in its first frame, as the task says. One of these edits removed a list that the code still used, and the next edit put it back. Once, the model answered without a valid tool call and got a reminder.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not submit a working game in either mode, although its first game in Instruct mode was correct. In Instruct mode, the first version compiled and passed all of its own tests. The model did not submit it, changed the bomb code and broke the file with two edits. In Thinking mode, the model never made a valid tool call, so no game file existed. Both runs are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium, and all but one in Instruct mode. In Think: medium, the model built the game in small steps, from a short skeleton to the full rules (see "One run step by step").

In Instruct mode, the game was correct except for one rule: the formation did not get fast enough at the moment when 15 aliens were left (see below). The run also had a problem that its own tests found: the cannon moved in a while loop as long as an arrow key was held, so a test with a held key never finished. The model saw this only when it tried to hand in the game and one of its tests did not finish in time. It then changed the loop to a simple if.

All four setups that built a game passed the hidden tests for movement, shots, bombs, scores and drawing. The only failed test in these runs checks a design rule with a limit. With one run per setup, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

All setups that built a game wrote between 10 and 13 tests, and none of them tested the speed rule. The tests covered the start state, the cannon, the shot, the points, the first move of the formation, the restart and the pause. The speed rule was the only rule that one of these games broke. Qwen 3.6 35B is not in the chart: neither of its runs ended with a game that compiles.

Checks and edits

Qwen 3.6 35B in Instruct mode ran 40 checks, 35 of them on the same broken file. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.8 27B in Instruct mode made the most edits: it built the game in many small steps and had to repeat several of them. Of the setups that built a game, Qwen 3.6 27B needed the fewest checks in both modes.

Unwanted tool calls

Qwen 3.8 27B in Instruct mode had 15 rejected tool calls. Seven referred to text that the answer did not contain. Three repeated an edit that was already applied, so they were refused. Two had broken call syntax, and one was a call to a tool that does not exist. Two more tried to hand in the game while one of its own tests did not finish. Qwen 3.6 35B in Thinking mode made three calls, and none of them was a valid tool call. Qwen 3.6 27B had no rejected calls in either mode.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls read far more tokens. Qwen 3.6 35B in Instruct mode read over three million tokens, most of them during its 35 checks of the broken file. Qwen 3.8 27B in Instruct mode read about 1.4 million tokens because of its 41 tool calls. Thinking mode writes more tokens because the model reasons before each step. Qwen 3.6 35B wrote the most tokens in both modes: in Thinking mode, each of its seven answers contained the whole game.

One run step by step

Qwen 3.8 27B in Think: medium built the game in small steps and checked it after each larger step.

1. Its first call referred to text that the answer did not contain, so it was rejected. The model then wrote a skeleton of about 140 lines: the variables, the start state, the drawing and two tests. Both tests passed.
2. It added the cannon, the shot, the points and firing, and then the formation, the speed rule, the end of the game and the win.
3. It added the bombs, the explosion effect and the best score. Then it made sure that bombs stop when the game is over and added more tests.
4. One of its tests failed: the cannon did not fire. The test pressed Space for one frame, but the code fired only while the key was held. The model changed firing to react to a key press, and all 6 tests passed.
5. It added four more tests for the points, the move of the formation and the restart. All 10 tests passed.
6. It submitted the game, and all 16 hidden tests passed.

How the models fail

The formation speeds up too slowly

The game of Qwen 3.8 27B in Instruct mode lowers the move delay by only one step per frame. The task says that moveFrames, the number of frames between two moves of the formation, is at most 10 once 15 or fewer aliens are alive. The model chose good limits, but it moves towards them slowly:

The hidden test shoots 15 aliens, one in each frame, and then checks the delay. In normal play, kills are further apart, so the game would reach the limit a few frames later. But the rule is exact: the limit must hold as soon as 15 or fewer aliens are alive. Only this hidden test failed.

A correct game that was never handed in

Qwen 3.6 35B in Instruct mode had a correct game after its first tool call, but it never submitted it. The first version of about 510 lines compiled, and all 10 of its own tests passed. When we ran the hidden tests on this version afterwards, it passed all 16. Its bombs were random, so the rule of one bomb in every 90 frames held only by chance, but the tests passed.

The model got three reminders to call SubmitApp. Instead, it ran more checks and listed possible problems. On the third check, its own bomb test failed, although the game had not changed. The test itself depended on chance:

The model tried to fix the bomb code. Its first edit was incomplete: the bomb loop lost its closing brace, and old braces stayed in the file. The check reported an extra brace. The model's fix removed one brace too many. Now the main game function never ended, and the next function did not compile. The model ran the check 34 more times without any edit, until the run reached its limit of model rounds. The last version is graded, so the run scored 0.

No valid tool call

Qwen 3.6 35B in Thinking mode never created a game file. In its first three answers, it put the game code into the tool-call block, but not in the form of a valid call, so no tool ran. In the next four answers, it wrote the whole game as text: once as a copy of a file view and three times in tags it made up. Three reminders said that the file did not exist yet. The run stopped after three answers in a row without a tool call. The text version would not have compiled either: it assigned a value to a field of a struct, which SC# does not allow.

Conclusion

Qwen 3.6 27B passed every result in both modes, Qwen 3.8 27B passed every result in Think: medium and lost one in Instruct mode, and Qwen 3.6 35B built no working game.

• Qwen 3.6 27B: all hidden tests passed in both modes, with the fewest tool calls of all setups.
• Qwen 3.8 27B: all hidden tests passed in Think: medium. In Instruct mode, the formation reached the required speed a few frames too late.
• Qwen 3.6 35B: no working game. In Instruct mode, a correct first version was lost through edits. In Thinking mode, the model never made a valid tool call.

For a game like this, Qwen 3.6 27B in either mode and Qwen 3.8 27B in Think: medium in NVFP4 were reliable in this test. With any model, check that rules with limits are tested at the exact moment they start.

References

Benchmark runs with NVFP4 quantization