Introduction

Can a local model build a space rocks shooter game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a space rocks shooter like Asteroids, with power-ups, as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 14 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a space shooter on a screen that wraps around at the edges. The player turns a ship with the arrow keys, thrusts forward and shoots. Without thrust, the ship slowly drifts to a stop. Large rocks split into two medium rocks, medium rocks into two small ones, and small rocks disappear. Destroyed rocks can leave power-ups: triple shot, rapid fire and a shield. A rock that touches the ship costs a life, and a new wave with one more rock starts when the field is empty.

The exact rules the hidden tests check include the turning step, the thrust, the drift and the speed limit, the wrap-around, where a bullet starts, how fast it flies and how long it lives, the shot cooldown and the bullet limit, the distance at which a bullet hits a rock, the scores, the effects and timers of the power-ups, what happens when the ship is hit, the next wave and the colors and shapes on the screen. The task also gives the order of the steps in each frame. The model chooses how to build all of this.

Results

Qwen 3.6 27B and Qwen 3.8 27B passed every result in both modes, and Qwen 3.6 35B built no game that compiles. The chart below shows the score of each setup: the share of passed results, the build of the game and the 14 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B passed all 14 hidden tests in both modes. In both modes, the model wrote almost the whole game in its first file, about 700 lines. The first checks then showed code that SC# does not accept, one problem at a time, and the model fixed each one.

In Instruct mode, the model needed 10 checks and 13 edits. Once it described a fix as text without calling a tool and needed one reminder. In Thinking mode, it needed 6 checks and 8 edits, wrote 25 tests, more than any other setup, and tried to submit the game once while some of its own tests still failed (see "One run step by step"). In both modes, the failing tests were wrong, not the game (see "Own tests against the order of the frame").

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not build a game that compiles in either mode. In Instruct mode, it wrote the first file with two or more variables declared in one line. After the first check, it changed the line, but kept the same kind of declaration, and then it stopped calling tools. In Thinking mode, its first tool calls had broken call syntax, so no file was created, and its third answer turned into a long plan that never became a tool call. Both problems are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed all 14 hidden tests in both modes. In Instruct mode, it started with a small skeleton of 120 lines and added the rules in steps, as the task asks. This took 10 checks and 10 edits. It also made 7 unwanted tool calls: three refused repeated edits and four calls that referred to text that the answer did not contain.

In Think: medium, the model wrote a first version of 484 lines and needed only 3 checks and 5 edits, the fewest of the setups that built a game. It corrected the expected timer values in its own tests before it ran them (see "Own tests against the order of the frame").

Both 27B models passed every hidden test in both modes, so this game does not separate them. The difference is in the way to the result: Qwen 3.8 27B in Think: medium needed the least rework. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

All four 27B setups wrote tests that passed at the end, from 9 to 25 tests. Qwen 3.6 27B in Thinking mode wrote the most. More tests did not change the result here, because all four games passed every hidden test. Qwen 3.6 35B is not in the chart: neither of its runs produced a game that compiles.

Checks and edits

Of the setups that built a game, Qwen 3.8 27B in Think: medium needed the fewest checks and edits, and the two Instruct setups the most. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.8 27B in Instruct mode made many edits because it built the game in small steps. Qwen 3.6 27B in Instruct mode made many edits partly because its first file had several parts that SC# does not accept. Qwen 3.6 35B made almost no edits: in Instruct mode it stopped after one edit, and in Thinking mode it never created a file.

Unwanted tool calls

Qwen 3.8 27B in Instruct mode made the most unwanted tool calls: 7. Three of them repeated an earlier edit after the file had changed, so they were refused. Four referred to text that the answer did not contain. Qwen 3.8 27B in Think: medium made two calls of the second kind. Qwen 3.6 27B in Thinking mode had one rejected call: it tried to submit the game while 6 of its own tests failed. Qwen 3.6 27B in Instruct mode made no wrong calls. Qwen 3.6 35B in Thinking mode made two calls with broken call syntax and no working call at all.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls read far more tokens. Qwen 3.6 27B in Instruct mode and Qwen 3.8 27B in Instruct mode each read about one million tokens, most of them because of their many checks and edits. Thinking mode writes more tokens because the model reasons before each step. Qwen 3.6 35B in Thinking mode wrote the most tokens of all setups in only three answers, because its last answer was a plan that did not end.

One run step by step

Qwen 3.6 27B in Thinking mode wrote almost the whole game at once, then fixed the code and its own tests until everything passed.

1. It created a first file of 733 lines with the game and a first set of tests, and changed one function before the first check.
2. The checks found four kinds of code that SC# does not accept, one after another: an output parameter, two variables declared in one line, a do-while loop and case labels that were named constants. It fixed each one.
3. It tried to submit the game without running a check after the last fix. The submission was rejected: 6 of its 25 tests failed.
4. All six failures were in its tests, not in the game: wrong start positions in the test setup and timer values that did not take the end of the frame into account. It fixed the tests.
5. One test still failed. It expected an empty field after the last small rock was destroyed, but the task starts the next wave in the same frame. The model fixed the test to expect wave 2 with 5 rocks.
6. All 25 tests passed. It submitted the game, and all 14 hidden tests passed.

How the models fail

A declaration the model could not fix

Qwen 3.6 35B in Instruct mode lost its run because of one line at the top of the file. SC# does not allow several variables in one declaration, and global variables need a start value. The first check reported the missing start value. The model added start values, but kept the commas:

After this edit, the model did not run another check. In its next three answers, it wrote the file and a list of changes as text, including a correct version of these lines, but it did not call a tool to save them. After three answers in a row without a tool call, the run ended. The graded file did not compile, so no hidden test could run.

Qwen 3.6 27B made the same kind of mistake in both modes, with two variables in one line inside a function. In both runs, it read the error and split the line in the next edit.

A plan that never became a tool call

Qwen 3.6 35B in Thinking mode never created a game file. Its first two answers contained tool calls with broken call syntax, so the runtime could not run them. In its third answer, the model planned the next part of the game and checked details again and again. Several times it wrote that it would make the tool call now, but it went on checking. After about 196,000 characters, the runtime stopped the answer as broken output. In this run, a broken answer ended the work, so there was no file to grade.

Own tests against the order of the frame

All four 27B setups first wrote tests that expected a timer value one frame too early. The task gives the order of the steps in each frame. A shot sets the cooldown to 8, and a hit sets the time without damage to 90. But step 10 of the same frame decreases every timer by 1, so after the frame the values are 7 and 89:

All four setups changed the tests and not the game, which was the right choice. Qwen 3.8 27B in Think: medium found the problem before it ran the tests. Qwen 3.6 27B in Instruct mode went one step too far: it also expected the life of a new bullet to drop from 40 to 39, but the task says that a bullet fired in this frame does not move yet. The next check showed this, and the model set the test back to 40.

Conclusion

Qwen 3.6 27B and Qwen 3.8 27B passed every hidden test in both modes, and Qwen 3.6 35B built no game that compiles.

• Qwen 3.6 27B: all hidden tests passed in both modes, after several fixes for code that SC# does not accept.
• Qwen 3.8 27B: all hidden tests passed in both modes. In Think: medium, it needed the fewest checks and edits of the setups that built a game.
• Qwen 3.6 35B: no game that compiles. In Instruct mode, it could not fix a declaration and stopped calling tools. In Thinking mode, it never created a file.

For building a game like this as an agent, both 27B models in NVFP4 worked in both modes. Qwen 3.6 35B was not able to finish the task in this test.

References

Benchmark runs with NVFP4 quantization