Introduction

Can a local model build a turn-based artillery game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write an artillery duel game like Scorched Earth as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 11 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

Four tanks stand on hilly terrain and take turns to fire at each other. The player sets the angle and the power of the turret with the arrow keys and fires with Space. The three computer tanks choose their own angle and power. A shell flies under gravity and wind, and when it hits the ground or a tank, it explodes: it digs a crater, damages the tanks nearby, and tanks without ground under them fall and lose health. The last tank alive wins the round.

The exact rules the hidden tests check include the terrain formula, the start state, how a shell flies frame by frame (and that it does not move in the frame in which it is fired), the crater, the damage and the falling rules, the order of the turns, the large missile with its limited ammunition, the limit of 60 frames for a computer tank to fire, and the rule that the wind changes by at most 3 between turns. The model chooses how the computer tanks aim and how the wind changes within this limit.

Results

Qwen 3.8 27B in both modes and Qwen 3.6 35B in Thinking mode passed every result, Qwen 3.6 27B lost one result in each mode, and Qwen 3.6 35B in Instruct mode built no game that compiles. The chart below shows the score of each setup: the share of passed results, the build of the game and the 11 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a working game in both modes and lost one hidden test in each. In Instruct mode, the wind could change by 4 between turns, one more than the task allows. The model first had a correct wind rule, then changed it twice, and the second change broke it (see below). In Thinking mode, a shell did not move in the first frame after it was fired, so its whole flight came one frame late.

In Instruct mode, the model wrote the whole game at once and then reviewed it against the task in many small edits. In Thinking mode, it built the game in steps from a short first version and wrote 20 tests, more than any other setup. One of these tests checked the flight of the shell, but it was written around the same extra frame as the game, so it passed.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B passed every hidden test in Thinking mode, but its game in Instruct mode did not compile. In Thinking mode, the model wrote a game of more than a thousand lines at once. The game was correct, but four of its own tests expected wrong values, so the model never handed the game in. The work ended at the limit of 50 answers, and the last version of the file was graded: it passed every hidden test.

In Instruct mode, the first version used array syntax that SC# does not have. The model fixed that, but a later edit left an extra opening brace in one function. The model did not find it for more than 20 checks, and when it finally removed lines it took for duplicates, it deleted a brace that the code needed. The game never compiled, and the work ended at the limit of 50 answers. Both runs are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in both modes, with the fewest tool calls of all setups. In Instruct mode, the model started with a short first version, added the game and its tests in a few large edits, wrote 8 tests and submitted the game after 15 tool calls. In Think: medium, it needed 15 tool calls as well. Its first version already had a test for the rule that a shell does not move in the frame in which it is fired (see "One run step by step").

Both modes chose the same correct wind rule: a random change from -3 to 3, written with the right bounds for the random function of SC#.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests did not decide the result. Qwen 3.6 27B in Thinking mode wrote the most tests and still missed one rule, because its test expected the same wrong timing as the game. Qwen 3.6 35B in Thinking mode had a correct game, but four of its 15 tests expected wrong values, and they kept it from handing the game in. Qwen 3.6 35B in Instruct mode is not in the chart: its game never compiled, so its tests never ran.

Checks and edits

Qwen 3.6 35B ran more than 20 checks in each mode, most of them without any change to the file. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Both runs of Qwen 3.6 35B ended at the limit of 50 answers. Qwen 3.8 27B needed the fewest checks and edits in both modes.

Unwanted tool calls

Qwen 3.6 35B in Thinking mode had 18 rejected tool calls. Ten of them referred to text that the answer did not contain, three were incomplete edits, one asked for a tool that does not exist, and four tried to rewrite the whole file, which is not allowed once the file exists. In Instruct mode, the same model tried to rewrite the whole file eight times and repeated one edit that was already applied. The other setups had at most four rejected calls each.

Tokens

Every tool call sends the whole conversation to the model again, so long runs read far more tokens. Both runs of Qwen 3.6 35B read more than three million tokens, because they used all 50 answers. Qwen 3.8 27B read about half a million tokens in each mode. Thinking modes write more tokens, because the model reasons before each step.

One run step by step

Qwen 3.8 27B in Think: medium tested the trickiest timing rule in its first version and needed only four checks.

1. It started with a first version of 117 lines: the state, the new game, the player's controls and firing, and three tests. One test fired with Space and checked that the shell starts at the end of the turret and has not moved yet.
2. One check was rejected because it passed a parameter that the tool does not accept. The next check passed all three tests.
3. It added the pause, the effects and the wind rule, then the flight of the shell, the explosion, the turns and the computer tanks, in three edits. Three other edits were rejected because they referred to text that the answer did not contain.
4. The check found one compile error: a do-while loop, which SC# does not have. The model rewrote it as a while loop.
5. It added four tests: a shell that hits the ground makes a crater and ends the turn, a computer tank fires within 60 frames, the pause freezes the game, and the round ends with one tank left. All seven tests passed.
6. It submitted the game, and all 11 hidden tests passed.

How the models fail

The wind changes by 4

The game of Qwen 3.6 27B in Instruct mode could change the wind by 4 between turns, but the task allows at most 3. The random function of SC# includes the lower bound and excludes the upper bound. The model's first version used a valid range. During its review, it decided twice that its own range was wrong, and the second change made it wrong:

The model went through three versions: first a range from -3 to 2 with an extra rule, then RandomInt(-3, 3), which is also within the limit, and finally RandomInt(-4, 4). Each time it explained the range in its answer, and the last explanation was wrong. None of its seven tests checked the wind. The hidden test plays nine turns and checks every change of the wind; a change of -4 appeared in every check we ran.

The shell waits one frame too long

In the game of Qwen 3.6 27B in Thinking mode, a fired shell stayed in place for one more frame. The task says that a shell does not fly in the frame in which it is fired. The game already did nothing else in that frame, because firing and flying are different branches of the frame. The model added a flag on top of this, and the flag skipped the next frame as well:

The hidden test fires with Space and checks the shell after every frame: after the first frame, it was still at the end of the turret. The model had a test for the flight, but the test fired the shell directly, ran one frame, and only then measured the movement in the next frame. It was built around the extra frame, so it passed. Qwen 3.6 27B in Instruct mode had written in its answer why the same rule needs no extra code: flight and firing are in different branches.

A correct game that was never handed in

Qwen 3.6 35B in Thinking mode wrote a game that passed every hidden test, but it did not hand it in. After three compile errors were fixed, 11 of its 15 tests passed. The four other tests expected wrong values: for example, that a shell is below the bottom of the screen after ten frames, or that a text is on the screen before any frame was drawn. The game was right and the tests were wrong. The model tried to change the tests with calls that referred to text the answer did not contain, and then ran the same check 15 times in a row without any change. The work ended at the limit of 50 answers. Because the last version of the file is graded, the game still passed every hidden test.

A deleted brace

Qwen 3.6 35B in Instruct mode never got its game to compile.

• The first version used array syntax from C#, which SC# does not have. The model replaced the arrays with lists.
• An edit of the explosion effect left a second opening brace at the start of one function. The compile error pointed at the next function, a few dozen lines lower, and the model did not find the cause.
• It ran more than 20 checks that all showed the same compile error, without changing the file, and its attempts to rewrite the whole file were refused, because this is not allowed once the file exists.
• Near the end, it removed four lines that it took for duplicates. One of them was the opening brace of an if block, which the code needed. The extra brace was still there. The game did not compile when the work ended at the limit of 50 answers.

Conclusion

Qwen 3.8 27B passed every hidden test in both modes and needed the fewest tool calls; Qwen 3.6 35B in Thinking mode also passed every test.

• Qwen 3.6 27B: all but one hidden test in each mode: a wind change one step too large in Instruct mode, and a shell that started to fly one frame late in Thinking mode.
• Qwen 3.6 35B: all hidden tests passed in Thinking mode, although the model never handed the game in. In Instruct mode, the game never compiled.
• Qwen 3.8 27B: all hidden tests passed in both modes, with 15 tool calls in each.

For a turn-based game with exact physics like this one, Qwen 3.8 27B in NVFP4 was the most reliable setup in this test. With Qwen 3.6 27B, check the rules with limits and the timing of the first frame with tests.

References

Benchmark runs with NVFP4 quantization