Introduction

Can a local model build a top-down shooter on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a top-down shooter in the style of Counter-Strike 2D as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 17 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a round-based shooter seen from above. The player walks through a map of walls and crates with the arrow keys, turns with Z and X, shoots with Space and reloads with R. Three bots patrol the map and shoot back when they see the player. Health and armor pickups lie on the map. A round ends when the player has no health or no bot is left. Between rounds, a shop sells armor and a rifle for the money the player earned.

The exact rules the hidden tests check include the map, the start of a round, walking with sliding along walls, turning, firing and reloading with two weapons, how bullets fly and where walls and crates stop them, hits on bots and on the player with armor, the line of sight, the speed and the reaction time of the bots, the pickups, winning and losing a round, the shop and the drawing. The task also fixes the order of the nine steps of every frame. How the bots patrol and when exactly they shoot within the given limits is the model's design.

Results

Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in Instruct mode passed every result. The other 27B setups lost three or four, Qwen 3.6 35B in Thinking mode lost six, and Qwen 3.6 35B in Instruct mode built no game. The chart shows the score of each setup: the share of passed results, the build of the game and the 17 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B passed every hidden test in Instruct mode and lost three in Thinking mode. In Instruct mode, the model wrote a game of 1,323 lines with 42 tests in its first step. Ten of its tests failed at first. While it looked into a failing test of bullets and walls, the model noticed that a new bullet moved in the frame in which it was fired, and it changed the game so that only older bullets move.

In Thinking mode, the model built a complete game with 15 tests, but it moved the new bullet in the frame in which it was fired. Two hidden tests failed because of this (see below). A third test failed because a bot could move faster than the limit of 1.5 pixels per frame when it walked diagonally.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B built a working game in Thinking mode, but not in Instruct mode. In Instruct mode, the model wrote the code three times as text in its answer and never created the game file, even after two reminders. The run stopped after three answers without a tool call.

In Thinking mode, the model wrote a game of 667 lines in its first step. Then it spent most of its tool calls on one compile error and on calls that the runtime rejected. When the game finally compiled, the model had only 4 tests, and it submitted the game soon after. Six hidden tests failed: the new bullet moved too early, bots fired in every frame for a while instead of every 20 to 40 frames, a pickup disappeared when it did not help the player, the player was never drawn, and after a won round with a rifle the magazine was refilled as a pistol magazine.

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium and lost four in Instruct mode. In Think: medium, the model started with a skeleton of 95 lines and grew it in steps. Before it ran the full test suite, it found two of its own mistakes by reading the code: a missing clear of the screen and a bot list that it did not clean up when a bot died.

In Instruct mode, the game moved the new bullet too early, like two other setups. The more serious problem was that a bot with no health was never removed, so a round could not be won. The model's own test found this, but the model decided again and again that its test was wrong. It used all 50 answers on this and never submitted the game (see below).

Each 27B model passed every hidden test in one mode and lost tests in the other. Qwen 3.8 27B did best in Think: medium, Qwen 3.6 27B in Instruct mode. With one run each, this says more about the single run than about the mode.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The two setups that passed every hidden test wrote the most tests. Qwen 3.6 27B in Instruct mode wrote 42 tests and Qwen 3.8 27B in Think: medium 37. Qwen 3.8 27B in Think: medium had a test that checked where a new bullet is right after the frame in which it was fired. Qwen 3.6 35B in Thinking mode had only 4 tests, all of them about the start state. Qwen 3.6 35B in Instruct mode is not in the chart: it created no file.

Checks and edits

Qwen 3.8 27B in Instruct mode made 30 edits and still did not submit a game. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Two thirds of its edits came after one of its tests started to fail, and almost all of them changed that test. Qwen 3.6 35B in Thinking mode ran 15 checks: 11 of them ended with a compile error, and three ran again on an unchanged file before it submitted.

Unwanted tool calls

Qwen 3.6 35B in Thinking mode spent 15 tool calls on nothing. Eleven of them referred to text that the answer did not contain. Four tried to rewrite the whole file, which is not allowed once the file exists. Qwen 3.8 27B in Instruct mode had five rejected calls and Qwen 3.6 27B in Instruct mode also five: refused repeated edits, references to missing text and, once, an empty tool call. Qwen 3.6 27B in Thinking mode and Qwen 3.8 27B in Think: medium made no wrong calls. Qwen 3.6 35B in Instruct mode made no calls at all.

Tokens

Every tool call sends the whole conversation to the model again, so long runs read far more tokens. Qwen 3.8 27B in Instruct mode read over three million tokens because it used all 50 answers on a game of more than 800 lines. Qwen 3.6 35B in Thinking mode read over two million tokens and wrote the most, because it reasoned at length before many of its calls. Qwen 3.6 35B in Instruct mode read the fewest tokens: its run ended after three answers.

One run step by step

Qwen 3.8 27B in Think: medium built the game in small steps and needed 21 tool calls.

1. It started with a skeleton of 95 lines: the variables, the start of a game and two tests. Both tests passed.
2. It added the map, the checks for walls and crates and the line of sight, then the player's movement, turning, reloading and firing, and the bullets.
3. It added the bots, the pickups, the end of a round, the timers and the shop, and ran a check. The game compiled.
4. It added the drawing and the extra features. Reading its own code, it noticed that it had dropped the clear of the screen at the start of each frame, and it restored it.
5. Before the next check, it found that a dead bot stayed in some of its own helper lists. It fixed this and wrote 35 more tests, 37 in total.
6. The check found two compile errors: two variables declared in one line, which SC# does not allow. It fixed both.
7. Eight of its tests failed, and in all of them the test was wrong: some pressed keys outside a frame, others expected a wrong tile center. The model fixed the tests and checked twice more until all 37 passed.
8. It submitted the game, and all 17 hidden tests passed.

How the models fail

The new bullet moves in the frame it is fired

Three setups moved a new bullet in the same frame in which it was fired. The task fixes the order of each frame: step 4 fires a bullet, and step 5 moves "every bullet that existed before this frame". A bullet fired in step 4 did not exist before this frame, so it must stay where it was created. Qwen 3.6 27B in Thinking mode counted the bullets after firing; Qwen 3.8 27B in Think: medium counted them before:

The hidden test for firing found the bullet 16 pixels in front of the player instead of 8. A second test failed for the same reason: the bullet reached the right wall one frame early and was removed. Qwen 3.6 35B in Thinking mode and Qwen 3.8 27B in Instruct mode had the same mistake; they moved every bullet in the list. None of these three games had an own test that looked at a bullet's position.

A failing test that was right

Qwen 3.8 27B in Instruct mode never removed a bot with no health, and its own test showed it. The task says: "a bot with botHealth <= 0 is removed". The model's code lowers the bot's health, but no line of the game removes the bot:

if (dx * dx + dy * dy < 49.0)
{
    botHealth[j] = botHealth[j] - Damage(weapon);
    hit = true;
    break;
}

Without the removal, the list of bots never becomes empty, so the player cannot win a round. Two hidden tests failed: the bot that should be removed stayed, and the round did not end with a win.

The model wrote a test for exactly this, and from its 16th tool call on, this test failed in every check: one bot was still there. The model's answer each time was that the test was wrong, not the game: "The remaining failure is in my own test, not the game logic." It moved the bot in the test, closed it in with walls, changed the angle and the number of frames, and once blamed the bots' random movement. Almost all of its 20 edits after that changed this test, and none added the removal to the game. The run ended at the limit of 50 answers, and the last version of the file was graded.

The bot walks too fast on a diagonal

The bots of Qwen 3.6 27B in Thinking mode could move up to 2.1 pixels per frame. The task says that a bot moves at most 1.5 pixels per frame. The model limited each direction on its own:

double moveX = patDir * 1.5;
double moveY = Math.Cos(i * 2.3 + roundStartFrame * 0.07) * 1.5;

When both parts are close to 1.5, the step along the diagonal is longer than 1.5 pixels. The hidden test measured a step of 1.555 pixels in the third frame. A limit on the length of the whole step, not on each part, keeps the bot within the rule.

Many small gaps and no game at all

Qwen 3.6 35B lost six hidden tests in Thinking mode and built no game in Instruct mode.

• Thinking mode: besides the early bullet, four rules were wrong. The code for pickups was turned around: it gave the bonus without removing the pickup, and removed a pickup only when it did not help. A bot shot in every frame from the 10th to the 30th frame in which it saw the player, instead of once and then every 20 to 40 frames. The player was never drawn. A new round always refilled the magazine of the pistol, so after a won round with the rifle the player had 12 bullets instead of 30. The game had 4 tests, and none of them covered these rules.
• Instruct mode: the model wrote the game as text in its answer three times and never created the file. After each answer it got a reminder to create the file with a tool. The run stopped after three answers in a row without a tool call, so no game existed.

Conclusion

Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in Instruct mode passed every result. The other two 27B setups were close.

• Qwen 3.6 27B: all hidden tests passed in Instruct mode. In Thinking mode, the early bullet and a too fast bot cost three tests.
• Qwen 3.8 27B: all hidden tests passed in Think: medium. In Instruct mode, the game never removed a dead bot, and the model blamed its own correct test until it ran out of answers.
• Qwen 3.6 35B: a game with six failed hidden tests in Thinking mode, and no game in Instruct mode.

For a game with this many exact rules, the 27B models in NVFP4 can build a complete version in one attempt. The setups that passed every test wrote the most tests, and one of them checked a single frame after firing. In our runs, the mode that worked best was different for each model, so one run is not enough to pick a mode for this kind of task.

References

Benchmark runs with NVFP4 quantization