Introduction

Can a local model build a racing game with computer opponents on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a top-down racing game like Super Speed as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 17 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a race of four cars on a small track seen from above. The player steers a car with the arrow keys around a loop of road with walls, grass and four checkpoints. Three computer cars drive the same loop. The player collects wrenches on the road, and after three laps the race ends and the shop opens. In the shop the player buys upgrades for acceleration, top speed and turning, and then starts the next race.

The exact rules the hidden tests check include the speed changes for each key, the movement in the car's direction, the slower top speed on grass, sliding along a wall, the order of checkpoints for a lap, the place in the race, the prizes and the prices in the shop. The computer cars have only general rules: they stay on the road, keep a distance from other cars, point in the direction they move and finish a lap in time. The model designs how they drive.

Results

Qwen 3.8 27B in Think: medium passed every result, Qwen 3.8 27B in Instruct mode and Qwen 3.6 27B in Thinking mode lost one each, Qwen 3.6 27B in Instruct mode lost four, and Qwen 3.6 35B built no game that works. The chart below shows the score of each setup: the share of passed results, the build of the game and the 17 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a playable game in both modes, but in both modes the computer cars failed. In Thinking mode, everything except the computer cars worked: 16 of the 17 hidden tests passed. The computer cars drove a few pixels and then stopped at a wall for the rest of the race (see below).

In Instruct mode, the computer cars drove across walls and grass in straight lines. The game also counted a lap one checkpoint too early, so the race did not end where the task says. Its own tests found this lap mistake, but the model kept sending an old edit again and used up the limit of model calls before it fixed it. Three more hidden tests failed because of it.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not build a working game in either mode. In Instruct mode, an edit broke the first line of the track data, the game did not compile, and the model then stopped calling tools. In Thinking mode, the game compiled, but the race stayed frozen for the first six seconds while it showed the controls. Most hidden tests check the first frames of a race, so 15 of them failed. Both problems are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium and all but one in Instruct mode. In Think: medium, the model built the game in a few large steps, found mistakes in its own tests and was the only setup with a test that lets the computer cars drive a full lap. It needed the fewest checks and edits of the setups with a working game.

In Instruct mode, the driving, the laps, the computer cars and the shop all worked. Only the drawing of the cars was wrong: each car showed only part of its rectangle (see below). The model used up the limit of model calls, and its last version of the game was graded.

Both Qwen 3.8 27B setups made computer cars that finish their laps, and neither Qwen 3.6 27B setup did. This was the main difference in this test. Both models ran in NVFP4 with the same settings. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

Only one setup tested the computer cars over a full lap, and it was the only one that passed every hidden test. Qwen 3.6 27B in Thinking mode wrote the most tests, 17, all passing, but none of them let the computer cars drive. Qwen 3.6 27B in Instruct mode had one failing test at the end, the lap test that found its real mistake. Qwen 3.6 35B in Instruct mode is not in the chart: its game does not compile.

Checks and edits

Qwen 3.8 27B in Think: medium needed only 4 checks and 10 edits for a game that passed every hidden test. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.6 27B in Thinking mode made the most edits, 31. Qwen 3.8 27B in Instruct mode and Qwen 3.6 27B in Instruct mode both ran about 20 checks and used up the limit of model calls before they submitted the game.

Unwanted tool calls

Qwen 3.6 27B in Instruct mode spent 10 tool calls on edits that were refused because they repeated an earlier edit. It sent the same old edit again and again near the end of the run. Qwen 3.8 27B in Instruct mode had 9 rejected calls: seven referred to text that the answer did not contain, one asked for a tool that does not exist, and one had broken call syntax. Qwen 3.6 35B had a few rejected calls in both modes, including a refused rewrite of the whole file. Qwen 3.8 27B in Think: medium had one rejected call, and Qwen 3.6 27B in Thinking mode none.

Tokens

Every tool call sends the whole conversation to the model again, so long runs read far more tokens. Four setups read more than two million tokens. Qwen 3.8 27B in Think: medium read less than half of that, because it needed few tool calls. Thinking mode writes more tokens because the model reasons before each step: Qwen 3.8 27B in Think: medium wrote the most, Qwen 3.8 27B in Instruct mode the fewest of the setups with a working game.

One run step by step

Qwen 3.8 27B in Think: medium planned the game, built it in a few large steps and fixed its own tests before the game.

1. It wrote a plan and a first file: the variables, the track built from the map in the task and the starting values.
2. It added the player's car: steering, speed, walls, grass, wrenches and checkpoints. Its next answer described an edit as text instead of calling a tool, and it needed one reminder.
3. The reminder showed compile errors: SC# does not join text and numbers with +. It switched to text with inserted values.
4. It added the computer cars, the place in the race, the end of the race and the drawing. Then it rewrote the route of the computer cars so that it follows the middle of the road.
5. The check found two variables declared in one line, which SC# does not allow. It fixed them.
6. Four of its nine tests failed. The model found that four of them expected wrong values and one used a wrong tile, and it fixed the tests.
7. Five of its 14 tests still failed. The model found the reason: the tests called the game's functions directly, so the held keys had no effect. It changed the tests to run whole frames.
8. All 14 tests passed, it submitted the game, and all 17 hidden tests passed.

How the models fail

The lap counts one checkpoint too early

The game of Qwen 3.6 27B in Instruct mode counts a lap when the car passes the last checkpoint, not the start line. The task says that a lap grows when the car reaches the start line after checkpoints 1, 2 and 3. The model's code first moves to the next checkpoint and then checks the new value:

In a normal race the number of laps is the same at most moments, so the hidden test for laps passed. But the race ends a quarter of a lap early, and three hidden tests for the end of the race, the shop and the next race failed. The model's own lap test found the problem: it expected one lap and got none. The model did not fix the code. It kept sending an earlier edit that had already been applied, and these edits were refused until the run reached its limit.

Computer cars that stop at a wall

In the game of Qwen 3.6 27B in Thinking mode, all three computer cars stop after a short drive and never finish a lap. The model gave the cars a list of points around the track and let each car aim at the point two steps ahead of the nearest one. From the start, that point is on the other side of the inner wall:

// Target is 2 waypoints ahead
int targetIdx = (bestIdx + 2) % wpX.Count;
...
int ntt = TileType(newX, newY);
if (ntt == 0 || ntt == 3)
{
    // move only when the next position is road
}

The cars turn toward the target, the next position is a wall, and the car does not move. Nothing makes it turn away, so it waits at the same place for the whole race. The hidden test let the cars drive 1,000 frames and none of them finished a lap. In Instruct mode, Qwen 3.6 27B made the opposite mistake: its cars drove straight toward each checkpoint, across walls and grass, and left the road in the 14th frame.

A car with half of its body missing

Qwen 3.8 27B in Instruct mode drew each car with four triangles around the center, but two of them have no area. The task asks for a 12 x 8 rectangle turned to the heading and drawn as two triangles. The model computed the four corners correctly, but joined them in the wrong order:

The player's car had 42 red pixels instead of about 96, so the hidden test for the screen failed. All other hidden tests passed. None of the model's ten tests looked at the screen.

A frozen start and a broken first line

Qwen 3.6 35B lost both runs to mistakes outside the driving rules.

• Thinking mode: part B of the task asks to show the controls during the first seconds of a race. The model showed them, but it also stopped the whole race for 180 frames, six seconds. In this time no key had an effect and no car moved. The hidden tests for speed, steering, walls, wrenches, laps and the shop check the first frames of a race, so they all failed. The computer cars also failed after the start: their direction did not match their movement. The code had one more problem that the tests did not reach: it turned the car in a while loop that runs as long as the key is held, so the first turn at speed would freeze the game.
• Instruct mode: the model wrote a short first version that compiled. Its rewrite of the whole file was refused. Then it replaced the first line of the track list with a comment, so the list had no start and the game no longer compiled. In its next three answers, the model described this problem correctly but did not call a tool to fix it, and the run stopped.

Conclusion

Qwen 3.8 27B in Think: medium passed every result, and Qwen 3.8 27B in Instruct mode and Qwen 3.6 27B in Thinking mode lost one hidden test each.

• Qwen 3.8 27B: all hidden tests passed in Think: medium. In Instruct mode, only the drawing of the cars was wrong.
• Qwen 3.6 27B: the computer cars failed in both modes: they stopped at a wall in Thinking mode and drove across walls and grass in Instruct mode. Instruct mode also counted laps at the wrong checkpoint.
• Qwen 3.6 35B: no working game, because of a frozen start in Thinking mode and a broken edit in Instruct mode.

For a game where the model must design behavior itself, like computer opponents, Qwen 3.8 27B in Think: medium in NVFP4 was the safer choice in this test. With any model, ask for a test that runs the designed behavior for a long time.

References

Benchmark runs with NVFP4 quantization