Introduction

Can a local model build a toy car racing game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a toy car racing game like Micro Machines as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 13 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a top-down race of tiny cars on a track that is bigger than the screen. The player drives one car with the arrow keys, and three computer cars race on the same track. The camera follows the player's car. The car slides in curves, more on sand than on the road, and it bounces off barriers. The player can push computer cars. The race has three laps, and a lap counts only when the car passes three checkpoints in order and then the start line.

The exact rules the hidden tests check include the track as a map of 40 x 30 characters, the steps of every frame in a fixed order (steering, throttle, grip, speed limit, barriers, pushing, checkpoints), the position in the race, the camera and the drawing of the track and the cars. The computer cars have rules too: they must stay on the road, move at most 3 pixels per frame, keep a distance of 12 pixels from every other car and reach the first checkpoint within 400 frames. How they drive is the model's own design.

Results

Qwen 3.8 27B in Think: medium passed the most results, 12 of 14. Qwen 3.8 27B and Qwen 3.6 27B in Instruct mode passed 11, Qwen 3.6 35B in Instruct mode 6, and both remaining thinking setups none. No setup passed every result: the test of the computer cars failed in every game. The chart shows the score of each setup: the share of passed results, the build of the game and the 13 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a good game in Instruct mode, but no working game in Thinking mode. In Instruct mode, 10 of 13 hidden tests passed. The physics, the barriers, the camera and the drawing were correct. Three rules failed: the computer cars stopped at the first sand tile, a pushed computer car still drove in the same frame, and a lap counted one checkpoint too early (see below). The model wrote 27 tests, more than any other setup, and all of them passed. They tested its own reading of the rules, so the wrong lap rule passed its own test too.

In Thinking mode, the model copied the track map with two wrong rows. The game stopped with an error at the start, so every hidden test failed. The model tried to find the wrong row by counting characters in its reasoning, looked at the wrong row and repeated the same sentence until the answer was stopped as broken output. In this test series, a broken answer ended the work.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B passed 6 of 14 results in Instruct mode and none in Thinking mode. In Instruct mode, it submitted a game with many separate errors. One row of the track map was missing, so the bottom part of the track moved up by one row. A pushed computer car still drove, the computer cars left the road, the position in the race was wrong, and the race did not end after three laps. Steering, throttle, grip, the speed limit and the camera were correct. The model wrote only 3 tests, and at the end it ran the same check nine times without a change.

In Thinking mode, the model wrote a short first version that did not compile. Then two tool calls had broken call syntax, and its next answer planned the same steps again and again ("Let me start now. ... Actually, I realize ...") until it was stopped as broken output.

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B built a good game in both modes: 12 of 13 hidden tests passed in Think: medium and 10 in Instruct mode. In Think: medium, only the computer cars failed: one of them got stuck at an obstacle. The model's own test found this problem, and the model tried to fix it several times. In its last answer, it counted the characters of one map row one by one and started to repeat the letter R until the answer was stopped as broken output. The game was not submitted, so the build result counts as failed. The hidden tests ran on the last version of the file.

In Instruct mode, the game was not submitted either. The model spent the second half of its run on one of its own tests, which expected the start line in the wrong column of the map. Its game was right, but it did not see that the test was wrong, and the run reached its limit of 50 rounds. In the last version, the computer cars drove onto the sand, a pushed computer car still drove, and the cars were drawn as diamonds instead of rectangles.

Qwen 3.8 27B did best in this test, and Qwen 3.6 27B in Instruct mode was close behind. Both models ran in NVFP4 with the same settings. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests did not decide the result, but a good test helped. Qwen 3.6 27B in Instruct mode wrote 27 tests, and all of them passed, also the one for its wrong lap rule. The test of Qwen 3.8 27B in Think: medium for the computer cars found the same error that the hidden tests found later. Qwen 3.8 27B in Instruct mode lost its run to a wrong test. Qwen 3.6 27B in Thinking mode wrote 4 tests, and all of them failed because the game stopped at the start. Qwen 3.6 35B in Thinking mode is not in the chart: its file did not compile.

Checks and edits

Many checks did not mean much progress. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.6 35B in Instruct mode ran 21 checks, and nine of them came at the end, one after another, without a change. Qwen 3.6 27B in Instruct mode did the same: after all its tests passed, it ran ten checks with the same answer before it submitted the game. Qwen 3.8 27B in Instruct mode made the most edits, most of them on its wrong test. The two thinking setups of Qwen 3.6 stopped early, after a broken answer.

Unwanted tool calls

Qwen 3.8 27B in Instruct mode spent 11 tool calls on nothing. Five of them repeated an earlier edit after the file had changed, so they were refused. Three asked for a tool that does not exist, two referred to text that the answer did not contain, and one had broken call syntax. Qwen 3.8 27B in Think: medium had two calls that referred to missing text, and Qwen 3.6 35B in Thinking mode two calls with broken call syntax. The other setups had no unwanted calls.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls read far more tokens. Qwen 3.6 35B in Instruct mode read over three million tokens, most of them because of its many checks. The runs that stopped early read the fewest. Qwen 3.8 27B in Think: medium wrote the most tokens because it reasoned before each step. Qwen 3.6 35B in Thinking mode wrote over 160,000 tokens in only four answers, and the last one repeated its plan.

One run step by step

Qwen 3.8 27B in Think: medium built the game in parts and found its last problem with its own test.

1. It started with a skeleton: the variables, the track, a new race, an empty game loop and two tests. Both tests passed.
2. It added the whole physics of the player's car in one edit, then the computer cars with a route of points around the track, the position and the end of the race. The check found a missing brace in the game loop, and the model fixed it.
3. It moved the route away from the two obstacles and wrote the drawing. Some of its own edits left code outside functions and a test without its opening brace. It found and fixed these problems itself.
4. It added more tests. Some of them put the car on the wrong tile, one of them inside a wall, and the model fixed these tests.
5. Its test for the computer cars failed: one car did not reach the first checkpoint in time. The model gave each car a different starting point on the route and changed their speeds. Now 17 of its 18 tests passed, but this test still failed.
6. While it planned another fix, it started to count the characters of a map row one by one and then repeated the letter R. The answer was stopped as broken output, and the run ended. The hidden tests found the same problem as its own test: one computer car got stuck at an obstacle. All 12 other hidden tests passed.

How the models fail

The computer cars do not get around the track

The test of the computer cars failed in all four games that compiled. The task leaves the driving to the model, but the cars must stay on the road and reach the first checkpoint within 400 frames. The track is a ring around a large wall block, with strips of sand along the inner edge. Qwen 3.6 27B and Qwen 3.8 27B in Instruct mode both drove the cars in a straight line to the center of the next checkpoint:

Qwen 3.8 27B in Think: medium used a route of points along the road, which is a better plan. But its route passed right next to the obstacle in the bottom road. One car drove up to the obstacle and could not get past it, and a second car waited behind it, because the cars must keep 12 pixels apart. In the game of Qwen 3.6 35B in Instruct mode, the cars left the road too.

A pushed car still drives

Three of the four games let a pushed computer car drive in the same frame. The task says that when the player's car runs into a computer car, that car is pushed, and "a pushed computer car does not drive in step 8 of this frame". Only Qwen 3.8 27B in Think: medium kept this rule:

A lap counts one checkpoint too early

The game of Qwen 3.6 27B in Instruct mode counts a lap when the car passes the third checkpoint, not the start line. The task says: "nextCheckpoint becomes (nextCheckpoint + 1) % 4, and when that checkpoint was 0 (the start line), laps grows by 1." The model checked the new value instead of the passed one:

In a normal race, the count is the same, only one quarter of the track earlier. But the hidden test starts the last lap just before the start line, and in this game the race did not end there. The model's own test used its own rule, so it passed.

A map copied with mistakes

Two models copied the track map with mistakes. The map has 30 rows of 40 characters, and many rows look the same. A third model, Qwen 3.8 27B in Think: medium, lost its run while it counted the characters of a correct row in its reasoning.

• Qwen 3.6 27B in Thinking mode wrote one row with 39 characters and the next row with 42. The game stopped at the start with the message "String index 39 is out of range". The model then counted the characters of another row in its reasoning, decided that this correct row had 41 characters and wrote it again without a change. In its next answer, it repeated "I'm recounting the R's in the second row" until the answer was stopped.
• Qwen 3.6 35B in Instruct mode left out one of the many equal rows. Its game read only 28 rows and filled the rest of the world with walls. This moved the bottom part of the track up by one row, together with the start line and an obstacle, and cost it two hidden tests.

Conclusion

Qwen 3.8 27B did best in this test, Qwen 3.6 27B in Instruct mode was close behind, and no setup passed every result.

• Qwen 3.8 27B: 12 of 14 results in Think: medium and 11 in Instruct mode. Neither run submitted the game: one ended with a broken answer, the other at the round limit.
• Qwen 3.6 27B: 11 of 14 results in Instruct mode, none in Thinking mode because of a broken track map.
• Qwen 3.6 35B: 6 of 14 results in Instruct mode with many separate errors, none in Thinking mode.

The rule that no model got right was the driving of the computer cars. For a game like this, Qwen 3.8 27B or Qwen 3.6 27B in Instruct mode in NVFP4 is a good start, but check the computer players with a test that drives a full lap.

References

Benchmark runs with NVFP4 quantization