Introduction

Can a local model build a small game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a brick breaker game like Breakout as an app in SC#, our C#-like scripting language. The model does not answer once. It works like an agent: it writes the app into a file, compiles it, runs its own tests, reads the results, fixes the code and submits the app when it is done. Hidden tests, which the model never sees, then check whether the game follows the rules.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium, shown as "Think: medium" in the charts.
• Quantization: this article focuses on NVFP4, a 4-bit format that needs about half the memory of FP8, because it matters more for local users. Qwen 3.6 27B also ran in FP8 for comparison.
• Runtime: vLLM 0.30.0 with matching settings for all models, on our test server.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

How the test works

The process

1. Task. The model gets about 20,000 tokens of context: the task, a reference of the SC# language, a guide to SC# apps with a small example game, and four tools: create the game file, change lines of it, check it and submit it.
2. Code and own tests. The model writes the game and its own tests into one file. The tests press keys frame by frame and check the game state and the pixels the game drew.
3. Check and fix. A check compiles the file and runs the model's tests, and the model gets back the errors and the test results. A submission is rejected while the game does not compile or one of its own tests fails.
4. Hidden tests. After the submission, our benchmark service runs 19 hidden tests on the game. The model never sees them.
5. Game in the browser. The service translates the game from SC# to JavaScript for a web page. The tests the model wrote for itself stay out.

The work also ends when the model answers three times in a row without a tool call, or after 50 requests. Then we grade the last version of the file. The model never runs or watches its game: it knows only what the compiler and its own tests tell it.

The task

• Part A has the rules that the hidden tests check. It fixes what the tests need to read the game: the screen size, the names of the variables that hold the game state, the positions of the bricks and the colors of the objects. The behavior is described, but the model designs the mechanics. For example, a hit near the end of the paddle must send the ball more to the side than a hit near the middle, and the model chooses how.
• Part B lists features without exact rules: the controls shown on the screen, a short effect when a brick breaks, a bar that shows how long a bonus lasts, and a pause key. The hidden tests do not check part B. You see it when you play the game.

How we grade

The hidden tests decide the score. They check the game state and the screen, for example where the paddle is drawn or whether "GAME OVER" is shown. Where the model designs the mechanics, they check properties instead of exact values: they hit the paddle at many points from one end to the other and check that the further from the middle the ball hits, the more it goes to the side. Two reference games of our own, with different mechanics, pass all hidden tests, so a careful person can pass them from the task alone.

The score of a run is the share of passed results: the build of the game and the 19 hidden tests. If there is no game or the code does not compile, all hidden tests fail.

Results

Qwen 3.8 27B passed every result in both modes, Qwen 3.6 27B lost one hidden test in Thinking mode, and Qwen 3.6 35B lost the same three bonus tests in both modes. The chart below shows the score of each setup: the share of passed results, the build of the game and the 19 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a working game in both modes: all hidden tests passed in Instruct mode, and all but one in Thinking mode. In Thinking mode, the game fails when the ball hits the corner of the paddle (see below). The model submitted it after two checks, but then kept calling the submit tool.

In Instruct mode, the model made more edits than any other setup: more than 30 small changes until all its own tests passed. Before it could submit the game, the work stopped because of an error on our side: the request to the model grew larger than the size limit of the web server in front of our runtime. We graded the last version of the file, as for every game that was not submitted, and it passed all hidden tests.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B built a playable game in both modes, but both games failed the same three hidden tests. In both games, the falling bonuses stop while the ball rests on the paddle (see below). Everything else worked: the paddle, the bounces, the bricks, the lives and the screen.

In Thinking mode, the model wrote more than 30 tests, checked the game six times and changed it in many small edits before it submitted it. In Instruct mode, it wrote the whole game at once and checked it once. Then it broke the tool-call format eight times in a row, and the runtime stopped its next answer as broken output (see below). The game was not submitted, so we graded the last version of the file, and the build counts as broken.

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in both modes. It handled both traps of this task: a ball that hits the corner of the paddle, and bonuses that keep falling while the ball rests on the paddle.

It also needed the fewest steps. In Thinking mode, it wrote the whole game with its tests in one go, all its tests passed at the first check, and it submitted the game right away. In Instruct mode, it wrote fewer tests and needed a few small fixes before it submitted.

Qwen 3.8 27B did a little better than Qwen 3.6 27B and needed fewer steps. Qwen 3.6 27B failed the corner test in Thinking mode, in both FP8 and NVFP4, and Qwen 3.8 27B did not. Both models ran in NVFP4 with the same settings, so the difference points to the model. With one run each, it is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests. But the numbers show how differently the models work.

Own tests

The number of tests the models wrote for their games did not predict the result. Qwen 3.8 27B in Instruct mode wrote the fewest tests and passed every hidden test. Qwen 3.6 35B in Thinking mode wrote the most tests, but none of them checked the bonuses while the ball rests on the paddle.

Checks and edits

Qwen 3.6 27B in Instruct mode needed by far the most edits, and Qwen 3.8 27B in Thinking mode needed none. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.8 27B in Thinking mode wrote the game once, checked it once and submitted it. Qwen 3.6 35B in Instruct mode made no edits either, but only because its work stopped after the first check. Qwen 3.6 27B in Instruct mode fixed its game in many small steps.

Unwanted tool calls

Two runs spent most of their tool calls on nothing. Qwen 3.6 27B in Thinking mode repeated the submission after the game was final, and Qwen 3.6 35B in Instruct mode sent tool calls in a wrong format, which were rejected. The other runs had at most a few such calls. Both problems are described in "How the models fail".

Tokens

Every tool call makes the model read the whole conversation again, so runs with many tool calls read far more tokens. The three runs with the most tool calls read well over a million tokens each. For Qwen 3.6 27B in Thinking mode, most of them came from the repeated submission. Thinking mode writes more tokens because it reasons before each step. Qwen 3.8 27B in Thinking mode read the fewest tokens but wrote the most, because it wrote the whole game with its tests in one answer after long reasoning.

Example: one run step by step

A typical run takes a few rounds of checks and fixes. The run of Qwen 3.6 35B in Thinking mode shows the whole loop:

1. The model wrote the whole game with 31 tests into the file.
2. The first check failed to compile: the model declared two variables in one line, double ballX, ballY, and SC# does not allow that for script variables. It fixed the first line, the next check found the same mistake in another line, and it fixed that too.
3. Now 24 of its 31 tests passed. One failure: the game did not set won when all bricks were gone while the ball rested on the paddle.
4. After eight small edits, 28 tests passed, and after two more, all 31.
5. The model submitted the game.

All its tests passed, but three hidden tests failed. Its tests checked the bonuses only while the ball was in play, and the game stops the bonuses while the ball rests on the paddle.

How the models fail

Bonuses stop while the ball rests

Both games of Qwen 3.6 35B move the falling bonuses only while the ball is in play. The task lists the steps of a frame, and the step "every bonus falls" has no condition. But the model put the bonus code inside the branch for a launched ball, so after the player loses a ball, the bonuses stop in the air until the next launch:

The model's own tests checked the bonuses only while the ball was in play, so they did not find the mistake. Three hidden tests failed: a bonus falls, a Life bonus is caught and a Wide bonus is caught.

A corner hit breaks the speed limit

The task limits the sideways speed after a paddle bounce to 5 pixels per frame, and the rule of Qwen 3.6 27B broke it at the corners. The paddle is 48 pixels wide, so its end is 24 pixels from the middle. But the ball has a radius of 4, so it still bounces when its center is up to 28 pixels from the middle:

Qwen 3.6 27B made this mistake in Thinking mode in both FP8 and NVFP4, and in Instruct mode in FP8. Qwen 3.6 35B, Qwen 3.8 27B and our reference games limit the value.

The model changes the game to fit its own tests

A passing run of the model's own tests does not mean the game is correct. In a test run before the final runs, Qwen 3.6 27B in FP8 Instruct mode wrote a test that hit the paddle a few pixels next to the middle and expected the ball to go straight up. When the test failed, the model did not fix the test. It changed the game:

double innerZone = 6; // pixels from center with no deflection
if (absOffset <= innerZone)
{
    ballVX = 0;
}

Now every hit up to 6 pixels from the middle sends the ball straight up, but the task says that a hit right of the middle sends the ball to the right. In the final runs, the same model fixed such tests instead of the game.

Calls after the submission

After Qwen 3.6 27B in Thinking mode submitted its game, it called the submit tool 27 more times. Every answer said that the app was final and that no more tools were needed, but the model repeated the same call until it finally wrote a short note. The calls changed nothing, because the game was already final. But each call sent the whole conversation to the model again, so this run read more than a million tokens with only four edits. Qwen 3.6 35B in Thinking mode did the same twice.

We changed our test service after these runs: now the work ends right after an accepted submission.

Invalid tool calls

Qwen 3.6 35B in Instruct mode understood what to fix but could not call the tools. After its first check, it wanted to change a few functions. Eight times in a row, it put the code straight into the tool call in a format of its own instead of the format from the tool rules. Each call was rejected with an example of the correct format. In its next answer, the model wrote the tool rules themselves, and the runtime stopped that answer as broken output. The work ended there, and the game from the last file passed 16 of the 19 hidden tests. In earlier test runs, Qwen 3.6 35B in Instruct mode made the same kind of mistake.

Conclusion

Qwen 3.8 27B did best in this test, Qwen 3.6 27B came second and Qwen 3.6 35B third. All three models built a playable brick breaker game as an agent in NVFP4.

• Qwen 3.8 27B: all hidden tests passed in both modes, with the fewest steps.
• Qwen 3.6 27B: all hidden tests passed in Instruct mode, one failed in Thinking mode. NVFP4 did as well as FP8.
• Qwen 3.6 35B: playable games, but the same logic mistake with the bonuses in both modes, and in Instruct mode invalid tool calls ended the work.

References

The runs belong to the test plan "Brick breaker in SC#: building a game as an agent".

Benchmark runs with NVFP4 quantization

Benchmark runs with FP8 quantization