Introduction

Can a local model build a scrolling platform game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a Mario-like platformer as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 14 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a side-scrolling platformer on a level of 60 x 15 tiles. The player walks with the arrow keys, jumps, and the camera follows to the right. The level has ground with two pits, pipes, bricks, question blocks with a coin or a mushroom, coins, enemies that walk back and forth, and a flag at the end. A big player breaks bricks, a small one only bumps them. Stomping an enemy removes it; touching it from the side costs the mushroom or a life.

The exact rules the hidden tests check include the level given as a map of characters, the order of the steps in a frame, the jump and the gravity, how the player is pushed out of solid tiles, what a bumped block does, the camera, the enemies' turn at walls and edges, the mushroom's movement, stomping, hits, lives and the drawing. The model chooses how to build all of this, and it designs the speed of the enemies and the mushroom within given limits.

Results

Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in Thinking mode lost one result each, Qwen 3.6 27B in Instruct mode lost two, Qwen 3.8 27B in Instruct mode lost eight, and Qwen 3.6 35B built no working game. The chart below shows the score of each setup: the share of passed results, the build of the game and the 14 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a working game in both modes: 13 of 14 hidden tests passed in Thinking mode and 12 in Instruct mode. Both modes made mistakes in the rule that turns an enemy around at the edge of a pit (see below). In Instruct mode, the mushroom also never landed on the ground.

In Instruct mode, the model wrote a game of 716 lines at once. Near the end, it repeated one edit 19 times in a row, all refused, until the run reached the limit of 50 answers. The game was graded as it was at that moment, with 4 of its own 29 tests failing. In Thinking mode, the model stopped three times without a tool call and needed reminders, and its first submission was rejected because five of its own tests failed. It fixed them and submitted the game.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not build a working game in either mode. Both runs ended in the first answer. In Instruct mode, the model started to write the level as a list of 900 numbers and repeated "0, " until the runtime stopped the answer as broken output. In Thinking mode, the first answer used the whole output limit on planning and drafts of the code, written as text. The model never called a tool, so no game file existed. In this test series, a broken answer ended the work.

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed 13 of 14 hidden tests in Think: medium, but only 6 in Instruct mode. In Think: medium, the only mistake was one row of the level map, shifted by two tiles (see below). The model needed 20 tool calls, the fewest of all setups that built a game, and it was the only setup that submitted the game on the first try without reminders.

In Instruct mode, two mistakes broke most hidden tests. The level was entered as tile coordinates, and most of them were wrong. The code that pushes the player out of solid tiles never pushed, so the player slowly sank into the ground. The run used all 50 answers and ended without a submission.

In this test, the thinking setups of Qwen 3.6 27B and Qwen 3.8 27B came out on top, and Qwen 3.6 27B was more even across the two modes. Both models ran in NVFP4 with the same settings. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.8 27B in Think: medium wrote the fewest tests and had one of the two best scores. Qwen 3.8 27B in Instruct mode wrote 16 tests, all passing, but they checked where the player went sideways, not whether it stayed on the ground. Qwen 3.6 35B is not in the chart: neither of its runs produced a game file.

Checks and edits

Qwen 3.8 27B in Instruct mode needed the most checks and edits and still had the lowest score of the setups that built a game. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.8 27B in Think: medium needed the fewest edits. Qwen 3.6 35B made no tool calls in either mode.

Unwanted tool calls

Qwen 3.6 27B in Instruct mode spent 26 tool calls on nothing. Twenty of them repeated an edit that was already applied, so they were refused, and six were invalid edits. Qwen 3.8 27B in Instruct mode had seven rejected calls: five referred to text that the answer did not contain, one had broken syntax, and one repeated an earlier edit. Qwen 3.8 27B in Think: medium had two calls that referred to missing text. Qwen 3.6 27B in Thinking mode had one rejected submission, while some of its own tests still failed.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls read far more tokens. Qwen 3.6 27B and Qwen 3.8 27B in Instruct mode each read more than three million tokens, because both used all 50 answers. Qwen 3.8 27B in Think: medium read the fewest tokens of the setups that built a game. Thinking mode writes more tokens because the model reasons before each step. Qwen 3.6 35B read almost nothing: both runs ended in the first answer.

One run step by step

Qwen 3.8 27B in Think: medium built the game in a few steps and fixed its own tests where they were wrong.

1. Its first call referred to text that the answer did not contain, so it was rejected. The model then wrote a skeleton of 168 lines: the level map as text rows, the start state and two tests.
2. The check failed both tests: one row of the map was two characters short, so reading it went past its end. The model added two dots to the row (see below).
3. It added the movement and the collisions with tiles. The check found two variables declared in one line, which SC# does not allow, and the model fixed it.
4. It added coins, blocks, the mushroom, enemies and the drawing in three more edits. One check found text joined with a number in a way that SC# does not accept; the model fixed it.
5. Two of its nine tests failed. Both expected wrong values: one counted coins that the test never collected, and one expected 60 hurt frames, although the task lowers the counter in the same frame. The model corrected the two tests and left the game as it was.
6. All nine tests passed, and the model submitted the game. 13 of the 14 hidden tests passed.

How the models fail

A row of the map shifted by two tiles

Qwen 3.8 27B in Think: medium found a short row in its map, but it added the missing dots in the wrong place. The row of the map with the first pipes was two characters short at the start. The model's test reported that reading the row went past its end. The model added two dots in the middle of the row, so the length was right, but the first enemy and two pipes ended up two tiles too far to the left:

Only the hidden test that compares the level failed. Qwen 3.8 27B in Instruct mode did worse with the same map: it entered the level as single tile coordinates, and most of them were wrong. For example, it put the tall pipe in columns 26 and 27 instead of 30 and 31, and its own test checked the pipe in column 26, so the test confirmed the mistake.

The player sinks into the ground

In the game of Qwen 3.8 27B in Instruct mode, the player sinks into the ground by half a pixel in every frame. The task says: when the player moves down into a solid tile, the player is put on top of it. The model's code collects the candidate pushes, but it keeps only a push that is larger than zero. A push up out of the ground is negative, so it is never used:

The code sets onGround to true and the speed to 0, so the player can still walk and jump. But in every frame, gravity moves the player half a pixel deeper: from 192 at the start to 208.5 after 33 frames. The push up out of a ceiling has the same mistake with the opposite sign, and the side check has errors of its own, so the player can walk into a pipe. With the floor and ceiling fixed, five more hidden tests pass. The model's 16 tests all passed: its walking test checked only where the player went sideways, not its height.

Two ways to get the enemy's turn wrong

Both Qwen 3.6 27B runs failed the rule that turns an enemy around at the edge of a pit, in two different ways. The task says: if after the step the tile below the middle of the enemy's bottom edge would not be solid, the enemy does not move and turns around.

With the check fixed, both games pass the enemy test. In the Instruct game, the enemies in the real level never move, which a player sees at once; the model's own tests did not include a walking enemy.

The Instruct game had one more mistake: when the falling mushroom reached the ground, the code computed the landing height from the tile of its top edge instead of the tile of its bottom edge, so it put the mushroom one tile too high. The mushroom then fell again and never reached the ground.

Two runs that ended in the first answer

Qwen 3.6 35B lost both runs before a game file existed.

• Instruct mode: the model planned in steps, then started to write the level as a list of 900 numbers, one per tile, and repeated "0, " until the runtime stopped the answer as broken output.
• Thinking mode: the model planned the game and drafted large parts of the code as text, until the answer reached the output limit. It never called a tool. The only code block in the answer was a short example copied from the SC# documentation, and it did not compile on its own.

Conclusion

Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in Thinking mode built the best games, with one lost hidden test each.

• Qwen 3.6 27B: 13 of 14 hidden tests in Thinking mode and 12 in Instruct mode; both modes got the enemy's turn at a pit wrong.
• Qwen 3.8 27B: 13 of 14 hidden tests in Think: medium, with one shifted row of the map. In Instruct mode, a wrong map and a player that sinks into the ground left 6 passing tests.
• Qwen 3.6 35B: no game in either mode; both runs ended in the first answer.

For a game with many exact rules like this one, the thinking modes of Qwen 3.6 27B and Qwen 3.8 27B in NVFP4 gave the most reliable results. In Instruct mode, check the given level data and test the full state of the player after each move.

References

Benchmark runs with NVFP4 quantization