Introduction

Can a local model build a falling blocks game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a falling blocks game like Tetris as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 17 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a classic falling blocks game on a board of 10 columns and 20 rows. Pieces of four cells fall from the top. The player moves and rotates them with the arrow keys, drops them fast with Space or Down, and fills rows. A full row disappears, and the rows above it move down. The game gets faster as the player clears rows, and it is over when a new piece does not fit.

The exact rules the hidden tests check include the cells of all seven pieces and the formula for a clockwise rotation, when a position is free, the order of the steps in each frame, the gravity counter, hard drop and soft drop, locking, the removal of full rows, the scores for one to four rows, a fair order of the pieces, the speed limits, game over and restart, and the colors on the screen. The model chooses how to build all of this.

Results

Qwen 3.6 27B passed every result in both modes, Qwen 3.8 27B lost one result in Think: medium and three in Instruct mode, and Qwen 3.6 35B built no working game. The chart below shows the score of each setup: the share of passed results, the build of the game and the 17 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a correct game in both modes: all 17 hidden tests passed. Both games handle the soft drop and the removal of several rows correctly, the two rules where Qwen 3.8 27B made mistakes.

In Thinking mode, the model needed only eight tool calls. It wrote the whole game at once, fixed two of its own tests and submitted the game (see "One run step by step"). In Instruct mode, the model first fixed four compile errors one by one, mostly C# features that SC# does not have. Then it stopped three times without a tool call and without submitting the game. After each reminder it continued, and in the end it submitted a game that passed every hidden test.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not build a working game in either mode. In Instruct mode, its first answer got stuck in a loop while it was writing the empty board, and the runtime stopped the answer before a game file existed. In Thinking mode, the model wrote a game, but its edits broke the structure of the file again and again. The run ended at the limit of 50 rounds with a file that did not compile. Both problems are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed 16 of the 17 hidden tests in Think: medium and 14 in Instruct mode. Both games break the same rule: a soft drop step must also stop the gravity counter, but the games let it count on (see below). In Instruct mode, the game also removes only every second row when two or more full rows lie next to each other. This one mistake broke two more hidden tests.

In Think: medium, the model built the game in a few large steps and needed only 12 tool calls that did work. In Instruct mode, it needed 35, and many of its answers were full of repeated markup text (see "Tokens"). Both games handle the pieces, the rotations, the hard drop and the speed rules correctly.

Qwen 3.6 27B passed every hidden test in both modes, Qwen 3.8 27B came close in Think: medium. Both models ran in NVFP4 with the same settings. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

None of the setups wrote a test for the soft drop. Qwen 3.6 27B in Thinking mode wrote the most tests, 15, and both of its games got the soft drop right without a test. The two Qwen 3.8 27B games broke exactly this rule. Qwen 3.8 27B in Instruct mode tested only the removal of a single row, so it did not see that two full rows next to each other fail. Qwen 3.6 35B is not in the chart: neither of its runs produced a game that compiles.

Checks and edits

Qwen 3.6 35B in Thinking mode made 31 edits and still had no game that compiles at the end. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.8 27B in Instruct mode ran 17 checks, more than any other setup, partly because it often added a check to the same answer as an edit. Qwen 3.6 27B in Thinking mode needed two checks and four edits.

Unwanted tool calls

Qwen 3.6 35B in Thinking mode spent nine tool calls on nothing. Eight of them were incomplete edits, and one referred to text that the answer did not contain. Qwen 3.8 27B had three rejected calls in Instruct mode and two in Think: medium: broken call syntax and references to text that was missing. Qwen 3.6 27B made no wrong calls in either mode, but in Instruct mode it stopped three times without any tool call and needed three reminders.

Tokens

Every tool call sends the whole conversation to the model again, so long runs read far more tokens. Qwen 3.8 27B in Instruct mode read over 2.8 million tokens. Its answers repeated a closing markup tag more than 35,000 times in total, which made up most of the text it wrote. This text stayed in the conversation, so every later call read it again. Qwen 3.6 35B in Thinking mode read over two million tokens because it used all 50 rounds. Qwen 3.6 27B in Thinking mode read the fewest tokens of the setups that built a game.

One run step by step

Qwen 3.6 27B in Thinking mode wrote the whole game in its first answer and needed only eight tool calls.

1. It created the full game in one file: 537 lines with the rules, the drawing, the extra features and 13 tests.
2. The first check showed that the game compiled and 11 of its 13 tests passed. The model found the reason for the two failures: after a piece locks, the next piece starts at once, so its tests read the values of the new piece. It fixed the two tests, not the game.
3. In one larger edit, it changed the speed rule of the game and added two more tests. This edit broke the test code, and the model replaced the damaged part with clean test functions.
4. The second check showed that all 15 of its tests passed.
5. It submitted the game, and all 17 hidden tests passed.

How the models fail

The gravity counter keeps running during a soft drop

Both Qwen 3.8 27B games let the gravity counter run on while the player holds Down. The task gives a fixed order of steps in each frame. Step 4 is the soft drop: the piece moves down one row or locks, and then step 5, gravity, is skipped. The games skip gravity only when the piece locks. When the piece only moves down, the counter still grows:

The hidden test lets 10 frames pass, holds Down for 5 frames and then waits. The counter must still be at 10 after the soft drop, so the next gravity step must come 20 frames later. In both Qwen 3.8 27B games, the counter had grown to 15 during the soft drop, so the gravity step came five frames too early. The Instruct game has the same mistake with a different variable name.

Every second full row stays

The game of Qwen 3.8 27B in Instruct mode removes only one of two full rows that lie next to each other. The game checks the rows from the bottom up. When a row is full, it moves all rows above it down by one, and then it goes on to the next row up. But the row that just moved down into the current place is never checked:

With two full rows, the game removed one: lines was 1 instead of 2, and the score was 134 instead of 334. With four full rows, it removed two. The hidden test for the speed rule failed for the same reason: after three rounds of four rows, lines was 6 instead of 12, so the game never got faster. The model's own test removed only a single row, so it passed.

A loop and a broken file

Qwen 3.6 35B lost both runs before a working game existed.

• Instruct mode: in its first answer, the model started to write the empty board as a list of 200 zeros. It did not stop after 200 and repeated "0, " until the runtime stopped the answer as broken output. In this test series, a broken answer ended the work, so no game file existed.
• Thinking mode: the model wrote the game in parts. Many of its edits replaced the wrong lines, so braces went missing or appeared twice, and the model spent many rounds repairing the structure. Eight of its edits were rejected because they were incomplete. The game compiled for a while, and its three tests passed. Then the model added the extra features, made more edits and used its last round on an edit it never checked. The graded file did not compile: a closing brace was missing, so a function ended up inside another function.

Conclusion

Qwen 3.6 27B passed every result in both modes, and Qwen 3.8 27B in Think: medium was close behind.

• Qwen 3.6 27B: all hidden tests passed in both modes. In Instruct mode, it needed three reminders to submit the game.
• Qwen 3.8 27B: in Think: medium, all hidden tests but one passed. Both of its games let the gravity counter run during a soft drop, and the Instruct game also removes only every second full row.
• Qwen 3.6 35B: no working game in either mode, because of a loop and a file broken by its own edits.

For building a game like this as an agent, Qwen 3.6 27B in NVFP4 gave the most reliable result in this test. With Qwen 3.8 27B, check the rules that say what must not happen in a frame, and test the removal of several rows at once.

References

Benchmark runs with NVFP4 quantization