Introduction

Can a local model build a Snake game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write the classic Snake game as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 14 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is Snake on a grid of 32 x 22 cells. The snake moves one cell at a time in its direction, and the arrow keys turn it. When it eats the food, it grows by one cell and the score grows. It gets faster as it eats. The game is over when the snake hits a wall or its own body, and Enter starts a new game.

The exact rules the hidden tests check include the start position, the number of frames between two steps, which turns are allowed, how the body follows the head, when the tail's cell counts as free, the speed limits, what happens in the frame in which Enter starts a new game, and the colors on the screen. The model chooses how to build all of this.

Results

Qwen 3.6 35B in Thinking mode and Qwen 3.8 27B in Instruct mode passed every result, three setups lost one, and Qwen 3.8 27B in Think: medium lost three. All six setups built a game that compiles and plays. The chart below shows the score of each setup: the share of passed results, the build of the game and the 14 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a working game in both modes and lost one hidden test in each. In Instruct mode, the frame in which Enter starts a new game already counted as the first frame of the new game, so the first step came one frame too early. In Thinking mode, the snake got slower at the tenth food (both are described below).

The two modes worked very differently. In Instruct mode, the model wrote 15 tests, but it repeated the same review text in five answers in a row and ran the check again each time without a change. In Thinking mode, it wrote only five tests, but it read the rules again and fixed several of them on its own, among them the exact restart rule that Instruct mode missed.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B passed every hidden test in Thinking mode and all but one in Instruct mode. In Thinking mode, it wrote the whole game at once, fixed one compile error and one weak spot in its speed rule, and submitted (see "One run step by step").

In Instruct mode, the model needed three reminders. Its first two answers contained the code only as text, without the tool call that creates the file. When the file existed, the game was almost correct. Only one rule was wrong: a quick second key press could turn the snake back into itself (see below).

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Instruct mode, but only 11 of the 14 in Think: medium. In Instruct mode, the start was rough: its first five answers had six tool calls that were rejected, and only the sixth answer created a file. After that, it built the game in small steps, and every check passed.

In Think: medium, the model wrote the longest game and the most tests. But one wrong line in the step code removed a cell from the middle of the snake instead of its tail. This one line broke three hidden tests (see below).

In this small game, thinking mode was not always better. Qwen 3.6 35B did best in Thinking mode, but Qwen 3.8 27B did best in Instruct mode, and both modes of Qwen 3.6 27B lost one test. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.8 27B in Think: medium wrote 16 tests, the most, and still had the lowest score: its tests checked the head and the length of the snake after a step, but never the body. Qwen 3.6 27B in Thinking mode wrote only five tests and lost one hidden test.

Checks and edits

Qwen 3.6 27B in Instruct mode ran the most checks, and Qwen 3.6 27B in Thinking mode made the most edits. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.6 27B in Instruct mode ran three of its seven checks without any change to the file since the check before. In Thinking mode, the same model made seven edits and ran only two checks: it fixed the rules by reading them, not by testing them.

Unwanted tool calls

Qwen 3.8 27B in Instruct mode made nine rejected calls, six of them in its first five answers. Six referred to text that the answer did not contain, two were malformed, and one asked for a tool that does not exist. Each time, the model sent the call again in a correct form, and it still passed every hidden test. Qwen 3.6 27B in Instruct mode had three refused repeated edits and one malformed call. Qwen 3.6 35B in Instruct mode made no wrong calls, but three of its answers had no tool call at all.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls read more tokens. Qwen 3.6 27B in Instruct mode read the most, mainly because of its repeated checks. Qwen 3.6 35B in Thinking mode read the fewest: it needed only eight answers. Thinking setups write more tokens because the model reasons before each step. Qwen 3.8 27B in Instruct mode wrote only about 7,000 tokens for a full game.

One run step by step

Qwen 3.6 35B in Thinking mode built the game in eight answers and passed every hidden test.

1. Its first tool call was malformed: the code was inside the call instead of in the answer, so the call was rejected.
2. In the next answer, it wrote the whole game with 12 tests, 373 lines, in one file.
3. The check found a compile error: several variables declared in one line, which SC# does not allow. It rewrote the declarations, one variable per line.
4. Without a test pointing to it, the model found a weak spot in its speed rule and changed the order of the limits, so the delay can never drop below the minimum.
5. The check passed: all 12 of its tests were green.
6. It added the frame rate to the start of the game and submitted it.
7. All 14 hidden tests passed.

How the models fail

The tail stays, a middle cell goes

The game of Qwen 3.8 27B in Think: medium removed the wrong cell when the snake moved. A step adds a new head at the front of the lists and, if the snake does not eat, removes the last cell. The model saved the length before it added the head and then used that old length to find the last cell:

With this one fix, the game passes all 14 hidden tests. Without it, three tests failed: the body after a normal step, the length after eating, and a head that follows its own tail. The model's 16 tests all passed, because they checked the head and the length of the snake, but never the cells of the body.

The snake slows down at the tenth food

The game of Qwen 3.6 27B in Thinking mode got slower when the snake ate its tenth food. The task says that the delay between steps never grows during a game and is at most 4 frames after ten foods. The model's formula went down to 3 frames, and then a special case for ten foods set it back to 4:

Only the hidden speed test failed. The model wrote no test for the speed rule.

A turn back in two key presses

In the game of Qwen 3.6 35B in Instruct mode, the snake could turn back into itself with two quick key presses. The task says that a key is ignored when it points against the direction of the last step, and it gives this exact example: the snake moves right, the player presses Up and then Left before the next step, and Left must be ignored. The model compared each key with the current direction instead of the last step:

if (Input.WasPressed("Up"))   { if (dir != 2) { dir = 0; } }
if (Input.WasPressed("Left")) { if (dir != 1) { dir = 3; } }

After Up, dir is 0, so Left is accepted, and the next step goes left, back into the body. The model's own test checked only the simple case, Left while moving right, which this code handles correctly.

The restart frame counts

In the game of Qwen 3.6 27B in Instruct mode, the first step of a new game came one frame too early. The task says that the frame in which Enter starts a new game does nothing else, so the first step comes six frames later. The model's game loop starts the new game and then, in the same frame, counts the frame:

HandleInput();               // Enter starts a new game and clears gameOver
if (!gameOver && !paused)
{
    Update();                // counts this frame: the first step comes after 5 frames
}

The model had a test for the restart, but it checked only the score and the position, not when the first step comes. Qwen 3.6 27B in Thinking mode read the same rule, saw this problem in its own code and fixed it before it submitted.

Conclusion

All six setups built a working Snake game, and the differences came from single rules.

• Qwen 3.6 27B: one hidden test lost in each mode: the restart frame in Instruct mode, the speed rule in Thinking mode.
• Qwen 3.6 35B: all hidden tests passed in Thinking mode. In Instruct mode, one turn rule was wrong.
• Qwen 3.8 27B: all hidden tests passed in Instruct mode, after a rough start with rejected calls. In Think: medium, one wrong line broke three tests.

For a small game like this, each of the three models in NVFP4 can build a working version in one attempt. The details of the rules still need tests: the cases where a rule must not fire, and the full state after an action.

References

Benchmark runs with NVFP4 quantization