Introduction

Can a local model build a maze chase game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a maze chase game like Pac-Man as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 17 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result. One run, Qwen 3.6 27B in Thinking mode, was stopped by a server error before the model answered, so that setup has no result in this test.

The task

The game is a maze chase: Pac-Man eats dots in a maze while four ghosts hunt him. The maze is a grid of 19 x 21 cells, given in the task as a map of characters, with a tunnel that connects the left and the right edge. Pac-Man moves one cell every fourth frame and remembers the last arrow key until he can turn. The ghosts leave their house one after another, switch between chasing Pac-Man and going to their corners on a fixed schedule, and turn blue for a while after Pac-Man eats a power pill. A blue ghost can be eaten and returns to the house as eyes. A normal ghost on Pac-Man's cell takes a life.

The exact rules the hidden tests check include the maze and the start positions, the order of the steps inside a frame, the remembered turn, the tunnel, the speed of the ghosts in each state, when the ghosts leave the house, the schedule of the modes and the reversals at each change, the corners and the chase, the power pill and the scores for eaten ghosts, losing a life, game over and the restart, a complete level and the colors on the screen. The model chooses how to build all of this.

Results

Qwen 3.8 27B in Think: medium passed every result, Qwen 3.8 27B in Instruct mode lost five, and Qwen 3.6 27B in Instruct mode and Qwen 3.6 35B built no working game. The run of Qwen 3.6 27B in Thinking mode was stopped by a server error before the model answered, so it is not in the chart. The chart shows the score of the other five setups: the share of passed results, the build of the game and the 17 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B in Instruct mode built a game that compiles, but every hidden test failed; its Thinking run has no result. In Instruct mode, the first version of the maze was correct, and all 12 of the model's own tests passed. Then the model wrote the maze rows again and made the tunnel row two cells short. From then on, the game stopped with an error as soon as it read the maze. The model tried to fix the row, made it even shorter and then sent the same edit 26 more times until it reached the limit of 50 answers (see below). Only the build of the game passed.

The Thinking run was stopped by a server error before the model gave its first answer. With one attempt per setup and no reruns, Qwen 3.6 27B in Thinking mode has no result in this test.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not build a game that compiles in either mode. In Instruct mode, it wrote a game of 827 lines at once with an array type that SC# does not have, and then it answered three times with the whole file as text instead of changing it with a tool. In Thinking mode, it wrote 700 lines and changed them six times without running a single check. Both runs are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium and 12 of the 17 hidden tests in Instruct mode. In Think: medium, the model built the game in steps, checked it four times and read the rules once more before it submitted the game. It wrote only two tests of its own, and both passed.

In Instruct mode, most of the game was correct: the maze, the tunnel, the remembered turn, the ghost speeds, the house, the power pill and the scores. Three rules about the order of the steps inside a frame were missing, and they broke five hidden tests (see below). Most of its first ten checks were about the maze. It added 13 tests at the end and reached the limit of 50 answers before it could hand the game in.

Only Qwen 3.8 27B built a fully working game in this test. Qwen 3.6 27B has no Thinking result to compare with, so this test does not show how the two thinking setups compare.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests. Qwen 3.6 27B in Thinking mode is not in the charts, because its run has no result.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.8 27B in Think: medium wrote only two tests, of the maze and the start positions, and its game passed every hidden test. Qwen 3.8 27B in Instruct mode wrote 16 tests, but its test of a lost life called the meeting function directly instead of running a frame, and no test covered a complete level. Qwen 3.6 27B in Instruct mode had 12 passing tests before it broke the maze. Qwen 3.6 35B is not in the chart: neither of its runs produced a game that compiles.

Checks and edits

Qwen 3.8 27B in Instruct mode needed the most checks and edits, and Qwen 3.8 27B in Think: medium the fewest of the setups that built a game. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.6 35B in Thinking mode made six edits and no check at all, so it never saw a compile error. Qwen 3.6 35B in Instruct mode made one check and no edit.

Unwanted tool calls

Qwen 3.6 27B in Instruct mode spent 28 tool calls on refused repeated edits. At the end of its run, it sent the same edit of the tunnel row 26 times in a row, and each time the edit was refused because it had already been applied. Qwen 3.8 27B in Instruct mode had 14 rejected calls: eleven referred to text that the answer did not contain, two had broken call syntax, and one asked for a tool that does not exist. Qwen 3.6 35B in Thinking mode had two calls with broken syntax, and Qwen 3.8 27B in Think: medium one call that referred to missing text.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls read far more tokens. Qwen 3.8 27B in Instruct mode and Qwen 3.6 27B in Instruct mode both used all 50 answers and read about 2.5 million tokens each. Qwen 3.8 27B in Think: medium read a quarter of that. Thinking mode writes more tokens because the model reasons before each step. Qwen 3.6 35B in Thinking mode wrote about as many tokens as Qwen 3.8 27B in Think: medium, without ever getting a game that compiles.

One run step by step

Qwen 3.8 27B in Think: medium built the game in small steps and checked the rules once more before it submitted.

1. It started with a skeleton of 143 lines: the variables, the maze, the start positions, a minimal game loop and two tests. The check found two variables declared in one line, which SC# does not allow, and the model fixed it.
2. It added the frame loop and the movement of Pac-Man. The check showed functions that the loop called but that did not exist yet. The model added the ghosts, the meeting rule, the effect and the pause.
3. One of its two tests failed: the test expected a dot on a wall cell. The model fixed the test, not the game. In the same step, it removed the spaces at the end of the tunnel row and wrote the drawing.
4. A few answers later, it noticed that the tunnel row now had only 15 cells instead of 19, and that a helper used an output parameter that SC# does not have. It fixed both. Three of these answers had no tool call that the tools could read, so it needed three reminders.
5. Both tests passed. Before it submitted the game, the model read the rule for reversing ghosts again and added the forced step after a reversal, which the task asks for.
6. It submitted the game, and all 17 hidden tests passed.

How the models fail

A correct maze written again

Qwen 3.6 27B in Instruct mode broke a correct maze after all its tests had passed. The task gives the maze as 21 rows of characters and says that the tunnel row is 19 cells long, with four empty cells at each end. In the model's first version, the row was right. After all 12 of its tests passed, the model looked at the rules again, decided that the row was "longer than 19 characters" and wrote the maze rows again:

The next check showed that all 12 tests now failed because a row was too short. The model wrote that the row had "only 17 characters instead of 19", but its fix had 15. That edit was the last one that changed the file. The model sent the same edit 26 more times; each repeat was refused, and the run ended at the limit of 50 answers. Every hidden test stopped at the same error.

Qwen 3.8 27B made a similar slip in Think: medium: it removed the spaces at the end of the tunnel row. A few answers later, it noticed that the row had 15 cells instead of 19, restored them and went on.

The rest of the frame goes on

Qwen 3.8 27B in Instruct mode let the frame go on after a lost life or a complete level. The task says that in both cases everything goes to the start positions and "the rest of the frame is skipped". The model's game loop stops only when the game is over:

The same gap was inside the ghost loop. When a ghost took a life, the loop went on with the next ghosts, so ghost 3 counted down its waiting time once more: 269 instead of 270 at the start position. Three hidden tests failed for this reason. With only this change, the game passes 15 of the 17 hidden tests.

Two more rules about the frame were missing in the same game:

• Meeting in every frame. The task says that Pac-Man meets the ghosts on his cell "in every frame (also when he did not move)". The model checks this only in the frames in which Pac-Man may move:
  C#Copy

  void UpdatePacman()
  {
      if (tick % 4 != 0) { return; }
      ...
      MeetGhostsOnPacman();
  }

  A ghost that stands on Pac-Man's cell in another frame can step away before they meet, so Pac-Man does not lose a life.

• The step after a reversal. When the mode changes, a ghost reverses, and the task says that its next step goes in the new direction if that cell is open. The model only turns the direction around, so the ghost chooses its next step again and can go another way. Qwen 3.8 27B in Think: medium added exactly this rule before it submitted its game.

With all three rules added, the Instruct game passes every hidden test. The model's own test of a lost life called the meeting function directly, without running a frame, so it could not find these gaps.

No game that compiles

Qwen 3.6 35B lost both runs before it had a game that compiles.

• Instruct mode: the model wrote a game of 827 lines in its first step. The check found an array type that SC# does not have. The model then answered three times with the whole file as text, with its fixes inside, instead of changing the file with a tool. The run stopped after three answers in a row without a tool call, and the file still had the first error.
• Thinking mode: the model wrote a game of 700 lines and never ran a check. It changed the file six times, based on guesses about what SC# does not support. One guess was wrong: it removed the starting values of the lists that hold the game state, but SC# needs a starting value for every global variable, so the game did not compile. After two calls with broken syntax and several answers with the file as text, the run stopped after three answers in a row without a tool call.

Conclusion

Qwen 3.8 27B in Think: medium was the only setup that built a fully working game.

• Qwen 3.8 27B: all hidden tests passed in Think: medium. In Instruct mode, a mostly correct game lost five hidden tests because of three rules about the order of the steps inside a frame.
• Qwen 3.6 27B: in Instruct mode, the game failed every hidden test after the model broke a correct maze row. The Thinking run was stopped by a server error, so that setup has no result.
• Qwen 3.6 35B: no game that compiles in either mode.

For a game with many exact rules like this one, the thinking mode of Qwen 3.8 27B in NVFP4 did the job in one attempt. In Instruct mode, check that the given data stays unchanged and that the rules for special frames are tested.

References

Benchmark runs with NVFP4 quantization