Introduction

Can a local model build a mine field puzzle on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a mine field puzzle like Minesweeper as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 15 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a mine field puzzle on a board of 16 x 16 cells with 40 mines, played only with the keyboard. A highlighted cursor moves with the arrow keys. Space reveals the cell under the cursor, and F puts a flag on it or takes the flag away. A revealed cell shows how many of its eight neighbors hold a mine, and a cell with no mines around it opens its whole empty area. The game is lost when a mine is revealed and won when every safe cell is open. The first reveal is always safe.

The exact rules the hidden tests check include the state of a new game, where the mines may not be placed, how an empty area opens and where it stops, the flag counter, the timer, the chord rule and what the screen shows. The chord rule is the hardest part: Space on a revealed number whose flags are all placed reveals the remaining neighbors one by one, in a fixed order, and stops at the first mine. The model chooses how to build all of this.

Results

Qwen 3.6 27B in Instruct mode and Qwen 3.8 27B in Think: medium passed every result, Qwen 3.6 27B in Thinking mode and Qwen 3.8 27B in Instruct mode lost one each, and Qwen 3.6 35B built no working game. The chart below shows the score of each setup: the share of passed results, the build of the game and the 15 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a working game in both modes: all hidden tests passed in Instruct mode, and all but one in Thinking mode. In Thinking mode, the model wrote a game of 600 lines in one go and needed only four checks and five edits. Its chord code revealed the neighbors without the rule for empty cells, so an empty neighbor did not open its area (see below).

In Instruct mode, the game was correct, but the model never handed it in. One of its own tests failed for a long time; the model found that the first reveal opened the whole board and won the game at once. The model fixed this test in the end, but on the way ten of its edits were refused because it repeated an edit that was already applied. When all seven of its tests passed, it ran the same check again and again until the run reached its limit. The last version of the file was graded, and it passed every hidden test.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not build a working game in either mode. In Instruct mode, its first version of 404 lines passed all 15 hidden tests when we checked it later. The model did not hand it in. It ran the same check 24 times in a row on an unchanged file, then changed the chord code, left one closing brace too many and ran the check 17 more times without a fix. The graded file did not compile. In Thinking mode, the model never created a file: its tool calls had broken syntax or referred to text that the answer did not contain, and in between it wrote the game as plain text. Both runs are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium and all but one in Instruct mode. In Think: medium, the model built the game in steps from a small skeleton. It noticed a few problems by reading its own code, for example that the "GAME OVER" text was drawn inside the board area, which the task forbids. Its 13 tests found two more mistakes, both in the tests themselves (see "One run step by step").

In Instruct mode, the game was correct except for one detail of the chord rule: when a chord revealed a mine, the game did not stop revealing the other neighbors (see below). Its ten tests checked a chord without a mine, but not a chord that ends the game.

Both 27B models passed every hidden test in one mode and lost one chord test in the other. Both lost tests come from the same rule. With one run per setup, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.6 27B in Thinking mode wrote the most tests, 19, and still missed the empty area behind a chord. Qwen 3.6 27B in Instruct mode wrote only seven, and its game passed every hidden test. Qwen 3.6 35B is not in the chart: neither of its runs produced a game that compiles.

Checks and edits

Two runs spent most of their checks on nothing: Qwen 3.6 35B in Instruct mode ran 43 checks and Qwen 3.6 27B in Instruct mode 27. A check compiles the game and runs the model's tests. An edit changes some lines of the file. In both runs, most checks came in a row, with no edit in between, on a file that did not change. Qwen 3.6 27B in Thinking mode needed the fewest steps: four checks and five edits.

Unwanted tool calls

Qwen 3.6 27B in Instruct mode had ten refused edits, and Qwen 3.6 35B in Thinking mode did not make a single call that worked. The refused edits repeated an earlier edit after it was already applied. Qwen 3.6 35B in Thinking mode made seven calls: two had broken call syntax, four referred to text that the answer did not contain, and one asked for a check before a file existed. Qwen 3.8 27B in Instruct mode had three rejected calls: two referred to missing text, and one asked for a tool that does not exist. Qwen 3.6 27B in Thinking mode made no wrong calls.

Tokens

Every tool call sends the whole conversation to the model again, so the two runs with long chains of checks read the most tokens. Qwen 3.6 27B in Instruct mode read over 2.6 million tokens and Qwen 3.6 35B in Instruct mode over 2.2 million. Qwen 3.6 27B in Thinking mode read the fewest tokens, about 412,000, and still built a nearly complete game. Thinking mode writes more tokens because the model reasons before each step; Qwen 3.6 35B in Thinking mode also wrote the game several times as plain text.

One run step by step

Qwen 3.8 27B in Think: medium built the game in steps and found its mistakes before the hidden tests did.

1. Its first call referred to text that the answer did not contain, so it was rejected. The model then wrote a skeleton of 348 lines: the state, a new game, the drawing and three tests. All three passed.
2. It added the pause, the best time and the effects. In the next step, it noticed by itself that its last edit had dropped two lines that reset the exploded mine, and put them back.
3. It read its file again and fixed two things before any test failed: the "GAME OVER" and "YOU WIN" texts were inside the board area, which the task forbids, and the opening of empty areas used recursion, which it made a loop.
4. It added ten tests for the rules: the safe first reveal, losing, flags, the chord, winning, the timer, a restart and the pause. The check found a compile error: a list was created in a way that SC# does not accept. It fixed it.
5. Two of its tests failed. The model found the reason: with only a few mines on the board, the first reveal opened every safe cell, so the game was won at once, the timer stopped and the P key did nothing. The game was right; the tests were wrong. It surrounded the test cell with mines, and all 13 tests passed.
6. It submitted the game, and all 15 hidden tests passed.

How the models fail

Two ways to get the chord wrong

Both lost hidden tests come from the chord rule, and each 27B model broke a different half of it. The task says that a chord reveals the hidden neighbors "one by one" with the normal reveal rule, so an empty neighbor opens its area, and that "when one of them has a mine, the game is lost and the rest is not revealed".

Qwen 3.8 27B in Instruct mode used the normal reveal, but its loop did not stop after a mine:

Qwen 3.6 27B in Thinking mode stopped at the mine, but it marked each neighbor as revealed by hand instead of using the reveal rule:

We checked both causes: with the one-line change on the right, each game passes all 15 hidden tests. Both models had a test of the chord, and both tests passed. Qwen 3.8 27B tested a chord without a mine. Qwen 3.6 27B tested a chord with a mine, and in its other chord test it checked only that one neighbor with a number was revealed, not that the empty area behind it opened.

A working game that was never handed in

Qwen 3.6 35B in Instruct mode had a correct game early and lost it; Qwen 3.6 27B in Instruct mode made the same mistake and was lucky. Both runs reached a point where every test passed, and both then ran the same check again and again instead of submitting the game.

• Qwen 3.6 35B, Instruct: the first version of 404 lines passed all 15 hidden tests when we ran them on it later. The model's next edit broke the file, and the edit after it repaired it. Then the model wrote "Let me check the current status of the app." and ran the same check 24 times in a row, with the same result each time and no change to the file. Then it rewrote the chord loop. Its loop had checked far more cells than the eight neighbors, and the new loop was right, but the edit left one closing brace too many. The model ran the check 17 more times with the same compile error and did not change the file again. The run ended at its limit, and the graded file did not compile: 0 of 16.
• Qwen 3.6 27B, Instruct: after its tests passed, the model wrote "Let me check the current state of the app." 22 times, with three more edits in between. It never called the tool that hands in the game. The run ended at its limit, but the last file version compiled and passed every hidden test.

The same behavior gave the best and the worst score in this test. The difference was only which version of the file was last.

No file at all

Qwen 3.6 35B in Thinking mode wrote the game several times but never as a file. Its first two calls had broken call syntax. Then it wrote the whole game as plain text in its answer, three times in a row, and got a reminder after each answer. After that, it tried to create the file four times, but each call referred to text that the answer did not contain, so each call was rejected. After six reminders, the run stopped: three answers in a row had no tool call. No file existed, so all hidden tests failed.

Conclusion

Qwen 3.6 27B in Instruct mode and Qwen 3.8 27B in Think: medium passed every result, and the other two 27B setups lost one test each.

• Qwen 3.6 27B: all hidden tests passed in Instruct mode, but only because the last file version was correct; the model never handed in the game. In Thinking mode, the chord did not open empty areas.
• Qwen 3.8 27B: all hidden tests passed in Think: medium, and its own tests caught its mistakes before the hidden tests ran. In Instruct mode, the chord did not stop at a mine.
• Qwen 3.6 35B: no working game in either mode. In Instruct mode, a correct first version was lost to a long chain of checks and one broken edit. In Thinking mode, no file was created.

For a puzzle like this, both 27B models in NVFP4 can build a nearly complete game in one attempt. Rules that combine two other rules, like the chord, need their own tests, and a run that repeats the same check many times needs attention even when its tests pass.

References

Benchmark runs with NVFP4 quantization