Introduction

Can a local model build a checkers game against the computer on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a checkers game as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 14 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result. One run, Qwen 3.6 27B in Instruct mode, was stopped by a server error while the model was still working, so that setup has no result in this test.

The task

The game is checkers on an 8 x 8 board against the computer, played only with the keyboard. The player moves a cursor with the arrow keys, selects a piece with Space or Enter and then confirms the square to move to. Men move one square diagonally forward, captures are mandatory, a piece that can capture again must go on in the same turn, and a man that reaches the last row becomes a king. The computer plays its whole turn in one frame. The game ends when a side has no piece or no legal step, and it is a draw after 40 turns in a row with only king moves and no capture.

The exact rules the hidden tests check include the start position, the cursor, which pieces can be selected and when the selection changes, simple moves and captures, mandatory captures, multi-jumps, promotion (a man crowned by a capture ends its turn there), king moves, a legal turn of the computer, the win, loss and draw rules, the restart and the colors on the screen. The model chooses how to build all of this.

Results

Qwen 3.8 27B lost one hidden test in each mode, Qwen 3.6 27B in Thinking mode lost three, and Qwen 3.6 35B built no game that compiles. The run of Qwen 3.6 27B in Instruct mode was stopped by a server error before the model finished, so it is not in the chart. The chart shows the score of the other five setups: the share of passed results, the build of the game and the 14 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a working game in Thinking mode and lost three hidden tests; its Instruct run has no result. In Thinking mode, two mistakes caused the losses. A selected piece kept the selection when the player confirmed another movable piece, and a multi-jump never ended after its last jump, so the player's turn did not pass to the computer (both are described below).

The Instruct run was stopped by a server error after eight tool calls, while the model was still working: its last answer listed the fixes it was about to make. The game it had at that moment was graded, but this is not the model's finished work, so we do not count or analyse it. With one attempt per setup and no reruns, Qwen 3.6 27B in Instruct mode has no result in this test.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not build a game that compiles in either mode. In Instruct mode, it wrote a game of 800 lines with a type that SC# does not have, and then it sent its fixes in a format that the tools do not accept. In Thinking mode, it created a value with C# syntax that SC# does not support, and it never removed that pattern. Both runs are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B lost one hidden test in each mode, but a different one. In Instruct mode, a man that reached the last row by a capture was not crowned, and it went on jumping as a man (see below). In Think: medium, a selected piece kept the selection when the player confirmed another movable piece, the same mistake as Qwen 3.6 27B in Thinking mode.

The two modes worked differently. In Think: medium, the model built the game in steps and ran nine checks (see "One run step by step"). In Instruct mode, it ran only three checks, but it tried five times to call a tool that does not exist.

In this game, the selection rule was the most common mistake. Two of the four games that compiled lost the same hidden test for it. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests. The stopped run of Qwen 3.6 27B in Instruct mode is left out of all charts.

Own tests

All three setups that built a working game wrote between 10 and 12 tests, and none of them tested the rule that each one broke. Qwen 3.8 27B in Instruct mode tested promotion only with a simple move, not with a capture. Qwen 3.6 27B in Thinking mode tested a single capture, but no multi-jump to its end. No setup tested what happens when the player confirms another piece while one is selected. Qwen 3.6 35B is not in the chart: neither of its runs produced a game that compiles.

Checks and edits

Qwen 3.6 27B in Thinking mode made the most edits, 19, and lost the most hidden tests of the working games. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.8 27B in Think: medium ran as many checks as Qwen 3.6 27B in Thinking mode, but needed fewer edits. Qwen 3.6 35B in Instruct mode made no edit and ran no check after it created the file.

Unwanted tool calls

Qwen 3.8 27B in Instruct mode spent seven tool calls on nothing. Five of them asked for a tool that does not exist, one had broken call syntax, and one repeated an earlier edit after the file had changed, so it was refused. Qwen 3.6 27B in Thinking mode had one rejected edit. Qwen 3.6 35B in Thinking mode tried to submit a game that did not compile, and the submission was rejected. Qwen 3.6 35B in Instruct mode made only one tool call that worked; its later calls were written in a format that the tools do not read, so they are not counted as calls at all.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls read far more tokens. Qwen 3.6 27B in Thinking mode read over 1.5 million tokens, most of them because of its 31 tool calls. Thinking setups write more tokens because the model reasons before each step. Qwen 3.6 35B in Thinking mode also wrote many tokens: in its last answers, it printed the whole file as text instead of changing it with a tool.

One run step by step

Qwen 3.8 27B in Think: medium built the game in steps and found several of its own mistakes with its tests.

1. It wrote a first version of 507 lines with three tests: the start position, the cursor and the screen.
2. The check found a compile error: a function was called with two arguments instead of four. It fixed the call.
3. Only one of its three tests passed. Two tests expected wrong things: a man on a light square and a wrong pixel. It fixed both tests, and it also changed how the end of a turn counts the moves for the draw rule.
4. It added the computer's turn, the cursor and the marks of the legal landing squares from part B, and checked after each step.
5. It added the controls on the screen and more tests. A new test for mandatory captures failed, and it fixed how the selected piece moves.
6. A test for the computer's multi-jump ended with an endless chain of calls. The model rewrote the computer's jump, and all 12 of its tests passed.
7. It submitted the game, and 13 of the 14 hidden tests passed. The missing rule was the change of the selection (see below).

How the models fail

A selected piece keeps the selection

In two games, a selected piece stayed selected when the player confirmed another piece that could move. The task gives the confirm key a list of cases, checked in order: (b) if a piece is selected and the cursor is on one of its legal landing squares, the piece moves; (c) else, if the cursor is on a player piece with a legal step, that piece becomes the selected one; (d) otherwise nothing happens. Qwen 3.6 27B in Thinking mode even labelled its code with these letters, but it stopped after case b:

Qwen 3.8 27B in Think: medium made the same mistake in a different form: its branch for a selected piece tried a jump and a simple move, and had no way out to case c. In both games, the hidden test failed at the same point: the selection stayed on the old piece. When we added case c to copies of both games, the hidden test passed.

A capture to the last row does not crown the man

In the game of Qwen 3.8 27B in Instruct mode, a man became a king only after a simple move, not after a capture. The task says that a man that ends a move or a jump on the last row becomes a king, and that a man crowned by a capture ends its turn there.

In the hidden test, the man jumps to the last row next to another enemy piece. Because it was not crowned, the game saw a further capture for the man (men may capture backward) and went on with a multi-jump. The test found a man instead of a king, and the second enemy piece was still to be taken. The model's own promotion test used only a simple move. When we crowned the man after a capture too and ended the turn there, the copy passed all 14 hidden tests.

The multi-jump never ends

In the game of Qwen 3.6 27B in Thinking mode, the player's turn did not end after the last jump of a multi-jump. The model handled a capture in two places. After the first jump, it checked whether the piece could capture again. In the code for the later jumps, it trusted a value that only meant "this was a capture and the man was not crowned":

var cont = ExecuteMove(jumpPieceX, jumpPieceY, dx, dy);
// cont is true after every capture without a crown,
// even when no further capture is possible
if (!cont)
{
    EndTurn(true);
}

So after the second jump, the game stayed in the middle of a multi-jump with no capture left, and the turn never passed to the computer. Two hidden tests failed: the multi-jump test and the test that plays many turns against the computer, because the player's piece had no legal step left. With a check for further captures in this branch and case c from the section above, a copy of the game passed all 14 hidden tests.

Two games that do not compile

Qwen 3.6 35B lost both runs to compile errors that it never fixed.

• Instruct mode: the model wrote the whole game, 800 lines, in its first answer. It used a list of tuples, a C# type that SC# does not have. In the next answers, it wrote its fixes as text that looked like tool calls but was not in the format that the tools read, among them a replacement of the whole file. After three answers without a working tool call, the run ended with the first version, which does not compile.
• Thinking mode: the model built the game in parts, with seven edits, and ran one check. The check reported that a value was created with C# syntax, new MoveInfo(), which SC# does not support. Its later edits changed other parts of the file, but this pattern stayed in it. Its submission was rejected, and then it printed the file as text in four answers instead of editing it. The run ended after five reminders.

Conclusion

Qwen 3.8 27B came closest in this test: it lost one hidden test in each mode.

• Qwen 3.8 27B: 13 of 14 hidden tests in both modes. In Instruct mode, a capture to the last row did not crown the man. In Think: medium, the selection rule was wrong.
• Qwen 3.6 27B: 11 of 14 hidden tests in Thinking mode, because of the selection rule and a multi-jump that never ended. The Instruct run was stopped by a server error while the model was still working, so this setup has no result.
• Qwen 3.6 35B: no game that compiles in either mode.

For a rule-heavy game like checkers, the 27B models in NVFP4 can build a nearly complete game in one attempt, but none of the working games was fully correct. The rules that failed were the ones the models did not test: a list of cases for one key, promotion by a capture, and the end of a multi-jump.

References

Benchmark runs with NVFP4 quantization