Introduction

Can a local model build a card game played with the mouse on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write Klondike solitaire as an app in SC#, our C#-like scripting language, with cards moved by dragging them with the mouse. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 15 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is Klondike solitaire with 52 cards. The cards are dealt into seven columns, with only the top card of each column face up, and the rest form the stock. A click on the stock turns its top card onto the waste, and a click on the empty stock turns the waste back. The player drags a face-up card, together with the cards above it, onto a column whose top card is one rank higher and of the other color. Only a King goes onto an empty column. Four foundations collect each suit from the Ace up, and a right click sends a card to its foundation when the move is legal.

The exact rules the hidden tests check include the order of the deal, which cards a press picks (also on the narrow visible strip of a covered card, exactly at its edges), which drops are legal, what happens after an illegal drop or a drop outside every pile, when a face-down card turns face up, the foundations and the right click, the win and a new game with Enter. On the screen, the tests check that the dragged card follows the mouse at the point where it was pressed and that face-up cards, face-down cards and the empty table look different. The model chooses how to build all of this.

Results

Qwen 3.8 27B in Think: medium passed every result, Qwen 3.6 27B lost two and three, and the other setups lost between six and eight. Every setup built a game that compiles and can be played with the mouse. Qwen 3.6 35B in Instruct mode did not submit its game: it used all 50 answers, and we graded the last version of its file. The chart below shows the score of each setup: the share of passed results, the build of the game and the 15 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a game that follows almost all rules in both modes. In Instruct mode, it lost two hidden tests. Its game turned a face-down card face up after every move from a column, also when the top card of the column was already face up (see below). While a card was dragged, the game also kept drawing it in its column.

In Thinking mode, it lost three hidden tests, all because of one loop. When several cards were dragged from a column, the loop that removed them from the column stopped halfway, so some cards stayed in both columns (see below).

In FP8, the same model lost one hidden test in Instruct mode and none in Thinking mode. Here FP8 scored a little higher than NVFP4, but with one run per setup, this is a hint, not a proof.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B built a playable game in both modes, but about half of the hidden tests failed. In Instruct mode, it worked until it ran out of answers. It made 32 edits, and 8 of its tool calls were rejected: references to text that its answer did not contain, a rewrite of the whole file, and edits it had already made. Its game let the player drag a face-down card, a right click sent a Two to an empty foundation, the winning move counted twice, and the dragged card was not drawn under the mouse.

In Thinking mode, the model submitted after 28 answers, with only two tests of its own. Its deal function took the cards out of the deck, so the deck was empty after every new game (see below). The game also drew the dragged card in its column and under the mouse at the same time, and some drops on the foundations went wrong.

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium, but lost six in Instruct mode. In Think: medium, it needed only 14 answers and painted its own card images (see "One run step by step").

In Instruct mode, the model needed six reminders: six of its answers had no tool call. Six other calls referred to text that the answer did not contain. Its game removed the dragged cards from a column with a wrong index, so picking up two or more cards ended with an error (see below). A second mistake broke every drop of a card from the waste or a foundation onto a column: after such a drop, the game tried to turn up a card in a column with a negative number. And when a card went from a column to a foundation, the card below it stayed face down.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.6 27B in Instruct mode wrote 16 tests, the most, and still lost two hidden tests. Qwen 3.8 27B in Think: medium wrote five and lost none. Both modes of Qwen 3.6 35B wrote only two tests each, and none of them covered the drag of a card.

Checks and edits

Qwen 3.6 35B in Instruct mode made by far the most edits. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.6 35B in Instruct mode made 32 edits but ran only eight checks, and it never reached a version it was ready to submit. Qwen 3.8 27B in Think: medium needed five edits and six checks for a game that passed everything.

Unwanted tool calls

Qwen 3.6 35B in Instruct mode had the most rejected calls, and Qwen 3.6 27B had none in either mode. Refused calls tried to rewrite the whole file or to repeat an edit that was already made. Other rejected calls referred to a block of text that the answer did not contain, or asked for lines that the file did not have. Qwen 3.8 27B in Instruct mode had six such calls, and it also needed six reminders because some of its answers had no tool call at all.

Tokens

Every tool call sends the whole conversation to the model again, so long runs read far more tokens. Qwen 3.6 35B in Instruct mode read the most because it used all 50 answers. Qwen 3.8 27B in Think: medium read the fewest, because it needed only 14 answers. The Thinking setups write more tokens because the model reasons before each step; Qwen 3.6 27B in Thinking mode wrote the most.

One run step by step

Qwen 3.8 27B in Think: medium built the game in 14 answers and passed every hidden test.

1. It wrote a first version of 339 lines with two tests: the deal and a click on the stock.
2. The check found compile errors: a text joined to a number with +, which SC# does not allow. It fixed the two lines.
3. Both tests passed. It added the drawing of the cards, but its first edit asked for lines the file did not have, so it was rejected. It sent the edit again with the right lines.
4. One test failed because the game drew a card back image before it was painted. It painted its card images first.
5. It added the drag, the foundations and the right click, with three more tests. All five tests passed.
6. It added the rest of the drawing, checked again and submitted.
7. All 15 hidden tests passed.

How the models fail

Cards removed from a list that gets shorter

Two setups broke the drag of several cards with the same kind of loop. When the player drags a card with the cards above it, the game must take all of them off the column. Both loops removed cards from a list while the list got shorter:

Removing one card at a time from the end, with the current length, works in both cases. We checked this: with only this loop fixed, the game of Qwen 3.6 27B in Thinking mode passes all 15 hidden tests. Without the fix, the loop broke three: the drag of a run, a King on an empty column and a press on the strip of a covered card. In the game of Qwen 3.8 27B in Instruct mode, it broke the same three tests. Both models had tests for the drag, but only of a single card.

The face-down card turns up too early

The game of Qwen 3.6 27B in Instruct mode turned a card face up after every move from a column. The task says that a face-down card turns face up when it becomes the top card of its column. The model's function did not check that:

With only this condition fixed, the game passes the hidden test for the strips of covered cards. In a normal game, this mistake is easy to miss: most moves take the last face-up card of a column, and then the result is the same. The hidden test moved part of a run and left a face-up King in the column, and the card below the King turned face up too.

The deck is used up by the deal

The game of Qwen 3.6 35B in Thinking mode emptied the deck at every deal. The task says that deck holds the 52 cards a game is dealt from, and that the deal takes the cards in order without shuffling. The model's deal took each card out of the list:

tableau[c].Add(deck[0]);
deck.RemoveAt(0);

After a new game, deck was empty. The tests of the deal and of a new game failed because they read the deck, and a second deal from the same deck had no cards. The model wrote two tests, and neither of them read the deck after the deal.

Conclusion

Only Qwen 3.8 27B in Think: medium built a game that follows every rule, and most mistakes were in moving several cards at once.

• Qwen 3.6 27B: a nearly complete game in both modes, with two and three hidden tests lost. Each mode had one wrong rule for the cards left in a column.
• Qwen 3.6 35B: a playable game, but about half of the hidden tests failed in both modes. In Instruct mode, it ran out of answers before it submitted.
• Qwen 3.8 27B: every hidden test passed in Think: medium, in only 14 answers. In Instruct mode, three mistakes in moving cards broke six tests.

For a card game played with the mouse, Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in either mode give a working base in one attempt. The drag of several cards and the rule for the card that turns face up need tests with more than one card.

References

Benchmark runs with NVFP4 quantization

Benchmark runs with FP8 quantization (side note)