Introduction

Can a local model build a billiards game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write an 8-ball pool game, seen from the top, as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 16 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is 8-ball pool for two players on one keyboard. The player turns the cue with the arrow keys, sets the power with Up and Down and shoots with Space. The balls roll, bounce off the cushions, hit each other and slow down. A ball that comes close to one of the six pockets falls in. The first ball a player pockets gives them a group: the solids or the stripes. A player who pockets a ball of their group shoots again. The player who pockets the black 8-ball after all balls of their group wins; pocketing it earlier loses the game.

The exact rules the hidden tests check include the start position of the 16 balls, aiming and power, the start of a shot, the physics of one frame in four sub-steps with a fixed order (move, pockets, cushions, ball collisions), friction, the end of a shot with the rules for fouls, groups, turns and the 8-ball, and the drawing. Part B adds an aim guide, a power indicator, the pocketed balls on the rails and a pause. The task says that part B may only use colors that part A does not use.

Results

Qwen 3.6 27B in Instruct mode and Qwen 3.8 27B in Think: medium passed every result. Qwen 3.8 27B in Instruct mode lost six, Qwen 3.6 35B in Thinking mode lost 13, and two setups built no game. The chart shows the score of each setup: the share of passed results, the build of the game and the 16 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B passed every hidden test in Instruct mode, but built no game in Thinking mode. In Instruct mode, the model wrote a game of 571 lines in its first step, fixed one compile error and grew the game to more than 800 lines with 20 tests. All its tests passed from its 18th tool call on, but it never submitted the game. It made a few more small changes and ran the same check again and again until the limit of 50 answers. The last version of the file was graded, and it passed every hidden test.

In Thinking mode, the first answer got stuck in a loop: while the model planned the start of a game, it repeated the same two sentences about the start position of the balls. The runtime stopped the answer as broken output. In this test series, a broken answer ended the run, so no game file existed.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B built a game in Thinking mode that passed only 3 of 16 hidden tests, and no working game in Instruct mode. In Thinking mode, one physics frame did only one of the four sub-steps, so every ball moved at a quarter of its speed. The list of balls that fell into a pocket was also emptied in every sub-step, so the rules at the end of a shot almost never saw a pocketed ball. Its own test of the physics expected the wrong distance and passed (see below).

In Instruct mode, the model wrote a short first version of 128 lines. Its only check found a compile error, and the model fixed it with two edits without checking again. A second error stayed: SC# does not add a number to a text with +. Then the model wrote a long answer that repeated "making the tool call" without making one, and the runtime stopped it as broken output. The graded file did not compile.

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in Think: medium and lost six in Instruct mode. In Think: medium, the model needed only nine tool calls. After it added the part B features, it noticed by itself that the pocketed balls on the rails used the colors of part A, and it changed them (see "One run step by step").

In Instruct mode, the physics was correct, but the end of a shot was not. The game decided whether the 8-ball was pocketed from whether it was on the table, not from whether it fell during the shot. In most hidden tests, the 8-ball is not on the table, so the game ended after every shot (see below). The game also drew the pocketed balls on the rails in their own colors, which the task forbids, and it had no texts for the players' groups.

Each 27B model passed every hidden test in one mode. Qwen 3.6 27B did it in Instruct mode, Qwen 3.8 27B in Think: medium. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The two setups that passed every hidden test wrote the most tests. Qwen 3.6 27B in Instruct mode wrote 20 tests and Qwen 3.8 27B in Think: medium 12, among them tests of collisions, pockets, groups and the 8-ball. Qwen 3.6 35B in Thinking mode and Qwen 3.8 27B in Instruct mode wrote only 3 tests each, about the start position, aiming and the first frame of a shot. Qwen 3.6 27B in Thinking mode and Qwen 3.6 35B in Instruct mode are not in the chart: they had no game that compiled.

Checks and edits

Qwen 3.6 27B in Instruct mode ran 24 checks, and 20 of them came after all its tests already passed. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Qwen 3.8 27B in Think: medium needed only three checks and three edits. Qwen 3.6 35B in Thinking mode spent most of its checks on compile errors: declarations in C# syntax that SC# does not accept, two variables in one line and a number added to a text.

Unwanted tool calls

Qwen 3.6 27B in Instruct mode repeated an edit ten times that was already applied, so all ten were refused. Qwen 3.6 35B in Thinking mode had seven rejected calls: one refused repeated edit, five calls that referred to text that the answer did not contain and one submission that was rejected because the game did not compile yet. Qwen 3.8 27B had two rejected calls in Instruct mode and one in Think: medium, all of them references to missing text. Qwen 3.6 35B in Instruct mode had one call with broken call syntax.

Tokens

Every tool call sends the whole conversation to the model again, so long runs read far more tokens. Qwen 3.6 27B in Instruct mode read almost 2.5 million tokens, because it used all 50 answers on a game of more than 800 lines. Qwen 3.6 35B in Thinking mode read 1.7 million tokens and wrote the most, because it reasoned at length and needed three reminders to keep calling tools. Qwen 3.8 27B in Think: medium read the fewest tokens of the setups that built a game. Qwen 3.6 27B in Thinking mode read almost nothing: its run ended after the first answer.

One run step by step

Qwen 3.8 27B in Think: medium built the game in nine tool calls and checked its own work against the task.

1. It wrote a first version of 433 lines: the variables, the start of a game, the physics in four sub-steps, the rules at the end of a shot, the drawing and three tests. The check passed all three tests.
2. It added the part B features: the aim guide, a power bar, the controls and the pocketed balls on the rails. The first try referred to text that the answer did not contain and was rejected; the second try worked, and the check passed.
3. Then it compared the new code with the task. It found two problems: the text "PAUSED" was drawn on the cloth, and the pocketed balls on the rails used the ball colors of part A. Both break the rules of part B, so it moved the text and gave the rail markers new colors.
4. It added nine tests: collisions, pockets, groups, a foul, winning and losing with the 8-ball, the power limits and the pause. The check passed all 12.
5. It submitted the game, and all 16 hidden tests passed.

How the models fail

Pocketed or just off the table

The game of Qwen 3.8 27B in Instruct mode ended after every shot in most hidden tests. The task says that the rules at the end of a shot use the balls that fell during this shot, and that tests may take balls off the table before a shot. Qwen 3.8 27B in Instruct mode checked the table instead:

In a game that starts from the beginning, the 8-ball stays on the table until it falls, so the mistake does not show while playing. But most hidden tests place only a few balls on the table, without the 8-ball. In these tests, the game ended after the first shot: the cue ball did not come back after a foul, no group was given and the turn did not pass. Five hidden tests failed for this reason; one of them also checks the texts of the groups, which the game never drew. With this one line fixed, the game passed 14 of 16 hidden tests. The model's three tests all started from a full table, so they could not find it.

A quarter of the physics

The game of Qwen 3.6 35B in Thinking mode did one sub-step per frame instead of four. The task says: "A physics frame has 4 sub-steps, and then friction". The model wrote the steps of one sub-step and the friction in one function and called it once per frame:

void PhysicsStep()
{
    for (int i = 0; i < 16; i++)
    {
        if (ballOnTable[i])
        {
            ballX[i] += ballVX[i] / 4;
            ballY[i] += ballVY[i] / 4;
        }
    }

    pocketedThisShot.Clear();
    // pockets, cushions, collisions, friction

Every ball moved only a quarter of its speed per frame, so all hidden tests that follow a moving ball failed already in the first frame. The second line of the excerpt is the second mistake: the list of balls that fell during the shot was emptied in every step, so the rules at the end of a shot almost never saw a pocketed ball. With both fixed, the game passed 15 of 16 hidden tests.

The model's own test checked the physics, but with the wrong expectation:

void Test_PhysicsMovement()
{
    AppInit();
    ballVX[0] = 4; ballVY[0] = 0;
    PhysicsStep();
    Assert(ballX[0] == 91 && ballY[0] == 120);
}

After one frame with speed 4, the ball must be at 94: four sub-steps of a quarter each. The test expected 91 and passed, because it was written for the same wrong reading of the rule.

Ball colors on the rails

Two setups drew the pocketed balls on the rails in the colors of the balls. Part B asks for the balls each player has pocketed on the rails, and the task also says: "Use only colors that are not used in part A." Qwen 3.8 27B in Instruct mode and Qwen 3.6 35B in Thinking mode used the ball colors anyway:

double cx = startX + slot * 14;
Draw.Circle(cx, y, 5, BallFillColor(i));
if (i >= 9)
{
    Draw.Circle(cx, y, 2, Color.Hex(0xFAFAFA));
}

The hidden test of the drawing looks for each ball by its color. With an extra red mark on the rail, the area of red pixels was much larger than one ball, so the test failed. Qwen 3.8 27B in Think: medium had made the same mistake, but it found it by reading its own code against the task and changed the colors before it submitted.

A loop and a broken file

Two setups lost their runs to a broken answer before a working game existed.

• Qwen 3.6 27B, Thinking mode: in its first answer, the model planned the start of a game and then repeated the same two sentences about the position of the balls 15 times. The runtime stopped the answer as broken output. In this test series, a broken answer ended the run, so no game file existed.
• Qwen 3.6 35B, Instruct mode: the model wrote a first version of 128 lines and checked it once. It fixed the reported error with two edits but did not check again, so a second error stayed: it added numbers to texts with +, which SC# does not allow. Its next answer repeated "making the tool call" and "writing now" for about 200,000 characters without a single call, and the runtime stopped it as broken output. The graded file did not compile.

Conclusion

Qwen 3.6 27B in Instruct mode and Qwen 3.8 27B in Think: medium passed every result.

• Qwen 3.6 27B: all hidden tests passed in Instruct mode, although the model never submitted its finished game. In Thinking mode, the first answer looped and no game existed.
• Qwen 3.8 27B: all hidden tests passed in Think: medium, with only nine tool calls. In Instruct mode, the game ended after every shot when the 8-ball was off the table, and it used the ball colors on the rails.
• Qwen 3.6 35B: in Thinking mode, the physics ran at a quarter of the speed and the end of a shot missed the pocketed balls. In Instruct mode, the game did not compile.

For a game with exact physics like this one, the 27B models in NVFP4 can build a complete version in one attempt. The two setups that passed every test wrote the most tests and covered the rules at the end of a shot. The setups that failed had three tests each, all starting from a new game.

References

Benchmark runs with NVFP4 quantization