Introduction

Can a local model build the game 2048 on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write 2048, the sliding tile puzzle, as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 10 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is 2048 on a 4 x 4 board. The player moves all tiles left, right, up or down with the arrow keys. Two equal tiles that meet merge into one tile with their sum, and after every move that changes the board a new tile appears. The player wins with a 2048 tile, and the game is over when the board is full and no move is possible.

The exact rules the hidden tests check include the position and color of every cell, the game state in a few named variables, a new game, the slide and merge rule with five examples, the order of the steps in a frame (one move per key press, a new tile only after a move that changed the board, then the win and game over checks), the best score and the texts on the screen. The merge rule has one detail that matters below: a tile made by a merge does not merge again in the same move, and the score grows by the value of every tile made by a merge. The model chooses how to build all of this.

Results

Five of the six setups passed every result. Only Qwen 3.6 35B in Instruct mode built no working game. The chart below shows the score of each setup: the share of passed results, the build of the game and the 10 hidden tests.

2048 has fewer rules than most games in this series, and the results show it. The differences are in how the models got there: the number of steps, the quality of their own tests and how they handled a wrong test.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B passed every hidden test in both modes. In Instruct mode, it wrote the whole game with 15 tests in one go and needed only three checks. It was the shortest run of all: 9 tool calls that did work. Its own tests found two real problems and one wrong expectation in a test, and the model fixed the test, not the code (see "One run step by step").

In Thinking mode, the model wrote a first version with 6 tests, fixed one compile error and then added the effects and the texts in 10 more edits, but ran only three checks in total. Between two checks, one of its edits renamed the game loop function by mistake. The model noticed it and renamed it back in its next edit. It needed one reminder before it submitted.

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B passed every hidden test in Thinking mode, but built no working game in Instruct mode. Both modes made the same mistake on the way: they used an out parameter, a C# feature that SC# does not have.

In Thinking mode, the model twice answered with code as text instead of changing the file, and once it pasted the whole file into its answer; it needed three reminders. It removed the out parameter, fixed every call of the function and submitted a short game of 301 lines with only two tests. Two edits that would have rewritten the whole file were refused. In Instruct mode, the model fixed the function but not its four calls, and the game never compiled again (see "How the models fail").

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B passed every hidden test in both modes. In Think: medium, the model built the game in a few large steps and wrote 13 tests. One test expected the wrong score for a merge, and the model fixed the expected value in one edit.

In Instruct mode, the same wrong expectation cost much more. The model changed its correct merge code to fit the wrong test and broke the merge rule. It got back to correct code, but in total it needed 14 checks and 20 edits. In the end, the wrong test was not fixed: it stopped running (see "How the models fail"). The submitted game was correct, but the run read over 1.5 million tokens.

How the models work

The charts in this section show what each model did on the way to its game. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.6 35B in Thinking mode wrote only two tests, and none of them moved a tile, but its game passed every hidden test. Qwen 3.8 27B in Instruct mode ended with four tests because its merge test had stopped running. Qwen 3.6 35B in Instruct mode is not in the chart: its game did not compile.

Checks and edits

Qwen 3.8 27B in Instruct mode needed by far the most checks and edits. A check compiles the game and runs the model's tests. An edit changes some lines of the file. Most of the work of Qwen 3.8 27B in Instruct mode went into the merge rule (see below). Qwen 3.6 27B in Thinking mode made 11 edits but only three checks, so most of its edits were not checked one by one.

Unwanted tool calls

All setups made few unwanted tool calls. Qwen 3.6 35B in Thinking mode had the most: two edits that would have rewritten the whole file and one call that referred to text that the answer did not contain. Qwen 3.8 27B in Think: medium had one refused rewrite, Qwen 3.6 27B in Instruct mode one refused repeated edit and Qwen 3.6 27B in Thinking mode one call that referred to missing text. Qwen 3.8 27B in Instruct mode made 36 calls, and all of them did work.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls read far more tokens. Qwen 3.8 27B in Instruct mode read over 1.5 million tokens, more than twice as many as any other setup. Thinking mode writes more tokens because the model reasons before each step. Qwen 3.6 35B in Instruct mode wrote many tokens for a short run, because its last answers repeated the whole file as text.

One run step by step

Qwen 3.6 27B in Instruct mode wrote the whole game at once and let its tests guide three small fixes.

1. It wrote the full game of 603 lines with 15 tests in one go.
2. The first check passed 8 of 15 tests. Six tests crashed because the code for the merge effect read a list that was still empty when a test called the move function directly. One test used a move that did not change the board, so the move counter correctly stayed at 0. The model fixed the effect code and the test.
3. The second check passed 14 of 15 tests. The failing test expected 16 points for the row [4,4,8,0]. The model read the rule again: only the two 4s merge, so the score is 8. It fixed the test, not the code.
4. It repeated the same edit once, and the repeated edit was refused. The third check passed all 15 tests.
5. Before it submitted, it compared its drawing code with the task and found that the board rectangle was 206 x 206 pixels at (61, 24) instead of 200 x 200 at (60, 24). It fixed the rectangle.
6. It submitted the game, and all 10 hidden tests passed.

How the models fail

A wrong expectation in the model's own test

Three setups wrote the same wrong test: they expected 16 points for the row [4,4,8,0]. The task gives the example "[4,4,8,0] gives [8,8,0,0]" and says that the score grows by the value of every tile made by a merge. Only one tile is made by a merge here: 4 + 4 = 8. The 8 that was already in the row does not count, so the score is 8.

Qwen 3.6 27B in Instruct mode and Qwen 3.8 27B in Think: medium fixed the expected value. Qwen 3.8 27B in Instruct mode trusted its test and changed its correct merge code instead. The new code merged again and again until no equal neighbors were left:

The model's own test for [2,2,2,2] caught the new error in the next check, and the model went back to code that merges each tile once. But the score test still failed. The model decided that the test cases shared state, and it added a helper function that clears the board before each case. It placed the helper between the merge test and the marker that makes a function a test. From then on, the helper counted as a test, and the merge test no longer ran. The number of tests stayed at five, so the change was easy to miss, and two checks later all five passed. Later the model turned the marker above the helper into a plain comment, and only four tests were left. The merge test with the wrong value never ran again.

The submitted game was correct, so this cost only time and tokens here. In a game where the code and the test disagree on a real rule, the same pattern would hide a bug.

A function changed, but not its calls

Qwen 3.6 35B in Instruct mode broke its game with an incomplete fix and then stopped calling tools. It wrote a game of 709 lines at once. To return both a new row and the points of a move, it gave the function SlideLine an out parameter, which SC# does not have. The check reported the error, and the model rewrote the function to keep the points in a script variable. But the four calls of the function still passed the out parameter:

The next check reported the first of these calls. In its next four answers, the model wrote each time that it had to fix the calls of SlideLine. But instead of an edit, every answer repeated the file, whole or in part, as text. After three reminders the run ended, and the graded file did not compile.

Qwen 3.6 35B in Thinking mode made the same out mistake. It changed the function and all of its calls, and its game passed every hidden test.

Conclusion

Five of the six setups built a correct 2048 game: both modes of Qwen 3.6 27B and Qwen 3.8 27B, and Qwen 3.6 35B in Thinking mode.

• Qwen 3.6 27B: all hidden tests passed in both modes. Instruct mode needed the fewest steps of all setups.
• Qwen 3.8 27B: all hidden tests passed in both modes. In Instruct mode, a wrong test led the model away from correct code, and the run needed many more steps and tokens.
• Qwen 3.6 35B: all hidden tests passed in Thinking mode. Instruct mode did not compile after an incomplete fix, and the model stopped calling tools.

For a small game with clear rules like 2048, any of these setups except Qwen 3.6 35B in Instruct mode did the job in our run. The main risk was not the game code but the models' own tests: a wrong expected value can pull a model away from correct code.

References

Benchmark runs with NVFP4 quantization