Introduction

Can a local model build an app with nested menus and screen transitions on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a keyboard app with a main menu, options, high scores, a pause menu and a name entry around a small game, as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the app, ran its own tests, fixed the code and submitted the app. Then 15 hidden tests checked whether the app follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

This test is about the flow of an app, not about game logic. The app has eleven screens: the main menu, options with sound, controls and difficulty, high scores with a reset confirmation, the game, a pause menu, a name entry for a new high score and a goodbye screen. The snake game in the middle is small. Most of the work is in moving between the screens and keeping the right item selected.

The exact rules the hidden tests check include the items of every menu and their order, how Up and Down skip disabled items and wrap around, which item is selected when a screen opens or when the user goes back, how a new key is assigned to a direction and when it is rejected, when the snake makes its steps after a start or a pause, how a new high score is sorted in, and the texts on every screen. The task also lists the variables that hold the whole state of the app: tests read them and may set them directly. The model chooses how to build all of this.

Results

No setup passed every result. Qwen 3.6 27B lost two results in both modes, Qwen 3.8 27B lost three in both modes, and Qwen 3.6 35B built no working app. The chart below shows the score of each setup: the share of passed results, the build of the app and the 15 hidden tests.

Qwen 3.6 27B in NVFP4

Qwen 3.6 27B built a working app in both modes and lost one hidden test in Instruct mode and two in Thinking mode. Both modes kept the item that opens a sub-screen in a separate variable instead of using the fixed item from the task. This broke the return to the options menu when a test opened the controls screen directly (see below). In Thinking mode, the same idea also broke the way back to the pause menu.

In Instruct mode, the run ended early: an answer got stuck in a loop of repeated planning text, and the runtime stopped it as broken output. The last version of the file was graded, and it passed 14 of 15 hidden tests. Because the run ended with a broken answer, the build result does not count as passed. In Thinking mode, the model wrote a large first version, fixed it in several rounds and submitted it with 23 passing tests of its own (see "One run step by step").

Qwen 3.6 35B in NVFP4

Qwen 3.6 35B did not build a working app in either mode. In Instruct mode, the app compiled and its menus worked in part, but the game screen was handled by the code of the pause menu: there was no code that moves the snake. In Thinking mode, the model wrote the whole app without a single check, and the file did not compile. Its next answer got stuck in a loop and was stopped as broken output. Both runs are described in "How the models fail".

Qwen 3.8 27B in NVFP4

Qwen 3.8 27B built a working app in both modes and lost two hidden tests in both modes because of the same idea about the start of the game. After the game starts or resumes from the pause menu, both of its apps skip one more frame than the task asks (see below). The first snake step therefore comes one frame too late. In Think: medium, the skipped frame also swallowed a key press, so one more test failed.

In Instruct mode, the third lost test was a text: when a key is already used by another direction, the app showed the name of the direction instead of the key. The Instruct run also needed the most edits of all setups.

Qwen 3.6 27B and Qwen 3.8 27B came close, but each model missed one rule about state in both of its modes. Both models ran in NVFP4 with the same settings. With one run each, this is a hint, not a proof.

How the models work

The charts in this section show what each model did on the way to its app. Fewer steps are not better by themselves: what counts is the result of the hidden tests.

Own tests

The number of tests the models wrote did not predict the result. Qwen 3.6 27B in Thinking mode wrote 23 tests and still missed two rules. Qwen 3.6 27B in Instruct mode wrote only 3 tests, and its last file lost one hidden test. All tests of Qwen 3.8 27B passed, and they even checked that the step counter keeps its value during a pause, but they checked it with the same extra frame the app had. Qwen 3.6 35B in Thinking mode is not in the chart: it never ran a check, and its file did not compile.

Checks and edits

Qwen 3.8 27B in Instruct mode needed the most edits, and Qwen 3.6 27B in Thinking mode the most checks. A check compiles the app and runs the model's tests. An edit changes some lines of the file. Qwen 3.8 27B in Instruct mode started with a short skeleton and added the menus and the game in many steps. Qwen 3.6 35B made almost no checks: two in Instruct mode and none in Thinking mode.

Unwanted tool calls

Few calls were wasted in this test. Qwen 3.6 35B in Instruct mode tried three times to rewrite the whole file, which is not allowed, so the calls were refused. After that, it answered three times without any tool call and needed five reminders before the run ended. Qwen 3.8 27B in Instruct mode had three rejected calls that referred to text that the answer did not contain. Qwen 3.6 27B in Instruct mode had one such call, and Qwen 3.8 27B in Think: medium one call with broken call syntax. Qwen 3.6 27B in Thinking mode and Qwen 3.6 35B in Thinking mode made no wrong calls.

Tokens

Every tool call sends the whole conversation to the model again, so runs with many tool calls and a long file read far more tokens. Qwen 3.6 27B in Thinking mode read the most, about 1.4 million tokens: its file grew to more than 1,100 lines. Qwen 3.6 35B in Thinking mode wrote the most tokens, but most of them were the loop at the end of its run.

One run step by step

Qwen 3.6 27B in Thinking mode wrote a large first version and then let its checks and tests guide the fixes.

1. It wrote the whole app at once, about 900 lines with four tests.
2. The checks found code that SC# does not accept, one kind after another: a function name with a dot, a list method that SC# does not have and color values of the wrong number type. The model fixed each of them.
3. Three of its four tests then failed with an error in the drawing code. It rewrote the drawing of the texts, and one test was left: the volume did not change. The model changed this test, and all four passed.
4. It added 19 more tests, and nine of them failed.
5. While fixing them, it added a second copy of many tests by mistake. The check reported the duplicate names, and the model deleted the copy.
6. The remaining failures showed two real bugs: the high score screen read past the end of a list, and the app did not end after Quit. The model fixed both, corrected one of its tests, added more tests, and all 23 passed.
7. It submitted the app, and 13 of 15 hidden tests passed. The two failures came from its way of remembering where to go back to (see below).

How the models fail

One frame too many after a start or a pause

Both apps of Qwen 3.8 27B skip the first frame of the game after it starts or resumes. The task says that the frame in which the game starts, and the frame in which it resumes from the pause menu, do nothing else. In that frame the menu handles the Enter key. The next frame is already a normal game frame: the step counter grows, and with the Normal speed the snake steps in the sixth frame. Qwen 3.8 27B sets a flag when the game starts or resumes, and the flag skips one more frame:

The same flag broke the test of the pause and resume. In Think: medium the flag is checked before the pause key, so a P pressed in the first game frame was ignored, and a third test failed. The Instruct app checks the pause key first, so it lost only the two step tests. The tests of Qwen 3.8 27B checked the counter after a pause, but with values from the app itself, so they confirmed the extra frame.

The way back is remembered in a variable

Both apps of Qwen 3.6 27B remember which item opened a sub-screen instead of using the fixed item from the task. The task says that Back and Escape on the sound, controls and difficulty screens return to the options menu with the item that opened them selected: 0, 1 or 2. Each of these screens is opened by only one item, so the item to select is fixed. The task also lists the variables that hold the whole state and says that tests may set them.

The hidden test opens the controls screen by setting the screen variable. Escape must then return to the options menu with Controls selected. Both apps returned with the first item selected, because the variable they kept for the way back still had its starting value 0:

In Thinking mode, the same idea broke one more rule. The app had one pair of variables for the way back and used it for two levels of menus. When the options menu was opened from the pause menu and then the sound screen from the options menu, the second step overwrote the first. After two Escape presses the user should be back in the pause menu with Options selected, but the app stayed in the options menu.

A wrong text for a key in use

The Instruct app of Qwen 3.8 27B showed the wrong word when a key was already in use. The task asks for the text "KEY IN USE: {key}", for example "KEY IN USE: W" when the user tries to give W to a second direction. The app wrote the name of the direction that was waiting for a key, "KEY IN USE: Right". The rest of the key assignment worked: the key was rejected, and the app kept waiting.

An app without a game and a file that never compiled

Qwen 3.6 35B lost both runs before it had a working app.

• Instruct mode: the model wrote about 540 lines at once. Its main loop sent the game screen to the code of the pause menu, and the file had no code that moves the snake, so all game tests failed. Back on the sound screen went to the main menu instead of the options menu, and the app never ended after Quit. After its first fix, the model tried three times to rewrite the whole file, and these calls were refused. It then answered three times without a tool call, and the run ended without a submitted app. The last file passed 3 of 15 hidden tests.
• Thinking mode: the model wrote the app in five steps without a single check. The file used an array type from C# that SC# does not have, so it did not compile. In the next answer, the model checked rule after rule in its reasoning and repeated "OK, WRITING THE RESPONSE NOW." until the runtime stopped the answer as broken output. In this test series, a broken answer ended the run.

Conclusion

No setup passed every hidden test. Qwen 3.6 27B came closest, and Qwen 3.8 27B was one test behind.

• Qwen 3.6 27B: one hidden test lost in Instruct mode and two in Thinking mode, all because the way back to a menu was kept in a variable. The Instruct run ended with a broken answer, so its build result does not count.
• Qwen 3.8 27B: two hidden tests lost in Instruct mode and three in Think: medium. Both apps skipped an extra frame after a start or a pause; the Instruct app also showed a wrong text.
• Qwen 3.6 35B: no working app in either mode: a missing game in Instruct mode and a file that never compiled in Thinking mode.

For an app with many screens like this one, Qwen 3.6 27B and Qwen 3.8 27B in NVFP4 build most of the flow correctly in one attempt. The rules that went wrong were about state and timing. Tests that set the state directly and count frames from the key press would have found them.

References

Benchmark runs with NVFP4 quantization