Articles

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build Klondike solitaire in SC# as an agent, played with the mouse: write the code and its own tests, check them, fix the problems and submit the game. 15 hidden tests checked the result. Only Qwen 3.8 27B in Think: medium passed every hidden test, Qwen 3.6 27B lost two and three, and most mistakes were in moving several cards at once.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Pac-Man-like maze chase game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 17 hidden tests checked the result. Qwen 3.8 27B in Think: medium passed every hidden test, Qwen 3.8 27B in Instruct mode missed three rules about the order of a frame, and Qwen 3.6 27B in Instruct mode and Qwen 3.6 35B built no working game.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build an 8-ball pool game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 16 hidden tests checked the result. Qwen 3.6 27B in Instruct mode and Qwen 3.8 27B in Think: medium passed every hidden test, Qwen 3.8 27B in Instruct mode lost six, and the other three setups built no working game or one with physics at a quarter of the speed.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Mario-like scrolling platformer in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 14 hidden tests checked the result. Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in Thinking mode failed one hidden test each, Qwen 3.8 27B in Instruct mode failed eight, and Qwen 3.6 35B built no working game.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Super Speed-like top-down racing game with computer opponents in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 17 hidden tests checked the result. Qwen 3.8 27B in Think: medium passed every hidden test, both Qwen 3.6 27B setups lost tests because their computer cars did not finish a lap, and Qwen 3.6 35B built no working game.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Counter-Strike 2D-like top-down shooter in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 17 hidden tests checked the result. Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in Instruct mode passed every hidden test, the other 27B setups lost three or four, and Qwen 3.6 35B built a working game only in Thinking mode.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a checkers game against the computer in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 14 hidden tests checked the result. Qwen 3.8 27B lost one hidden test in each mode, Qwen 3.6 27B in Thinking mode lost three, and Qwen 3.6 35B built no game that compiles; the Qwen 3.6 27B Instruct run was stopped by a server error and has no result.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Battle City-like tank battle game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 16 hidden tests checked the result. Qwen 3.8 27B in Think: medium passed every hidden test, Qwen 3.6 35B in Thinking mode and Qwen 3.6 27B in Instruct mode came close, and the run of Qwen 3.6 27B in Thinking mode was stopped by a server error and has no result.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build an Icy Tower-like tower jumper game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 13 hidden tests checked the result. Qwen 3.6 27B passed every hidden test in both modes, and Qwen 3.6 35B in Thinking mode did the same. The other setups lost tests on the screen and a new game, and Qwen 3.8 27B in Think: medium stopped after a first draft.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Minesweeper-like mine field puzzle in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 15 hidden tests checked the result. Qwen 3.6 27B in Instruct mode and Qwen 3.8 27B in Think: medium passed every hidden test, the other two 27B setups lost one chord test each, and Qwen 3.6 35B built no working game.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Scorched Earth-like turn-based artillery game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 11 hidden tests checked the result. Qwen 3.8 27B passed every hidden test in both modes, Qwen 3.6 35B passed them all in Thinking mode but built no working game in Instruct mode, and Qwen 3.6 27B lost one test in each mode.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Sokoban-like box pushing puzzle in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 14 hidden tests checked the result. Qwen 3.8 27B in Think: medium passed every hidden test, four other setups lost one test each at the transitions between levels, and Qwen 3.6 35B in Instruct mode built no game.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build an Asteroids-like space rocks shooter game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 14 hidden tests checked the result. Qwen 3.6 27B and Qwen 3.8 27B passed every hidden test in both modes, and Qwen 3.6 35B built no game that compiles.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build the sliding tile puzzle 2048 in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 10 hidden tests checked the result. Five of the six setups passed every hidden test; only Qwen 3.6 35B in Instruct mode built no working game, and a wrong test of its own sent Qwen 3.8 27B in Instruct mode on a long detour.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a good-looking Minecraft-like 3D block game in SC# as an agent, and 22 hidden tests checked the rules. Qwen 3.6 27B in Thinking mode and Qwen 3.8 27B in Think: medium lost only one result each, and Qwen 3.6 27B in Instruct mode lost four. Qwen 3.6 35B in Thinking mode built a world where no block can be targeted, and the Instruct modes of Qwen 3.6 35B and Qwen 3.8 27B failed almost every test.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Space Invaders-like fixed shooter game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 16 hidden tests checked the result. Qwen 3.6 27B in both modes and Qwen 3.8 27B in Think: medium passed every hidden test, Qwen 3.8 27B in Instruct mode failed one, and Qwen 3.6 35B submitted no working game, although its first version in Instruct mode was correct.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Tetris-like falling blocks game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 17 hidden tests checked the result. Qwen 3.6 27B passed every hidden test in both modes, Qwen 3.8 27B broke the same soft drop rule in both modes, and Qwen 3.6 35B built no working game.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Supaplex-like digger puzzle game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 14 hidden tests checked the result. Qwen 3.8 27B in Think: medium passed every hidden test, two setups lost three tests each because they typed the start level by hand, and Qwen 3.6 35B built no game that compiles.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a game app with nested menus and screen transitions in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the app. 15 hidden tests checked the result. No setup passed every test: Qwen 3.6 27B lost one or two tests in each mode, Qwen 3.8 27B two or three because of an extra frame after a start or a pause, and Qwen 3.6 35B built no working app.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Diamond Dash-like clicking game in SC# as an agent: a board of diamonds played with the mouse, where a click removes a group of matching diamonds. 14 hidden tests checked the result. Qwen 3.8 27B in Think: medium passed every hidden test, four setups lost one to four, and Qwen 3.6 35B in Instruct mode stopped calling tools and handed in a game without clicks.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build Mahjong solitaire, played with the mouse, in SC# as an agent. 12 hidden tests checked the result. Qwen 3.6 27B in Thinking mode, Qwen 3.6 35B in Thinking mode and Qwen 3.8 27B in Think: medium each lost one hidden test, mostly because two different tile faces looked the same, and Qwen 3.6 35B in Instruct mode built no working game.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Snake game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 14 hidden tests checked the result. Qwen 3.6 35B in Thinking mode and Qwen 3.8 27B in Instruct mode passed every hidden test, three setups failed one, and Qwen 3.8 27B in Think: medium failed three because of one wrong line.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Contra-like side-scrolling run-and-gun game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 15 hidden tests checked the result. No setup passed every hidden test: Qwen 3.6 27B in Thinking mode came closest with 13 of 15, most failed tests came from the order of steps in a frame, and Qwen 3.6 35B built no working game.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Micro Machines-like toy car racing game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 13 hidden tests checked the result. Qwen 3.8 27B in Think: medium passed 12 of 14 results, Qwen 3.8 27B and Qwen 3.6 27B in Instruct mode passed 11, and no setup built computer cars that follow every rule.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Bomberman-like maze bomber game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 14 hidden tests checked the result. Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in Instruct mode passed every hidden test, Qwen 3.6 27B in Thinking mode failed one, and Qwen 3.8 27B in Instruct mode passed only three.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a bubble shooter game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 17 hidden tests checked the result. Qwen 3.6 27B in Thinking mode and Qwen 3.8 27B in Think: medium passed every hidden test, Qwen 3.6 27B in Instruct mode failed one, and Qwen 3.6 35B built no working game.

We test local models by asking them to build small 2D and 3D apps in SC#, controlled by the keyboard and the mouse, as agents: the model writes the app and its own tests, checks and fixes them, and submits the app. Hidden tests that the model never sees then check the rules of the task. This article describes the method in detail: the SC# app model, the tools, the task, the hidden tests, how we make sure the tests are fair, how we grade and what the method does not show.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build an Arkanoid-like brick breaker game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 19 hidden tests checked the result. Qwen 3.8 27B passed every hidden test in both modes. Qwen 3.6 27B failed one test in Thinking mode, and Qwen 3.6 35B failed the same three bonus tests in both modes.

On vLLM 0.30.0, NVFP4 weights, an FP8 KV cache and MTP finally run together without crashes for the Qwen 3.6 and Qwen 3.8 models built on the Qwen 3.5 architecture. On older versions, the same combination crashed, stopped responding under load or needed features turned off, even MTP with FP8 models. This article lists the symptoms that suggest an upgrade, the problems we had on vLLM 0.25.0 to 0.29.0 and working start commands; only Qwen 3.8 Flash Next, on the experimental Qwen 4.0 architecture, still has errors.