Introduction

Can a local model build a good-looking 3D block game on its own, test it and submit working code? We gave Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B the same task: write a small Minecraft-like block world as an app in SC#, our C#-like scripting language. Each model worked as an agent: it wrote the game, ran its own tests, fixed the code and submitted the game. Then 22 hidden tests checked whether the game follows the rules.

How a run works, how the task and the hidden tests are built and how we make sure the tests are fair is described in How we test local models that build apps as agents. This article covers only the results and the mistakes.

• Models: Qwen 3.6 27B; Qwen 3.6 35B, a mixture-of-experts model with about 3B active parameters; Qwen 3.8 27B, a newer version of the 27B model.
• Modes: Instruct mode and Thinking mode. Qwen 3.8 27B has thinking levels; we ran Think: medium.
• Quantization: NVFP4, a 4-bit format that needs about half the memory of FP8.
• Runtime: vLLM 0.30.0 with matching settings for all models.
• Limit: this task is larger than the 2D games, so a run may use up to 100 model answers instead of 50.

Every setup ran the task once, and that run counts. We did not rerun failed runs. The aim is to show what a user can expect from one attempt, not the best possible result.

The task

The game is a 3D block world of 16 x 16 x 16 cells, seen through the player's eyes. The player walks with the arrow keys, looks around with the mouse or the keys, and jumps with Space. B or the left mouse button breaks the block in front of the crosshair, and P or the right button places a new block on the face the player looks at. Walls, the edge of the world and blocks above the head stop the player. A HUD shows how many blocks were broken and placed and which block type is selected.

The exact rules the hidden tests check include the heights of the starting world, the speed of walking and turning, how walls stop the player on each axis, the jump and the landing on the highest block, the respawn after a fall, which block is targeted within a distance of 5, on which side a new block goes (also when the view meets a block at a slant), the HUD texts and what the screen shows in the middle. Part B asks for the look described above and for smooth drawing in the browser. The model chooses how to build all of this.

Results

Qwen 3.6 27B in Thinking mode and Qwen 3.8 27B in Think: medium lost one result each, and Qwen 3.6 27B in Instruct mode lost four. Qwen 3.6 35B in Thinking mode built a game in which the player cannot target any block. The Instruct modes of Qwen 3.6 35B and Qwen 3.8 27B handed in a game that compiles but fails almost every hidden test. The chart below shows the score of each setup: the share of passed results, the build of the game and the 22 hidden tests.

Qwen 3.6 27B in NVFP4

Both modes built a playable block world, and both turned the view the wrong way when the mouse moves up. In Thinking mode this was the only failed test. In Instruct mode, the new block also went into a wrong cell, see the section "The new block goes into a diagonal cell" below. Instruct mode used all 100 model answers: for the last 83 of them it sent the same answer and the same edit again, and each time the edit was refused because it had already been applied. Its last version compiles, so it was graded.

In FP8, the same model scored about the same in total: all results in Instruct mode, and four lost results in Thinking mode, all of them about the target and the place cell. In this task, NVFP4 does not cost quality.

Qwen 3.6 35B in NVFP4

Thinking mode handed in a textured world, but its ray never finds a block, so the player can neither break nor place. The details are in the section "The ray stays in its first cell" below. Its world also misses one stone cell in every column, because the stone condition stops one cell too early. In the lowest columns this leaves a hole at the bottom. One of its first answers was cut off before the file existed; the model started again and went on normally, so only the game it handed in counts.

Instruct mode stopped after six model answers. It found the real problem of its first version, an empty list of blocks, but then wrote the whole fixed file as text into its answer four times instead of calling the tool that changes the file. After three answers without a tool call, the run ended. The first version compiles, so the build passed, and every hidden test failed.

Qwen 3.8 27B in NVFP4

Think: medium passed every test except one about jumping under a ceiling. Its collision check treats a block that only touches the head as a block that overlaps it. So the jump stops one frame too early, and the next frame the player already falls. The task says that touching faces do not overlap.

Instruct mode handed in a game that stops with an error in its first frame, so every hidden test except one failed. The details are in the section "One division by zero stops every frame" below. It wrote only one own test, which does not run a frame, and it ran the check 53 times.

How the models work

The Instruct modes made the least progress per answer: one run ended after six answers, and two used 79 and 100 model answers with few edits. The thinking modes changed the file in many steps and needed 22 to 56 answers.

Own tests

Qwen 3.6 27B wrote the most own tests in both modes. Qwen 3.8 27B in Instruct mode wrote a single test, which only checks the new world and never runs a frame. The chart below shows how many tests each setup wrote for its game.

Checks and edits

Qwen 3.6 27B in Thinking mode changed the file most often, and Qwen 3.8 27B in Instruct mode checked most often without changing the file. Its 53 checks came with only two edits: it ran the check again and again, and its single test always passed. The chart below shows the checks and the edits of each setup.

Unwanted tool calls

Qwen 3.6 27B in Instruct mode made the most unwanted calls: 83 edits that repeated an edit already applied. The other setups had at most eight unwanted calls. The chart below shows the refused edits, the rejected submissions and the calls in a wrong format.

Tokens

Long runs read the most tokens, because every model answer sends the whole conversation again. Qwen 3.6 27B in Instruct mode read the most, with its 100 answers, and the run that ended after six answers read the least. The thinking modes wrote the most tokens. The charts below show the tokens read and written by each setup.

How the models fail

Most lost tests come from the ray that finds the targeted block and from the side on which a new block goes. These rules need exact 3D geometry, and the models' own tests rarely checked them at a slant.

The new block goes into a diagonal cell

Qwen 3.6 27B in Instruct mode moves the new block one cell along every axis, not only across the face the player looks at. Its ray walks in small steps. When it reaches a block, it compares the point with the middle of the previous cell on all three axes, and each comparison gives -1 or +1, never 0.

Its own test for placing reads the place cell from the game and checks that the new block appears there. It never checks which cell that should be, so the wrong cell passed.

The view goes down when the mouse goes up

Both modes of Qwen 3.6 27B add the vertical mouse movement to the pitch. On the screen, y grows downward, so moving the mouse up gives a negative movement. The view then looks down, while the task asks for the opposite.

Both modes copied the turn to the side correctly and changed the sign only for the vertical movement.

One division by zero stops every frame

Qwen 3.8 27B in Instruct mode guards each division of its ray with a check that is never false. It sets the step to +1 or -1 and then checks that the step is not 0. When the player looks straight ahead, the x part of the direction is exactly 0, and the division fails.

The ray stays in its first cell

Qwen 3.6 35B in Thinking mode moves its ray along one axis only, and only up to the edge of the cell. Each step adds the distance to the next cell edge to one coordinate. When the ray goes toward -Z, it stops exactly on the edge, the same cell is found again, and the next distance is 0. The ray never reaches the block in front of the player.

Without a target, breaking, placing and the HUD counters fail as well. Its three own tests checked the new world, the start position and the block type selection, so none of them used the ray.

Conclusion

Two setups built an almost complete 3D block game: Qwen 3.6 27B in Thinking mode and Qwen 3.8 27B in Think: medium.

• Qwen 3.6 27B: one lost test in Thinking mode and four in Instruct mode, about the mouse look and the place cell. It is the only model with a good result in both modes, and its FP8 version scored about the same.
• Qwen 3.6 35B: Thinking mode built a textured world with a ray that never finds a block. Instruct mode stopped early because it wrote the file as text instead of calling the tool.
• Qwen 3.8 27B: one lost test in Think: medium, the best result together with Qwen 3.6 27B in Thinking mode. Instruct mode handed in a game that stops in its first frame.

For a 3D game of this size, a thinking mode is the safer choice with every tested model. 3D geometry, such as the ray, the place cell and the mouse direction, needs own tests that run real frames at a slant.

The hidden tests do not grade the look. Open the games with "Play in browser" on their benchmark pages to compare the textures, the hotbar and the highlights.

References

Benchmark runs with NVFP4 quantization

Benchmark runs with FP8 quantization (side note)