AI model benchmarks, articles and videos

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build Klondike solitaire in SC# as an agent, played with the mouse: write the code and its own tests, check them, fix the problems and submit the game. 15 hidden tests checked the result. Only Qwen 3.8 27B in Think: medium passed every hidden test, Qwen 3.6 27B lost two and three, and most mistakes were in moving several cards at once.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Pac-Man-like maze chase game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 17 hidden tests checked the result. Qwen 3.8 27B in Think: medium passed every hidden test, Qwen 3.8 27B in Instruct mode missed three rules about the order of a frame, and Qwen 3.6 27B in Instruct mode and Qwen 3.6 35B built no working game.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build an 8-ball pool game in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 16 hidden tests checked the result. Qwen 3.6 27B in Instruct mode and Qwen 3.8 27B in Think: medium passed every hidden test, Qwen 3.8 27B in Instruct mode lost six, and the other three setups built no working game or one with physics at a quarter of the speed.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Mario-like scrolling platformer in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 14 hidden tests checked the result. Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in Thinking mode failed one hidden test each, Qwen 3.8 27B in Instruct mode failed eight, and Qwen 3.6 35B built no working game.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Super Speed-like top-down racing game with computer opponents in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 17 hidden tests checked the result. Qwen 3.8 27B in Think: medium passed every hidden test, both Qwen 3.6 27B setups lost tests because their computer cars did not finish a lap, and Qwen 3.6 35B built no working game.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a Counter-Strike 2D-like top-down shooter in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 17 hidden tests checked the result. Qwen 3.8 27B in Think: medium and Qwen 3.6 27B in Instruct mode passed every hidden test, the other 27B setups lost three or four, and Qwen 3.6 35B built a working game only in Thinking mode.

We asked Qwen 3.6 27B, Qwen 3.6 35B and Qwen 3.8 27B in NVFP4 to build a checkers game against the computer in SC# as an agent: write the code and its own tests, check them, fix the problems and submit the game. 14 hidden tests checked the result. Qwen 3.8 27B lost one hidden test in each mode, Qwen 3.6 27B in Thinking mode lost three, and Qwen 3.6 35B built no game that compiles; the Qwen 3.6 27B Instruct run was stopped by a server error and has no result.