Ai4Users - LLM Benchmarks

LLM benchmarks with code generation in SC#

In coding-heavy tests, the LLM writes SC# code, adding an extra challenge: it must understand and follow a custom language specification included in the prompt context.

LLM Benchmarks
Model	Date	Status	Tokens	Score
SC# Coding & Agentic Benchmark by Ai4Users A custom Ai4Users benchmark suite for SC#, covering code generation, reasoning, and stateful agentic workflows where models use tools, observe results, adapt their actions, recover from failures, and verify the final state.
Qwen 3.6 27BFP8Thinking338 AI queries 523 tool calls 331 total 327 passed3 failed1 broken	2026-09-20	Finished	IN3 759 256 OUT781 494	98.8%
Qwen 3.6 27BNVFP4Thinking338 AI queries 522 tool calls 331 total 327 passed4 failed	2026-09-24	Finished	IN3 710 161 OUT739 830	98.8%
Qwen 3.8 27BNVFP4Think: medium343 AI queries 523 tool calls 331 total 326 passed5 failed	2026-09-24	Finished	IN3 745 894 OUT622 365	98.5%
Qwen 3.8 27BFP8Think: medium337 AI queries 514 tool calls 331 total 325 passed6 failed	2026-09-22	Finished	IN3 492 677 OUT743 750	98.2%
Qwen 3.8 Flash NextNVFP4Instruct353 AI queries 548 tool calls 331 total 323 passed8 failed	2026-09-26	Finished	IN4 398 019 OUT122 411	97.6%
Qwen 3.6 35BFP8Thinking375 AI queries 1157 tool calls 331 total 317 passed12 failed2 broken	2026-09-20	Finished	IN4 619 486 OUT1 152 416	95.8%
Qwen 3.6 27BFP8Instruct390 AI queries 537 tool calls 331 total 314 passed17 failed	2026-09-20	Finished	IN4 395 058 OUT152 253	94.9%
Qwen 3.6 35BNVFP4Thinking372 AI queries 844 tool calls 331 total 304 passed25 failed2 broken	2026-09-21	Finished	IN3 836 208 OUT983 071	91.8%
Qwen 3.8 27BFP8Instruct396 AI queries 1732 tool calls 331 total 302 passed23 failed6 broken	2026-09-22	Finished	IN4 695 066 OUT205 427	91.2%
Qwen 3.8 27BNVFP4Instruct413 AI queries 500 tool calls 331 total 301 passed28 failed2 broken	2026-09-24	Finished	IN4 682 064 OUT223 859	90.9%
Qwen 3.6 27BNVFP4Instruct389 AI queries 493 tool calls 331 total 298 passed29 failed4 broken	2026-09-24	Finished	IN3 951 507 OUT207 577	90.0%
Qwen 3.6 35BFP8Instruct523 AI queries 495 tool calls 331 total 295 passed36 failed	2026-09-20	Finished	IN6 119 641 OUT521 456	89.1%
Qwen 3.6 35BNVFP4Instruct487 AI queries 527 tool calls 331 total 291 passed39 failed1 broken	2026-09-21	Finished	IN5 602 042 OUT447 555	87.9%
Qwen 3.8 Flash NextNVFP4Think: medium857 AI queries 372 tool calls 331 total 152 passed176 failed3 broken	2026-09-26	Finished	IN8 520 134 OUT1 758 976	45.9%
HumanEval for SC# HumanEval adapted to SC#, adding an extra challenge: models must generate correct code in a custom sandboxed language while following its documentation and constraints.
Qwen 3.6 27BFP8Thinking162 AI queries 0 tool calls 162 total 161 passed1 failed	2026-09-20	Finished	IN1 433 411 OUT296 690	99.4%
Qwen 3.6 27BNVFP4Thinking162 AI queries 0 tool calls 162 total 158 passed4 failed	2026-09-24	Finished	IN1 412 351 OUT306 278	97.5%
Qwen 3.8 27BFP8Think: medium162 AI queries 0 tool calls 162 total 157 passed5 failed	2026-09-22	Finished	IN1 412 351 OUT219 912	96.9%
Qwen 3.8 27BNVFP4Think: medium162 AI queries 0 tool calls 162 total 156 passed5 failed1 broken	2026-09-24	Finished	IN1 412 351 OUT185 809	96.3%
Qwen 3.8 Flash NextNVFP4Instruct162 AI queries 0 tool calls 162 total 155 passed7 failed	2026-09-26	Finished	IN1 412 675 OUT28 074	95.7%
Qwen 3.6 35BFP8Thinking162 AI queries 0 tool calls 162 total 153 passed9 failed	2026-09-20	Finished	IN1 412 351 OUT354 618	94.4%
Qwen 3.6 35BNVFP4Thinking162 AI queries 0 tool calls 162 total 151 passed10 failed1 broken	2026-09-21	Finished	IN1 412 351 OUT321 944	93.2%
Qwen 3.6 27BNVFP4Instruct162 AI queries 0 tool calls 162 total 142 passed20 failed	2026-09-24	Finished	IN1 412 675 OUT24 745	87.7%
Qwen 3.6 27BFP8Instruct162 AI queries 0 tool calls 162 total 140 passed22 failed	2026-09-20	Finished	IN1 433 735 OUT24 438	86.4%
Qwen 3.8 27BNVFP4Instruct162 AI queries 13 tool calls 162 total 132 passed30 failed	2026-09-24	Finished	IN1 552 852 OUT40 751	81.5%
Qwen 3.6 35BNVFP4Instruct162 AI queries 0 tool calls 162 total 124 passed38 failed	2026-09-21	Finished	IN1 412 675 OUT120 409	76.5%
Qwen 3.6 35BFP8Instruct162 AI queries 0 tool calls 162 total 121 passed41 failed	2026-09-20	Finished	IN1 412 675 OUT155 599	74.7%
Qwen 3.8 27BFP8Instruct162 AI queries 50 tool calls 162 total 120 passed42 failed	2026-09-22	Finished	IN1 962 595 OUT31 668	74.1%
Qwen 3.8 Flash NextNVFP4Think: medium162 AI queries 0 tool calls 162 total 35 passed126 failed1 broken	2026-09-26	Finished	IN1 417 211 OUT186 262	21.6%
LiveCodeBench for SC# LiveCodeBench adapted to SC#, adding an extra challenge: models must solve newer coding problems while generating correct code in a custom sandboxed language from its provided specification.
Qwen 3.8 27BFP8Think: medium1022 AI queries 0 tool calls 1054 total 856 passed79 failed87 broken32 skipped	2026-09-22	Finished	IN9 425 269 OUT17 601 626	83.8%
Qwen 3.8 27BNVFP4Think: medium1022 AI queries 0 tool calls 1054 total 856 passed136 failed30 broken32 skipped	2026-09-24	Finished	IN9 425 269 OUT10 762 192	83.8%
Qwen 3.6 27BFP8Thinking1022 AI queries 0 tool calls 1054 total 845 passed129 failed48 broken32 skipped	2026-09-20	Finished	IN9 425 269 OUT9 838 473	82.7%
Qwen 3.6 27BNVFP4Thinking1022 AI queries 0 tool calls 1054 total 843 passed118 failed61 broken32 skipped	2026-09-24	Finished	IN9 425 269 OUT10 512 921	82.5%
Qwen 3.6 35BNVFP4Thinking1022 AI queries 0 tool calls 1054 total 654 passed293 failed75 broken32 skipped	2026-09-21	Finished	IN9 425 269 OUT11 662 666	64.0%
Qwen 3.6 35BFP8Thinking1022 AI queries 0 tool calls 1054 total 620 passed350 failed52 broken32 skipped	2026-09-21	Finished	IN9 425 269 OUT9 777 745	60.7%
Qwen 3.8 Flash NextNVFP4Instruct1022 AI queries 0 tool calls 1054 total 550 passed472 failed32 skipped	2026-09-26	Finished	IN9 457 973 OUT728 528	53.8%
Qwen 3.6 27BFP8Instruct1022 AI queries 0 tool calls 1054 total 524 passed490 failed8 broken32 skipped	2026-09-20	Finished	IN9 560 173 OUT1 548 300	51.3%
Qwen 3.8 27BNVFP4Instruct1022 AI queries 115 tool calls 1054 total 471 passed551 failed32 skipped	2026-09-24	Finished	IN11 028 946 OUT1 355 610	46.1%
Qwen 3.6 27BNVFP4Instruct1022 AI queries 0 tool calls 1054 total 443 passed578 failed1 broken32 skipped	2026-09-24	Finished	IN9 427 313 OUT882 428	43.3%
Qwen 3.8 27BFP8Instruct1022 AI queries 106 tool calls 1054 total 440 passed579 failed3 broken32 skipped	2026-09-22	Finished	IN10 904 145 OUT981 661	43.1%
Qwen 3.6 35BNVFP4Instruct1022 AI queries 0 tool calls 1054 total 439 passed558 failed25 broken32 skipped	2026-09-21	Finished	IN9 427 313 OUT7 277 353	43.0%
Qwen 3.6 35BFP8Instruct1022 AI queries 0 tool calls 1054 total 407 passed601 failed14 broken32 skipped	2026-09-21	Finished	IN9 427 313 OUT6 068 143	39.8%
Instruction Following Benchmark based on IFBench IFBench-based benchmark for precise instruction following, ported to the SC# evaluation runtime. It checks whether models follow verifiable formatting, structure, wording, length, and other response constraints.
Qwen 3.8 27BFP8Think: medium271 AI queries 0 tool calls 300 total 185 passed85 failed1 broken29 skipped	2026-09-22	Finished	IN255 833 OUT1 438 869	68.3%
Qwen 3.6 27BNVFP4Thinking271 AI queries 0 tool calls 300 total 177 passed58 failed36 broken29 skipped	2026-09-24	Finished	IN255 833 OUT2 259 275	65.3%
Qwen 3.8 27BNVFP4Think: medium271 AI queries 0 tool calls 300 total 173 passed96 failed2 broken29 skipped	2026-09-24	Finished	IN255 833 OUT1 273 038	63.8%
Qwen 3.6 27BFP8Thinking271 AI queries 0 tool calls 300 total 170 passed80 failed21 broken29 skipped	2026-09-20	Finished	IN225 823 OUT1 225 926	62.7%
Qwen 3.6 35BNVFP4Thinking271 AI queries 0 tool calls 300 total 158 passed62 failed51 broken29 skipped	2026-09-21	Finished	IN255 833 OUT2 494 363	58.3%
Qwen 3.6 35BFP8Thinking271 AI queries 0 tool calls 300 total 151 passed77 failed43 broken29 skipped	2026-09-20	Finished	IN255 833 OUT2 000 342	55.7%
Qwen 3.8 Flash NextNVFP4Instruct271 AI queries 0 tool calls 300 total 123 passed148 failed29 skipped	2026-09-26	Finished	IN256 375 OUT59 835	45.4%
Qwen 3.6 27BNVFP4Instruct271 AI queries 0 tool calls 300 total 121 passed137 failed13 broken29 skipped	2026-09-24	Finished	IN256 375 OUT319 835	44.6%
Qwen 3.6 27BFP8Instruct271 AI queries 0 tool calls 300 total 109 passed154 failed8 broken29 skipped	2026-09-20	Finished	IN254 388 OUT96 948	40.2%
Qwen 3.6 35BFP8Instruct271 AI queries 0 tool calls 300 total 104 passed155 failed12 broken29 skipped	2026-09-20	Finished	IN256 375 OUT136 048	38.4%
Qwen 3.8 27BFP8Instruct271 AI queries 0 tool calls 300 total 100 passed167 failed4 broken29 skipped	2026-09-22	Finished	IN256 375 OUT105 304	36.9%
Qwen 3.8 27BNVFP4Instruct271 AI queries 0 tool calls 300 total 99 passed171 failed1 broken29 skipped	2026-09-24	Finished	IN256 375 OUT113 252	36.5%
Qwen 3.6 35BNVFP4Instruct271 AI queries 0 tool calls 300 total 98 passed170 failed3 broken29 skipped	2026-09-21	Finished	IN256 375 OUT121 405	36.2%
Tool Calling Benchmark based on BFCL A BFCL-based tool-calling benchmark ported to the SC# evaluation runtime. It tests whether the model selects the correct function, uses the right arguments, and avoids calling tools that do not match the request. Each test can see all tools declared in its script, so the model often has more tools to choose from than in the original case.
Qwen 3.6 27BNVFP4Instruct1248 AI queries 1378 tool calls 1265 total 1209 passed37 failed2 broken17 skipped	2026-09-24	Finished	IN8 645 724 OUT257 654	96.9%
Qwen 3.8 27BFP8Think: medium1248 AI queries 1375 tool calls 1265 total 1203 passed45 failed17 skipped	2026-09-22	Finished	IN8 644 462 OUT749 731	96.4%
Qwen 3.6 27BFP8Instruct1248 AI queries 1391 tool calls 1265 total 1202 passed46 failed17 skipped	2026-09-23	Finished	IN8 696 455 OUT235 706	96.3%
Qwen 3.6 27BNVFP4Thinking1248 AI queries 1386 tool calls 1265 total 1201 passed47 failed17 skipped	2026-09-24	Finished	IN8 694 701 OUT930 527	96.2%
Qwen 3.8 27BNVFP4Think: medium1248 AI queries 1379 tool calls 1265 total 1200 passed48 failed17 skipped	2026-09-24	Finished	IN8 659 866 OUT693 101	96.2%
Qwen 3.8 27BFP8Instruct1248 AI queries 1367 tool calls 1265 total 1181 passed66 failed1 broken17 skipped	2026-09-22	Finished	IN8 675 108 OUT227 280	94.6%
Qwen 3.6 27BFP8Thinking1248 AI queries 1433 tool calls 1265 total 1157 passed89 failed2 broken17 skipped	2026-09-23	Finished	IN8 864 246 OUT763 817	92.7%
Qwen 3.8 Flash NextNVFP4Instruct1248 AI queries 1421 tool calls 1265 total 1156 passed92 failed17 skipped	2026-09-26	Finished	IN8 952 270 OUT198 020	92.6%
Qwen 3.8 27BNVFP4Instruct1248 AI queries 1356 tool calls 1265 total 1155 passed89 failed4 broken17 skipped	2026-09-24	Finished	IN8 672 294 OUT217 982	92.5%
Qwen 3.6 35BNVFP4Thinking1248 AI queries 1455 tool calls 1265 total 1151 passed95 failed2 broken17 skipped	2026-09-21	Finished	IN8 931 672 OUT1 897 177	92.2%
Qwen 3.6 35BFP8Thinking1248 AI queries 1416 tool calls 1265 total 1150 passed98 failed17 skipped	2026-09-21	Finished	IN8 782 803 OUT1 524 049	92.1%
Qwen 3.6 35BNVFP4Instruct1248 AI queries 8041 tool calls 1265 total 359 passed886 failed3 broken17 skipped	2026-09-21	Finished	IN40 059 390 OUT951 380	28.8%
Qwen 3.6 35BFP8Instruct1248 AI queries 13081 tool calls 1265 total 344 passed899 failed5 broken17 skipped	2026-09-21	Finished	IN84 747 746 OUT1 478 683	27.6%
Reasoning Benchmark based on BBH BBH-based reasoning benchmark ported to the SC# evaluation runtime. It tests multi-step reasoning across challenging tasks involving logic, deduction, symbolic manipulation, planning, and problem solving.
Qwen 3.6 27BNVFP4Instruct5388 AI queries 0 tool calls 5390 total 5186 passed202 failed2 skipped	2026-09-24	Finished	IN5 646 466 OUT3 141 819	96.3%
Qwen 3.8 Flash NextNVFP4Instruct5388 AI queries 0 tool calls 5390 total 5172 passed216 failed2 skipped	2026-09-26	Finished	IN5 646 466 OUT2 801 620	96.0%
Qwen 3.6 27BFP8Instruct5388 AI queries 0 tool calls 5390 total 5158 passed229 failed1 broken2 skipped	2026-09-23	Finished	IN5 646 466 OUT3 235 305	95.7%
Qwen 3.6 35BFP8Thinking5388 AI queries 0 tool calls 5390 total 5156 passed224 failed8 broken2 skipped	2026-09-22	Finished	IN5 635 690 OUT7 741 072	95.7%
Qwen 3.6 35BFP8Instruct5388 AI queries 0 tool calls 5390 total 5154 passed233 failed1 broken2 skipped	2026-09-22	Finished	IN5 646 466 OUT3 620 827	95.7%
Qwen 3.6 27BFP8Thinking5388 AI queries 0 tool calls 5390 total 5145 passed234 failed9 broken2 skipped	2026-09-23	Finished	IN5 635 690 OUT4 894 538	95.5%
Qwen 3.6 35BNVFP4Thinking5388 AI queries 0 tool calls 5390 total 5144 passed234 failed10 broken2 skipped	2026-09-22	Finished	IN5 635 690 OUT8 172 715	95.5%
Qwen 3.6 27BNVFP4Thinking5388 AI queries 0 tool calls 5390 total 5143 passed240 failed5 broken2 skipped	2026-09-24	Finished	IN5 635 690 OUT5 282 204	95.5%
Qwen 3.6 35BNVFP4Instruct5388 AI queries 0 tool calls 5390 total 5135 passed253 failed2 skipped	2026-09-22	Finished	IN5 646 466 OUT3 704 727	95.3%
Qwen 3.8 27BNVFP4Think: medium5388 AI queries 0 tool calls 5390 total 5079 passed309 failed2 skipped	2026-09-24	Finished	IN5 635 690 OUT3 199 564	94.3%
Qwen 3.8 27BFP8Instruct5388 AI queries 0 tool calls 5390 total 5052 passed334 failed2 broken2 skipped	2026-09-22	Finished	IN5 646 466 OUT3 002 152	93.8%
Qwen 3.8 27BFP8Think: medium5388 AI queries 0 tool calls 5390 total 5049 passed339 failed2 skipped	2026-09-22	Finished	IN5 635 690 OUT3 166 389	93.7%
Qwen 3.8 27BNVFP4Instruct5388 AI queries 0 tool calls 5390 total 5026 passed362 failed2 skipped	2026-09-24	Finished	IN5 646 466 OUT2 719 787	93.3%
Financial Reasoning Benchmark based on FinQA FinQA-based benchmark for numerical reasoning over financial reports, ported to the SC# evaluation runtime. Each test gives the model an excerpt from a company's annual report, with text and tables, and asks a question that needs the right figures and a correct calculation. During migration we checked every disputed task by hand: wrong FinQA reference answers were corrected and ambiguous questions were clarified.
Qwen 3.6 27BFP8Instruct1123 AI queries 0 tool calls 1125 total 1105 passed18 failed2 skipped	2026-10-04	Finished	IN2 510 516 OUT663 640	98.4%
Qwen 3.6 27BFP8Thinking1123 AI queries 0 tool calls 1125 total 1098 passed21 failed4 broken2 skipped	2026-10-04	Finished	IN2 508 270 OUT1 807 530	97.8%
2048 puzzle built in SC# as an agent The model builds 2048, the sliding tile puzzle in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct1 AI query 10 tool calls 11 total 11 passed	2026-10-10	Finished	IN341 066 OUT8 022	100.0%
Qwen 3.6 27BNVFP4Thinking2 AI queries 17 tool calls 11 total 11 passed	2026-10-10	Finished	IN600 419 OUT54 419	100.0%
Qwen 3.8 27BNVFP4Instruct1 AI query 36 tool calls 11 total 11 passed	2026-10-10	Finished	IN1 504 642 OUT14 020	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 15 tool calls 11 total 11 passed	2026-10-10	Finished	IN512 412 OUT64 321	100.0%
Qwen 3.6 35BNVFP4Thinking4 AI queries 15 tool calls 11 total 11 passed	2026-10-10	Finished	IN660 505 OUT58 988	100.0%
Qwen 3.6 35BNVFP4Instruct4 AI queries 5 tool calls 11 total 0 passed11 failed	2026-10-10	Finished	IN340 531 OUT28 962	0.0%
8-ball pool game built in SC# as an agent The model builds an 8-ball pool game, seen from the top in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct1 AI query 49 tool calls 17 total 17 passed	2026-10-10	Finished	IN2 492 152 OUT18 017	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 9 tool calls 17 total 17 passed	2026-10-10	Finished	IN306 067 OUT24 009	100.0%
Qwen 3.8 27BNVFP4Instruct1 AI query 19 tool calls 17 total 11 passed6 failed	2026-10-10	Finished	IN698 210 OUT9 044	64.7%
Qwen 3.6 35BNVFP4Thinking4 AI queries 26 tool calls 17 total 4 passed13 failed	2026-10-10	Finished	IN1 709 158 OUT80 516	23.5%
Qwen 3.6 27BNVFP4Thinking1 AI query 0 tool calls 17 total 0 passed17 broken	2026-10-10	Finished	IN21 212 OUT4 495	0.0%
Qwen 3.6 35BNVFP4Instruct1 AI query 5 tool calls 17 total 0 passed16 failed1 broken	2026-10-10	Finished	IN161 086 OUT66 559	0.0%
Arkanoid-like game built in SC# as an agent The model builds an Arkanoid-like brick breaker game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct1 AI query 39 tool calls 20 total 20 passed	2026-10-09	Finished	IN1 932 087 OUT24 541	100.0%
Qwen 3.8 27BNVFP4Instruct1 AI query 10 tool calls 20 total 20 passed	2026-10-09	Finished	IN319 660 OUT5 033	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 3 tool calls 20 total 20 passed	2026-10-09	Finished	IN120 452 OUT53 459	100.0%
Qwen 3.6 27BFP8Instruct1 AI query 10 tool calls 20 total 19 passed1 failed	2026-10-09	Finished	IN443 552 OUT10 824	95.0%
Qwen 3.6 27BFP8Thinking1 AI query 7 tool calls 20 total 19 passed1 failed	2026-10-09	Finished	IN267 874 OUT40 844	95.0%
Qwen 3.6 27BNVFP4Thinking1 AI query 35 tool calls 20 total 19 passed1 failed	2026-10-09	Finished	IN1 440 351 OUT34 336	95.0%
Qwen 3.6 35BNVFP4Thinking1 AI query 23 tool calls 20 total 17 passed3 failed	2026-10-09	Finished	IN1 257 344 OUT50 508	85.0%
Qwen 3.6 35BNVFP4Instruct1 AI query 10 tool calls 20 total 16 passed3 failed1 broken	2026-10-09	Finished	IN491 668 OUT28 218	80.0%
Asteroids-like game built in SC# as an agent The model builds an Asteroids-like space rocks shooter with power-ups in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct2 AI queries 25 tool calls 15 total 15 passed	2026-10-10	Finished	IN1 142 158 OUT17 162	100.0%
Qwen 3.6 27BNVFP4Thinking1 AI query 17 tool calls 15 total 15 passed	2026-10-10	Finished	IN789 237 OUT46 792	100.0%
Qwen 3.8 27BNVFP4Instruct1 AI query 29 tool calls 15 total 15 passed	2026-10-10	Finished	IN1 046 799 OUT18 285	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 12 tool calls 15 total 15 passed	2026-10-10	Finished	IN461 869 OUT55 863	100.0%
Qwen 3.6 35BNVFP4Instruct4 AI queries 3 tool calls 15 total 0 passed15 failed	2026-10-10	Finished	IN257 033 OUT33 053	0.0%
Qwen 3.6 35BNVFP4Thinking1 AI query 2 tool calls 15 total 0 passed14 failed1 broken	2026-10-10	Finished	IN70 889 OUT71 952	0.0%
Battle City-like game built in SC# as an agent The model builds a Battle City-like tank battle game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.8 27BNVFP4Think: medium1 AI query 20 tool calls 17 total 17 passed	2026-10-10	Finished	IN847 727 OUT67 352	100.0%
Qwen 3.6 35BNVFP4Thinking5 AI queries 45 tool calls 17 total 15 passed2 failed	2026-10-10	Finished	IN4 056 176 OUT163 739	88.2%
Qwen 3.6 27BNVFP4Instruct1 AI query 47 tool calls 17 total 14 passed3 failed	2026-10-10	Finished	IN3 508 073 OUT62 923	82.4%
Qwen 3.8 27BNVFP4Instruct1 AI query 28 tool calls 17 total 10 passed6 failed1 broken	2026-10-10	Finished	IN1 167 194 OUT19 269	58.8%
Qwen 3.6 27BNVFP4Thinking1 AI query 0 tool calls 17 total 0 passed17 failed	2026-10-10	Finished	IN0 OUT0	0.0%
Qwen 3.6 35BNVFP4Instruct4 AI queries 6 tool calls 17 total 0 passed17 failed	2026-10-10	Finished	IN537 583 OUT55 170	0.0%
Bomberman-like game built in SC# as an agent The model builds a Bomberman-like maze bomber game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct1 AI query 50 tool calls 15 total 15 passed	2026-10-10	Finished	IN3 408 498 OUT36 128	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 15 tool calls 15 total 15 passed	2026-10-10	Finished	IN519 721 OUT134 124	100.0%
Qwen 3.6 27BNVFP4Thinking1 AI query 15 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN522 702 OUT19 618	93.3%
Qwen 3.6 35BNVFP4Instruct4 AI queries 3 tool calls 15 total 13 passed2 failed	2026-10-10	Finished	IN256 855 OUT32 684	86.7%
Qwen 3.6 35BNVFP4Thinking2 AI queries 28 tool calls 15 total 7 passed8 failed	2026-10-10	Finished	IN1 308 802 OUT88 388	46.7%
Qwen 3.8 27BNVFP4Instruct1 AI query 23 tool calls 15 total 4 passed11 failed	2026-10-10	Finished	IN866 928 OUT16 550	26.7%
Bubble shooter game built in SC# as an agent The model builds a bubble shooter game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Thinking1 AI query 25 tool calls 18 total 18 passed	2026-10-10	Finished	IN1 023 777 OUT53 442	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 18 tool calls 18 total 18 passed	2026-10-10	Finished	IN852 039 OUT46 985	100.0%
Qwen 3.6 27BNVFP4Instruct1 AI query 15 tool calls 18 total 17 passed1 failed	2026-10-10	Finished	IN600 738 OUT13 339	94.4%
Qwen 3.8 27BNVFP4Instruct1 AI query 47 tool calls 18 total 7 passed11 failed	2026-10-10	Finished	IN2 117 303 OUT21 578	38.9%
Qwen 3.6 35BNVFP4Instruct6 AI queries 6 tool calls 18 total 0 passed18 failed	2026-10-10	Finished	IN572 608 OUT55 092	0.0%
Qwen 3.6 35BNVFP4Thinking1 AI query 0 tool calls 18 total 0 passed18 broken	2026-10-10	Finished	IN20 728 OUT13 710	0.0%
Checkers game against the computer built in SC# as an agent The model builds a checkers game against the computer in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct1 AI query 8 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN284 420 OUT8 752	93.3%
Qwen 3.8 27BNVFP4Instruct1 AI query 20 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN859 267 OUT15 201	93.3%
Qwen 3.8 27BNVFP4Think: medium1 AI query 23 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN1 063 301 OUT74 951	93.3%
Qwen 3.6 27BNVFP4Thinking1 AI query 31 tool calls 15 total 12 passed3 failed	2026-10-10	Finished	IN1 539 639 OUT35 031	80.0%
Qwen 3.6 35BNVFP4Instruct4 AI queries 1 tool call 15 total 0 passed15 failed	2026-10-10	Finished	IN187 931 OUT16 276	0.0%
Qwen 3.6 35BNVFP4Thinking6 AI queries 10 tool calls 15 total 0 passed15 failed	2026-10-10	Finished	IN870 201 OUT66 634	0.0%
Contra-like game built in SC# as an agent The model builds a Contra-like side-scrolling run-and-gun game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Thinking3 AI queries 17 tool calls 16 total 14 passed2 failed	2026-10-10	Finished	IN950 809 OUT35 549	87.5%
Qwen 3.8 27BNVFP4Instruct1 AI query 19 tool calls 16 total 13 passed3 failed	2026-10-10	Finished	IN789 520 OUT11 954	81.3%
Qwen 3.6 27BNVFP4Instruct1 AI query 33 tool calls 16 total 12 passed4 failed	2026-10-10	Finished	IN1 275 118 OUT34 190	75.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 23 tool calls 16 total 12 passed4 failed	2026-10-10	Finished	IN1 038 889 OUT139 717	75.0%
Qwen 3.6 35BNVFP4Instruct4 AI queries 5 tool calls 16 total 0 passed16 failed	2026-10-10	Finished	IN434 577 OUT59 199	0.0%
Qwen 3.6 35BNVFP4Thinking2 AI queries 0 tool calls 16 total 0 passed15 failed1 broken	2026-10-10	Finished	IN45 909 OUT59 522	0.0%
Counter-Strike 2D-like game built in SC# as an agent The model builds a Counter-Strike 2D-like top-down shooter in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct2 AI queries 29 tool calls 18 total 18 passed	2026-10-10	Finished	IN2 110 700 OUT40 576	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 21 tool calls 18 total 18 passed	2026-10-10	Finished	IN1 035 652 OUT98 172	100.0%
Qwen 3.6 27BNVFP4Thinking3 AI queries 21 tool calls 18 total 15 passed3 failed	2026-10-10	Finished	IN1 217 678 OUT36 325	83.3%
Qwen 3.8 27BNVFP4Instruct1 AI query 49 tool calls 18 total 14 passed4 failed	2026-10-10	Finished	IN3 177 571 OUT33 550	77.8%
Qwen 3.6 35BNVFP4Thinking1 AI query 42 tool calls 18 total 12 passed6 failed	2026-10-10	Finished	IN2 349 371 OUT117 379	66.7%
Qwen 3.6 35BNVFP4Instruct3 AI queries 0 tool calls 18 total 0 passed18 failed	2026-10-10	Finished	IN69 988 OUT9 626	0.0%
Diamond Dash-like game built in SC# as an agent The model builds a Diamond Dash-like gem clicker (groups of 3 or more gems vanish, the rest fall) played with the mouse with a polished look in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BFP8Thinking1 AI query 20 tool calls 15 total 15 passed	2026-10-14	Finished	IN1 061 436 OUT68 220	100.0%
Qwen 3.8 27BNVFP4Think: medium2 AI queries 14 tool calls 15 total 15 passed	2026-10-14	Finished	IN931 466 OUT94 843	100.0%
Qwen 3.8 27BNVFP4Instruct2 AI queries 13 tool calls 15 total 14 passed1 failed	2026-10-14	Finished	IN589 482 OUT8 194	93.3%
Qwen 3.6 27BNVFP4Instruct5 AI queries 17 tool calls 15 total 13 passed2 failed	2026-10-14	Finished	IN1 119 129 OUT17 254	86.7%
Qwen 3.6 27BNVFP4Thinking3 AI queries 12 tool calls 15 total 12 passed3 failed	2026-10-14	Finished	IN697 527 OUT27 043	80.0%
Qwen 3.6 35BNVFP4Thinking2 AI queries 16 tool calls 15 total 11 passed4 failed	2026-10-14	Finished	IN1 002 633 OUT54 154	73.3%
Qwen 3.6 27BFP8Instruct1 AI query 19 tool calls 15 total 6 passed9 failed	2026-10-14	Finished	IN1 224 031 OUT19 925	40.0%
Qwen 3.6 35BNVFP4Instruct4 AI queries 5 tool calls 15 total 3 passed12 failed	2026-10-14	Finished	IN410 138 OUT19 806	20.0%
Game menus and screen transitions built in SC# as an agent The model builds a keyboard app with a main menu, options, high scores, a pause menu and a name entry around a small game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct1 AI query 9 tool calls 16 total 14 passed1 failed1 broken	2026-10-10	Finished	IN393 236 OUT18 673	87.5%
Qwen 3.6 27BNVFP4Thinking1 AI query 24 tool calls 16 total 14 passed2 failed	2026-10-10	Finished	IN1 404 532 OUT43 816	87.5%
Qwen 3.8 27BNVFP4Instruct2 AI queries 30 tool calls 16 total 13 passed3 failed	2026-10-10	Finished	IN1 141 824 OUT19 175	81.3%
Qwen 3.8 27BNVFP4Think: medium1 AI query 18 tool calls 16 total 13 passed3 failed	2026-10-10	Finished	IN692 420 OUT86 571	81.3%
Qwen 3.6 35BNVFP4Instruct6 AI queries 8 tool calls 16 total 4 passed12 failed	2026-10-10	Finished	IN795 661 OUT67 823	25.0%
Qwen 3.6 35BNVFP4Thinking1 AI query 5 tool calls 16 total 0 passed15 failed1 broken	2026-10-10	Finished	IN217 166 OUT150 278	0.0%
Icy Tower-like game built in SC# as an agent The model builds an Icy Tower-like tower jumper game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct1 AI query 28 tool calls 14 total 14 passed	2026-10-10	Finished	IN1 117 678 OUT15 692	100.0%
Qwen 3.6 27BNVFP4Thinking1 AI query 5 tool calls 14 total 14 passed	2026-10-10	Finished	IN141 723 OUT50 515	100.0%
Qwen 3.6 35BNVFP4Thinking1 AI query 14 tool calls 14 total 14 passed	2026-10-10	Finished	IN432 951 OUT35 612	100.0%
Qwen 3.8 27BNVFP4Instruct1 AI query 39 tool calls 14 total 12 passed2 failed	2026-10-10	Finished	IN1 672 143 OUT18 351	85.7%
Qwen 3.6 35BNVFP4Instruct1 AI query 49 tool calls 14 total 12 passed2 failed	2026-10-10	Finished	IN1 916 583 OUT12 262	85.7%
Qwen 3.8 27BNVFP4Think: medium1 AI query 4 tool calls 14 total 1 passed12 failed1 broken	2026-10-10	Finished	IN118 127 OUT25 634	7.1%
Klondike solitaire game built in SC# as an agent The model builds Klondike solitaire played with the mouse (drag and drop) with a polished look in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BFP8Thinking1 AI query 47 tool calls 16 total 16 passed	2026-10-14	Finished	IN3 520 614 OUT249 344	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 14 tool calls 16 total 16 passed	2026-10-14	Finished	IN724 941 OUT32 314	100.0%
Qwen 3.6 27BFP8Instruct1 AI query 15 tool calls 16 total 15 passed1 failed	2026-10-14	Finished	IN871 049 OUT15 214	93.8%
Qwen 3.6 27BNVFP4Instruct1 AI query 19 tool calls 16 total 14 passed2 failed	2026-10-14	Finished	IN1 214 030 OUT19 624	87.5%
Qwen 3.6 27BNVFP4Thinking4 AI queries 19 tool calls 16 total 13 passed3 failed	2026-10-14	Finished	IN1 211 192 OUT61 206	81.3%
Qwen 3.8 27BNVFP4Instruct7 AI queries 28 tool calls 16 total 10 passed6 failed	2026-10-14	Finished	IN1 787 280 OUT18 212	62.5%
Qwen 3.6 35BNVFP4Thinking1 AI query 28 tool calls 16 total 9 passed7 failed	2026-10-14	Finished	IN1 753 298 OUT36 402	56.3%
Qwen 3.6 35BNVFP4Instruct1 AI query 49 tool calls 16 total 8 passed8 failed	2026-10-14	Finished	IN3 466 651 OUT30 320	50.0%
Mahjong solitaire game built in SC# as an agent The model builds Mahjong solitaire (72 tiles in 3 layers) played with the mouse with a polished look in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BFP8Thinking1 AI query 22 tool calls 13 total 13 passed	2026-10-14	Finished	IN1 388 246 OUT60 185	100.0%
Qwen 3.6 27BFP8Instruct1 AI query 29 tool calls 13 total 12 passed1 failed	2026-10-14	Finished	IN1 588 673 OUT17 391	92.3%
Qwen 3.6 27BNVFP4Thinking1 AI query 44 tool calls 13 total 12 passed1 failed	2026-10-14	Finished	IN3 159 990 OUT73 409	92.3%
Qwen 3.8 27BNVFP4Think: medium3 AI queries 18 tool calls 13 total 12 passed1 failed	2026-10-14	Finished	IN1 009 065 OUT68 087	92.3%
Qwen 3.6 35BNVFP4Thinking1 AI query 6 tool calls 13 total 12 passed1 failed	2026-10-14	Finished	IN287 194 OUT15 251	92.3%
Qwen 3.6 27BNVFP4Instruct1 AI query 32 tool calls 13 total 11 passed2 failed	2026-10-14	Finished	IN1 997 387 OUT22 245	84.6%
Qwen 3.8 27BNVFP4Instruct7 AI queries 32 tool calls 13 total 10 passed3 failed	2026-10-14	Finished	IN2 353 015 OUT16 695	76.9%
Qwen 3.6 35BNVFP4Instruct4 AI queries 1 tool call 13 total 0 passed13 failed	2026-10-14	Finished	IN228 136 OUT17 090	0.0%
Mario-like game built in SC# as an agent The model builds a Mario-like scrolling platformer in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Thinking4 AI queries 32 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN2 029 902 OUT109 028	93.3%
Qwen 3.8 27BNVFP4Think: medium1 AI query 22 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN945 290 OUT81 621	93.3%
Qwen 3.6 27BNVFP4Instruct1 AI query 49 tool calls 15 total 13 passed2 failed	2026-10-10	Finished	IN3 641 703 OUT55 027	86.7%
Qwen 3.8 27BNVFP4Instruct1 AI query 55 tool calls 15 total 7 passed8 failed	2026-10-10	Finished	IN3 324 006 OUT33 155	46.7%
Qwen 3.6 35BNVFP4Instruct1 AI query 0 tool calls 15 total 0 passed15 broken	2026-10-10	Finished	IN20 999 OUT19 327	0.0%
Qwen 3.6 35BNVFP4Thinking1 AI query 0 tool calls 15 total 0 passed14 failed1 broken	2026-10-10	Finished	IN20 997 OUT62 144	0.0%
Micro Machines-like game built in SC# as an agent The model builds a Micro Machines-like toy car racing game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.8 27BNVFP4Think: medium1 AI query 28 tool calls 14 total 12 passed1 failed1 broken	2026-10-10	Finished	IN1 719 911 OUT268 935	85.7%
Qwen 3.6 27BNVFP4Instruct2 AI queries 29 tool calls 14 total 11 passed3 failed	2026-10-10	Finished	IN1 891 466 OUT38 080	78.6%
Qwen 3.8 27BNVFP4Instruct1 AI query 50 tool calls 14 total 11 passed3 failed	2026-10-10	Finished	IN2 763 317 OUT24 448	78.6%
Qwen 3.6 35BNVFP4Instruct2 AI queries 35 tool calls 14 total 6 passed8 failed	2026-10-10	Finished	IN3 125 210 OUT73 100	42.9%
Qwen 3.6 27BNVFP4Thinking1 AI query 4 tool calls 14 total 0 passed13 failed1 broken	2026-10-10	Finished	IN125 895 OUT40 155	0.0%
Qwen 3.6 35BNVFP4Thinking1 AI query 3 tool calls 14 total 0 passed13 failed1 broken	2026-10-10	Finished	IN115 249 OUT164 492	0.0%
Minecraft-like 3D game built in SC# as an agent The model builds a Minecraft-like 3D block world with a polished look (pixel-art textures, hotbar, effects) in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BFP8Instruct1 AI query 100 tool calls 23 total 23 passed	2026-10-14	Finished	IN9 386 205 OUT41 122	100.0%
Qwen 3.6 27BNVFP4Thinking2 AI queries 55 tool calls 23 total 22 passed1 failed	2026-10-14	Finished	IN4 799 838 OUT117 663	95.7%
Qwen 3.8 27BNVFP4Think: medium3 AI queries 23 tool calls 23 total 22 passed1 failed	2026-10-14	Finished	IN1 458 174 OUT73 762	95.7%
Qwen 3.6 27BFP8Thinking1 AI query 43 tool calls 23 total 19 passed4 failed	2026-10-14	Finished	IN3 356 865 OUT200 809	82.6%
Qwen 3.6 27BNVFP4Instruct2 AI queries 98 tool calls 23 total 19 passed4 failed	2026-10-14	Finished	IN9 169 563 OUT56 194	82.6%
Qwen 3.6 35BNVFP4Thinking5 AI queries 18 tool calls 23 total 15 passed8 failed	2026-10-14	Finished	IN1 391 819 OUT99 862	65.2%
Qwen 3.8 27BNVFP4Instruct15 AI queries 65 tool calls 23 total 2 passed21 failed	2026-10-14	Finished	IN4 543 018 OUT18 830	8.7%
Qwen 3.6 35BNVFP4Instruct4 AI queries 2 tool calls 23 total 1 passed22 failed	2026-10-14	Finished	IN230 037 OUT6 036	4.3%
Minesweeper-like game built in SC# as an agent The model builds a Minesweeper-like mine field puzzle in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct1 AI query 49 tool calls 16 total 16 passed	2026-10-10	Finished	IN2 646 805 OUT28 364	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 16 tool calls 16 total 16 passed	2026-10-10	Finished	IN601 293 OUT89 817	100.0%
Qwen 3.6 27BNVFP4Thinking1 AI query 11 tool calls 16 total 15 passed1 failed	2026-10-10	Finished	IN412 335 OUT33 182	93.8%
Qwen 3.8 27BNVFP4Instruct1 AI query 27 tool calls 16 total 15 passed1 failed	2026-10-10	Finished	IN1 004 606 OUT10 453	93.8%
Qwen 3.6 35BNVFP4Instruct2 AI queries 48 tool calls 16 total 0 passed16 failed	2026-10-10	Finished	IN2 260 292 OUT18 482	0.0%
Qwen 3.6 35BNVFP4Thinking7 AI queries 7 tool calls 16 total 0 passed16 failed	2026-10-10	Finished	IN622 263 OUT86 293	0.0%
Pac-Man-like game built in SC# as an agent The model builds a Pac-Man-like maze chase game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.8 27BNVFP4Think: medium4 AI queries 14 tool calls 18 total 18 passed	2026-10-10	Finished	IN662 999 OUT82 755	100.0%
Qwen 3.8 27BNVFP4Instruct1 AI query 49 tool calls 18 total 13 passed5 failed	2026-10-10	Finished	IN2 619 892 OUT20 261	72.2%
Qwen 3.6 27BNVFP4Instruct1 AI query 49 tool calls 18 total 1 passed17 failed	2026-10-10	Finished	IN2 493 701 OUT17 448	5.6%
Qwen 3.6 27BNVFP4Thinking1 AI query 0 tool calls 18 total 0 passed18 failed	2026-10-10	Finished	IN0 OUT0	0.0%
Qwen 3.6 35BNVFP4Instruct4 AI queries 2 tool calls 18 total 0 passed18 failed	2026-10-10	Finished	IN268 765 OUT46 390	0.0%
Qwen 3.6 35BNVFP4Thinking4 AI queries 9 tool calls 18 total 0 passed18 failed	2026-10-10	Finished	IN862 950 OUT82 456	0.0%
Scorched Earth-like game built in SC# as an agent The model builds a Scorched Earth-like turn-based artillery duel in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.8 27BNVFP4Instruct1 AI query 15 tool calls 12 total 12 passed	2026-10-10	Finished	IN518 373 OUT11 066	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 15 tool calls 12 total 12 passed	2026-10-10	Finished	IN521 629 OUT93 248	100.0%
Qwen 3.6 35BNVFP4Thinking3 AI queries 47 tool calls 12 total 12 passed	2026-10-10	Finished	IN3 983 394 OUT136 919	100.0%
Qwen 3.6 27BNVFP4Instruct3 AI queries 27 tool calls 12 total 11 passed1 failed	2026-10-10	Finished	IN1 342 986 OUT22 525	91.7%
Qwen 3.6 27BNVFP4Thinking6 AI queries 18 tool calls 12 total 11 passed1 failed	2026-10-10	Finished	IN1 142 384 OUT80 764	91.7%
Qwen 3.6 35BNVFP4Instruct9 AI queries 41 tool calls 12 total 0 passed12 failed	2026-10-10	Finished	IN3 429 922 OUT79 413	0.0%
Snake game built in SC# as an agent The model builds the classic Snake game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.8 27BNVFP4Instruct1 AI query 20 tool calls 15 total 15 passed	2026-10-10	Finished	IN542 692 OUT7 166	100.0%
Qwen 3.6 35BNVFP4Thinking1 AI query 8 tool calls 15 total 15 passed	2026-10-10	Finished	IN232 727 OUT51 837	100.0%
Qwen 3.6 27BNVFP4Instruct1 AI query 18 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN606 766 OUT12 611	93.3%
Qwen 3.6 27BNVFP4Thinking1 AI query 11 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN332 398 OUT31 755	93.3%
Qwen 3.6 35BNVFP4Instruct4 AI queries 7 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN318 018 OUT14 750	93.3%
Qwen 3.8 27BNVFP4Think: medium1 AI query 14 tool calls 15 total 12 passed3 failed	2026-10-10	Finished	IN479 477 OUT81 376	80.0%
Sokoban-like game built in SC# as an agent The model builds a Sokoban-like box pushing puzzle in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.8 27BNVFP4Think: medium1 AI query 8 tool calls 15 total 15 passed	2026-10-10	Finished	IN237 575 OUT25 970	100.0%
Qwen 3.6 27BNVFP4Instruct1 AI query 49 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN2 698 956 OUT39 519	93.3%
Qwen 3.6 27BNVFP4Thinking1 AI query 19 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN637 036 OUT27 472	93.3%
Qwen 3.8 27BNVFP4Instruct1 AI query 31 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN874 201 OUT12 158	93.3%
Qwen 3.6 35BNVFP4Thinking8 AI queries 27 tool calls 15 total 14 passed1 failed	2026-10-10	Finished	IN1 623 967 OUT144 458	93.3%
Qwen 3.6 35BNVFP4Instruct3 AI queries 0 tool calls 15 total 0 passed15 failed	2026-10-10	Finished	IN66 815 OUT7 638	0.0%
Space Invaders-like game built in SC# as an agent The model builds a Space Invaders-like fixed shooter game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct1 AI query 9 tool calls 17 total 17 passed	2026-10-10	Finished	IN294 060 OUT8 549	100.0%
Qwen 3.6 27BNVFP4Thinking2 AI queries 11 tool calls 17 total 17 passed	2026-10-10	Finished	IN409 143 OUT37 032	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 14 tool calls 17 total 17 passed	2026-10-10	Finished	IN422 685 OUT36 643	100.0%
Qwen 3.8 27BNVFP4Instruct1 AI query 41 tool calls 17 total 16 passed1 failed	2026-10-10	Finished	IN1 395 508 OUT13 805	94.1%
Qwen 3.6 35BNVFP4Instruct6 AI queries 44 tool calls 17 total 0 passed17 failed	2026-10-10	Finished	IN3 189 816 OUT40 182	0.0%
Qwen 3.6 35BNVFP4Thinking4 AI queries 3 tool calls 17 total 0 passed17 failed	2026-10-10	Finished	IN241 333 OUT40 644	0.0%
Supaplex-like game built in SC# as an agent The model builds a Supaplex-like digger puzzle game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.8 27BNVFP4Think: medium1 AI query 14 tool calls 15 total 15 passed	2026-10-10	Finished	IN465 518 OUT56 201	100.0%
Qwen 3.6 27BNVFP4Thinking2 AI queries 24 tool calls 15 total 12 passed3 failed	2026-10-10	Finished	IN999 048 OUT61 826	80.0%
Qwen 3.8 27BNVFP4Instruct1 AI query 21 tool calls 15 total 12 passed3 failed	2026-10-10	Finished	IN643 384 OUT16 236	80.0%
Qwen 3.6 27BNVFP4Instruct2 AI queries 29 tool calls 15 total 7 passed8 failed	2026-10-10	Finished	IN1 405 767 OUT21 826	46.7%
Qwen 3.6 35BNVFP4Instruct12 AI queries 7 tool calls 15 total 0 passed15 failed	2026-10-10	Finished	IN1 111 322 OUT76 587	0.0%
Qwen 3.6 35BNVFP4Thinking1 AI query 3 tool calls 15 total 0 passed14 failed1 broken	2026-10-10	Finished	IN124 130 OUT71 359	0.0%
Super Speed-like game built in SC# as an agent The model builds a Super Speed-like top-down racing game with laps, wrenches and an upgrade shop in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.8 27BNVFP4Think: medium2 AI queries 17 tool calls 18 total 18 passed	2026-10-10	Finished	IN947 765 OUT144 742	100.0%
Qwen 3.6 27BNVFP4Thinking1 AI query 45 tool calls 18 total 17 passed1 failed	2026-10-10	Finished	IN2 255 995 OUT106 668	94.4%
Qwen 3.8 27BNVFP4Instruct1 AI query 50 tool calls 18 total 17 passed1 failed	2026-10-10	Finished	IN2 187 271 OUT15 979	94.4%
Qwen 3.6 27BNVFP4Instruct1 AI query 49 tool calls 18 total 14 passed4 failed	2026-10-10	Finished	IN2 600 815 OUT28 842	77.8%
Qwen 3.6 35BNVFP4Thinking3 AI queries 39 tool calls 18 total 3 passed15 failed	2026-10-10	Finished	IN2 450 613 OUT80 506	16.7%
Qwen 3.6 35BNVFP4Instruct4 AI queries 5 tool calls 18 total 0 passed18 failed	2026-10-10	Finished	IN305 750 OUT19 487	0.0%
Tetris-like game built in SC# as an agent The model builds a Tetris-like falling blocks game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Qwen 3.6 27BNVFP4Instruct4 AI queries 15 tool calls 18 total 18 passed	2026-10-10	Finished	IN738 357 OUT17 221	100.0%
Qwen 3.6 27BNVFP4Thinking1 AI query 8 tool calls 18 total 18 passed	2026-10-10	Finished	IN264 932 OUT52 746	100.0%
Qwen 3.8 27BNVFP4Think: medium1 AI query 14 tool calls 18 total 17 passed1 failed	2026-10-10	Finished	IN511 072 OUT47 287	94.4%
Qwen 3.8 27BNVFP4Instruct1 AI query 38 tool calls 18 total 15 passed3 failed	2026-10-10	Finished	IN2 814 499 OUT159 440	83.3%
Qwen 3.6 35BNVFP4Instruct1 AI query 0 tool calls 18 total 0 passed18 broken	2026-10-10	Finished	IN20 428 OUT19 888	0.0%
Qwen 3.6 35BNVFP4Thinking1 AI query 49 tool calls 18 total 0 passed18 failed	2026-10-10	Finished	IN2 076 394 OUT114 853	0.0%