Ai4Users - Qwen 3.6 35B FP8 Instruct

Benchmark items
Agentic - Blue-Green Deploy (health-gated cutover) 1 passed	100.0%	IN14 575 OUT179
✓	AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query 4 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN14 575 OUT179
Agentic - Bugfix (root-cause vs regression) 1 passed	100.0%	IN30 876 OUT510
✓	AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query 2 tool callscategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users	Passed	IN30 876 OUT510
Agentic - Cache Invalidation (exact set) 1 passed	100.0%	IN17 603 OUT665
✓	AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query 7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN17 603 OUT665
Agentic - Canary (metric-gated release) 1 passed	100.0%	IN46 354 OUT1 381
✓	AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN46 354 OUT1 381
Agentic - Canary variants (5 metric cases) 5 passed	100.0%	IN196 482 OUT5 394
✓	CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN31 055 OUT801
✓	CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query 11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN43 770 OUT1 276
✓	CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN31 067 OUT826
✓	CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query 14 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN56 584 OUT1 612
✓	CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN34 006 OUT879
Agentic - Code Review (find injected defect) 1 passed	100.0%	IN13 477 OUT869
✓	AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN13 477 OUT869
Agentic - Config Drift (remediate exact) 1 passed	100.0%	IN20 945 OUT523
✓	AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query 6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN20 945 OUT523
Agentic - Constraints (multi-section satisfaction) 1 passed	100.0%	IN22 097 OUT806
✓	AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN22 097 OUT806
Agentic - Constraints variants (5 cases) 5 passed	100.0%	IN50 869 OUT2 394
✓	ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN10 668 OUT684
✓	ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN9 833 OUT341
✓	ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN9 870 OUT341
✓	ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN10 428 OUT591
✓	ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN10 070 OUT437
Agentic - Data Reconciliation (per-field precedence) 1 passed	100.0%	IN14 188 OUT849
✓	AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN14 188 OUT849
Agentic - Diagnosis (root cause under budget) 1 failed	0.0%	IN7 256 OUT194
×	AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query 2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users Assertion expression evaluated to false: Globals.Get<bool>("diag.has_report", false) Context: Agent did not report a root cause.	Failed	IN7 256 OUT194
Agentic - Diagnosis variants (5 cases) 5 passed	100.0%	IN100 341 OUT4 356
✓	DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN14 841 OUT668
✓	DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN15 258 OUT721
✓	DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN14 902 OUT538
✓	DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN33 162 OUT816
✓	DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query 16 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN22 178 OUT1 613
Agentic - ETL Repair (root-cause in stage 2) 1 passed	100.0%	IN45 548 OUT950
✓	AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN45 548 OUT950
Agentic - Feature Flag (staged rollout + kill-switch) 1 passed	100.0%	IN24 529 OUT925
✓	AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN24 529 OUT925
Agentic - Game Backend: 1D bouncing ball 1 passed	100.0%	IN8 783 OUT224
✓	GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 783 OUT224
Agentic - Game Backend: 1D swept anti-tunneling 1 passed	100.0%	IN19 654 OUT2 260
✓	GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 654 OUT2 260
Agentic - Game Backend: 2048 left-move (slide + merge) 1 passed	100.0%	IN8 803 OUT2 038
✓	GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 803 OUT2 038
Agentic - Game Backend: 8-direction from vector 1 passed	100.0%	IN8 810 OUT260
✓	GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 810 OUT260
Agentic - Game Backend: AABB collision resolution (MTV) 1 passed	100.0%	IN29 252 OUT1 597
✓	GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN29 252 OUT1 597
Agentic - Game Backend: AABB containment 1 passed	100.0%	IN17 557 OUT207
✓	GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN17 557 OUT207
Agentic - Game Backend: Acceleration to max speed 1 passed	100.0%	IN8 741 OUT615
✓	GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 741 OUT615
Agentic - Game Backend: Achievement thresholds 1 passed	100.0%	IN26 404 OUT7 872
✓	GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN26 404 OUT7 872
Agentic - Game Backend: Active status effects 1 failed	0.0%	IN38 557 OUT1 415
×	GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users ActiveAt did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 11:18: Type 'List<string>' has no fields. Only struct values support '.Length'.	Failed	IN38 557 OUT1 415
Agentic - Game Backend: Animation frame at time 1 passed	100.0%	IN8 773 OUT909
✓	GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 773 OUT909
Agentic - Game Backend: Army upkeep income 1 passed	100.0%	IN18 282 OUT901
✓	GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 282 OUT901
Agentic - Game Backend: Axis-aligned path follow 1 passed	100.0%	IN42 775 OUT3 205
✓	GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN42 775 OUT3 205
Agentic - Game Backend: Battleship shot resolution 1 failed	0.0%	IN43 098 OUT3 018
×	GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Fire did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 15:15: Expected a variable name, found '['.	Failed	IN43 098 OUT3 018
Agentic - Game Backend: Bounce restitution 1 passed	100.0%	IN8 760 OUT1 528
✓	GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 760 OUT1 528
Agentic - Game Backend: Bresenham line rasterization 1 passed	100.0%	IN19 826 OUT2 338
✓	GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 826 OUT2 338
Agentic - Game Backend: Buff/debuff stacking 1 passed	100.0%	IN18 354 OUT976
✓	GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 354 OUT976
Agentic - Game Backend: Build queue completions 1 passed	100.0%	IN8 754 OUT2 237
✓	GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 754 OUT2 237
Agentic - Game Backend: Camera-relative movement 1 passed	100.0%	IN8 841 OUT1 257
✓	GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 841 OUT1 257
Agentic - Game Backend: Card hand sort 1 passed	100.0%	IN20 222 OUT3 144
✓	GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN20 222 OUT3 144
Agentic - Game Backend: Card-game rules (blackjack/deck/compare) 1 passed	100.0%	IN18 709 OUT1 463
✓	GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 709 OUT1 463
Agentic - Game Backend: Chart scaling & histogram bins 1 passed	100.0%	IN8 826 OUT898
✓	GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 826 OUT898
Agentic - Game Backend: Checkers captures 1 failed	0.0%	IN8 796 OUT3 414
×	GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Captures threw: String index 24 is out of range. The string has 23 character(s).	Failed	IN8 796 OUT3 414
Agentic - Game Backend: Collision primitives (circle/aabb/segment) 1 failed	0.0%	IN8 999 OUT3 355
×	GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users AabbOverlap did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: Entry point 'AabbOverlap' was not found. Available functions: AskAi, RunScript, SkipTest, __CreateInMemoryDatabase, __DbExecute, __DbExecuteScript, __DbInsert, __DbQueryFirstJson, __DbQueryJson, __DbScalarJson, CircleCircle.	Failed	IN8 999 OUT3 355
Agentic - Game Backend: Combat turn order (initiative) 1 passed	100.0%	IN8 735 OUT2 690
✓	GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 735 OUT2 690
Agentic - Game Backend: Combo scoring (rising multiplier) 1 passed	100.0%	IN8 758 OUT823
✓	GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 758 OUT823
Agentic - Game Backend: Control-point capture 1 passed	100.0%	IN8 771 OUT1 259
✓	GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 771 OUT1 259
Agentic - Game Backend: Controls & Inventory (spec-driven function battery) 1 passed	100.0%	IN18 192 OUT604
✓	GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 192 OUT604
Agentic - Game Backend: Cooldowns & resource regen 1 passed	100.0%	IN8 829 OUT1 436
✓	GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 829 OUT1 436
Agentic - Game Backend: Coyote-time jump 1 passed	100.0%	IN17 584 OUT92
✓	GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN17 584 OUT92
Agentic - Game Backend: Crafting capacity 1 passed	100.0%	IN19 675 OUT2 444
✓	GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 675 OUT2 444
Agentic - Game Backend: Cribbage fifteens 1 passed	100.0%	IN8 751 OUT1 410
✓	GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 751 OUT1 410
Agentic - Game Backend: Critical-hit damage 1 passed	100.0%	IN8 741 OUT335
✓	GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 741 OUT335
Agentic - Game Backend: Damage over time 1 passed	100.0%	IN8 767 OUT1 279
✓	GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 767 OUT1 279
Agentic - Game Backend: Day/night cycle phase 1 passed	100.0%	IN8 753 OUT739
✓	GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 753 OUT739
Agentic - Game Backend: Dice notation max roll 1 passed	100.0%	IN19 962 OUT1 930
✓	GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 962 OUT1 930
Agentic - Game Backend: Enemy AI state machine 1 passed	100.0%	IN8 854 OUT1 206
✓	GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN8 854 OUT1 206
Agentic - Game Backend: Equipment slot validation 1 passed	100.0%	IN17 637 OUT217
✓	GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN17 637 OUT217
Agentic - Game Backend: Flood-fill region size 1 failed	0.0%	IN43 028 OUT3 372
×	GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users RegionSize did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 27:12: Expected a variable name, found '['.	Failed	IN43 028 OUT3 372
Agentic - Game Backend: Fog of war visibility 1 passed	100.0%	IN19 040 OUT1 786
✓	GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN19 040 OUT1 786
Agentic - Game Backend: Fog reveal (Chebyshev radius) 1 failed	0.0%	IN41 127 OUT2 482
×	GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Revealed did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 8:11: Expected a variable name, found '['.	Failed	IN41 127 OUT2 482
Agentic - Game Backend: Friction stop frame 1 passed	100.0%	IN8 749 OUT1 101
✓	GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 749 OUT1 101
Agentic - Game Backend: Game of Life (code-gen vs oracle) 1 failed	0.0%	IN8 808 OUT1 295
×	GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users candidate did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 1:53: Char literals must contain exactly one character and close with '''. 4:24: Unexpected character '`'. 4:61: Unexpected character '`'. 21:8: Char literals must contain exactly one character and close with '''. 26:69: Unexpected character '×'. 37:20: Unexpected character '#'. 39:1: Unexpected character '`'. 39:2: Unexpected character '`'. 39:3: Unexpected character '`'. 105:1: Unexpected character '`'. 105:2: Unexpected character '`'. 105:3: Unexpected character '`'. 107:47: Char literals must contain exactly one character and close with '''. 109:58: Char literals must contain exactly one character and close with '''. 111:1: Unexpected character '`'. 111:2: Unexpected character '`'. 111:3: Unexpected character '`'. 172:1: Unexpected character '`'. 172:2: Unexpected character '`'. 172:3: Unexpected character '`'. 176:84: Char literals must contain exactly one character and close with '''.	Failed	IN8 808 OUT1 295
Agentic - Game Backend: Greedy coin change 1 passed	100.0%	IN18 434 OUT1 061
✓	GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 434 OUT1 061
Agentic - Game Backend: Grid inventory & stacking (Diablo-style) 1 passed	100.0%	IN30 444 OUT2 319
✓	GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.3 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN30 444 OUT2 319
Agentic - Game Backend: Grid pathfinding (shortest path, BFS) 1 passed	100.0%	IN46 467 OUT5 385
✓	GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.4 AI queries 0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN46 467 OUT5 385
Agentic - Game Backend: Grid raycast 1 passed	100.0%	IN19 032 OUT1 612
✓	GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 032 OUT1 612
Agentic - Game Backend: Health bar segments 1 passed	100.0%	IN18 372 OUT904
✓	GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 372 OUT904
Agentic - Game Backend: High-score table insert 1 failed	0.0%	IN43 765 OUT3 349
×	GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Insert did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 7:15: Expected a variable name, found '['.	Failed	IN43 765 OUT3 349
Agentic - Game Backend: Hit chance 1 passed	100.0%	IN17 902 OUT437
✓	GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN17 902 OUT437
Agentic - Game Backend: Inventory encumbrance 1 passed	100.0%	IN29 102 OUT1 517
✓	GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN29 102 OUT1 517
Agentic - Game Backend: Inventory first-fit placement 1 passed	100.0%	IN27 662 OUT872
✓	GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN27 662 OUT872
Agentic - Game Backend: Inventory stack split 1 passed	100.0%	IN17 706 OUT181
✓	GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN17 706 OUT181
Agentic - Game Backend: Isometric projection 1 passed	100.0%	IN8 775 OUT909
✓	GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 775 OUT909
Agentic - Game Backend: Jump buffer 1 passed	100.0%	IN8 763 OUT1 399
✓	GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 763 OUT1 399
Agentic - Game Backend: Knockback with arena clamp 1 passed	100.0%	IN18 645 OUT1 187
✓	GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 645 OUT1 187
Agentic - Game Backend: Leaderboard rank 1 passed	100.0%	IN18 573 OUT1 148
✓	GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 573 OUT1 148
Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over) 1 passed	100.0%	IN18 272 OUT707
✓	GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 272 OUT707
Agentic - Game Backend: Lifesteal healing 1 passed	100.0%	IN8 746 OUT1 081
✓	GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 746 OUT1 081
Agentic - Game Backend: Lights Out toggle 1 failed	0.0%	IN40 224 OUT2 356
×	GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users Toggle did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 5:8: Expected a variable name, found '['.	Failed	IN40 224 OUT2 356
Agentic - Game Backend: Match-3 cascade 1 passed	100.0%	IN8 820 OUT1 736
✓	GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN8 820 OUT1 736
Agentic - Game Backend: Match-3 detection 1 passed	100.0%	IN8 784 OUT724
✓	GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 784 OUT724
Agentic - Game Backend: Match-3 swap validity 1 passed	100.0%	IN29 785 OUT2 283
✓	GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN29 785 OUT2 283
Agentic - Game Backend: Memory match pairs 1 passed	100.0%	IN8 747 OUT1 972
✓	GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 747 OUT1 972
Agentic - Game Backend: Mercy invincibility (i-frames) 1 passed	100.0%	IN19 586 OUT2 105
✓	GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 586 OUT2 105
Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal) 1 passed	100.0%	IN21 533 OUT4 599
✓	GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN21 533 OUT4 599
Agentic - Game Backend: Minimap downsample 1 passed	100.0%	IN8 778 OUT457
✓	GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 778 OUT457
Agentic - Game Backend: Minimap marker position 1 passed	100.0%	IN18 893 OUT1 398
✓	GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 893 OUT1 398
Agentic - Game Backend: N-in-a-row win detection (Connect-4) 1 failed	0.0%	IN45 202 OUT3 912
×	GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.4 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Winner did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 3:8: Expected a variable name, found '['.	Failed	IN45 202 OUT3 912
Agentic - Game Backend: Nonogram row validation 1 passed	100.0%	IN19 407 OUT2 121
✓	GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 407 OUT2 121
Agentic - Game Backend: OBB collision (rotating rectangles, SAT) 1 passed	100.0%	IN8 905 OUT7 873
✓	GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN8 905 OUT7 873
Agentic - Game Backend: Particle lifetime & expiry 1 passed	100.0%	IN8 840 OUT1 296
✓	GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 840 OUT1 296
Agentic - Game Backend: Party XP share 1 passed	100.0%	IN8 758 OUT1 626
✓	GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 758 OUT1 626
Agentic - Game Backend: Patrol waypoints (ping-pong) 1 passed	100.0%	IN8 781 OUT1 316
✓	GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 781 OUT1 316
Agentic - Game Backend: Platformer jump arc 1 passed	100.0%	IN8 772 OUT109
✓	GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 772 OUT109
Agentic - Game Backend: Point on circle (trig) 1 passed	100.0%	IN28 208 OUT942
✓	GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN28 208 OUT942
Agentic - Game Backend: Poker hand comparison 1 failed	0.0%	IN69 550 OUT15 303
×	GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users Compare did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 29:28: Expected '{' after the struct name. Structs are created with 'new Name { Field = value }', found '('.	Failed	IN69 550 OUT15 303
Agentic - Game Backend: Poker hand ranking 1 passed	100.0%	IN19 436 OUT2 218
✓	GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN19 436 OUT2 218
Agentic - Game Backend: Pop-cap training 1 passed	100.0%	IN8 759 OUT1 162
✓	GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 759 OUT1 162
Agentic - Game Backend: Progress bar fill 1 passed	100.0%	IN8 741 OUT709
✓	GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 741 OUT709
Agentic - Game Backend: Projectile range (physics/trig) 1 passed	100.0%	IN8 763 OUT588
✓	GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 763 OUT588
Agentic - Game Backend: Projectile with ground bounce 1 passed	100.0%	IN28 299 OUT753
✓	GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN28 299 OUT753
Agentic - Game Backend: Resource gathering 1 passed	100.0%	IN8 758 OUT463
✓	GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 758 OUT463
Agentic - Game Backend: Reversi flip count 1 failed	0.0%	IN38 702 OUT1 702
×	GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users FlipCount did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 11:8: Expected a variable name, found '['.	Failed	IN38 702 OUT1 702
Agentic - Game Backend: Rotate 2D vector (trig) 1 passed	100.0%	IN17 966 OUT465
✓	GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN17 966 OUT465
Agentic - Game Backend: Round-robin deal 1 passed	100.0%	IN18 512 OUT1 039
✓	GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 512 OUT1 039
Agentic - Game Backend: RTS resource economy 1 passed	100.0%	IN8 754 OUT201
✓	GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 754 OUT201
Agentic - Game Backend: RTS unit counters & damage 1 passed	100.0%	IN8 806 OUT1 019
✓	GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 806 OUT1 019
Agentic - Game Backend: Safest neighbour (danger map) 1 passed	100.0%	IN19 592 OUT2 413
✓	GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 592 OUT2 413
Agentic - Game Backend: Scroll viewport 1 passed	100.0%	IN8 776 OUT1 657
✓	GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 776 OUT1 657
Agentic - Game Backend: Set card validity 1 passed	100.0%	IN8 755 OUT673
✓	GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 755 OUT673
Agentic - Game Backend: Shop affordability 1 passed	100.0%	IN8 735 OUT787
✓	GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 735 OUT787
Agentic - Game Backend: Sliding puzzle move 1 passed	100.0%	IN19 864 OUT2 529
✓	GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 864 OUT2 529
Agentic - Game Backend: Snake head movement 1 passed	100.0%	IN19 234 OUT1 822
✓	GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 234 OUT1 822
Agentic - Game Backend: Snake tick 1 failed	0.0%	IN19 448 OUT1 901
×	GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: Tick [1,0;0,0] R food(9,9) got "2,0" expected "2,0;1,0" Values: r.ResultJson = "\"2,0\"" exp = "\"2,0;1,0\""	Failed	IN19 448 OUT1 901
Agentic - Game Backend: Snap to grid 1 passed	100.0%	IN18 930 OUT1 459
✓	GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 930 OUT1 459
Agentic - Game Backend: Sokoban move/push legality 1 passed	100.0%	IN8 854 OUT662
✓	GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 854 OUT662
Agentic - Game Backend: Solitaire tableau move 1 passed	100.0%	IN8 793 OUT1 962
✓	GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 793 OUT1 962
Agentic - Game Backend: Spawn scheduler count 1 passed	100.0%	IN8 750 OUT779
✓	GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 750 OUT779
Agentic - Game Backend: Spawn-wave alive count 1 passed	100.0%	IN8 762 OUT1 125
✓	GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 762 OUT1 125
Agentic - Game Backend: Stat level scaling 1 passed	100.0%	IN19 412 OUT207
✓	GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query 1 tool callcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 412 OUT207
Agentic - Game Backend: Steering seek (greedy pursuit) 1 failed	0.0%	IN8 780 OUT1 208
×	GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: Seek(0,0->5,2,3) got "3,3" expected "3,2" Values: r.ResultJson = "\"3,3\"" exp = "\"3,2\""	Failed	IN8 780 OUT1 208
Agentic - Game Backend: Tactical movement range 1 passed	100.0%	IN20 901 OUT3 743
✓	GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN20 901 OUT3 743
Agentic - Game Backend: Tech tree eligibility 1 failed	0.0%	IN44 788 OUT3 912
×	GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users CanResearch did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 4:11: Expected a variable name, found '['.	Failed	IN44 788 OUT3 912
Agentic - Game Backend: Tetris line clearing 1 passed	100.0%	IN20 168 OUT2 822
✓	GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN20 168 OUT2 822
Agentic - Game Backend: Threat table top target 1 passed	100.0%	IN8 759 OUT2 364
✓	GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 759 OUT2 364
Agentic - Game Backend: Tic-tac-toe result 1 failed	0.0%	IN17 568 OUT172
×	GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: Winner([XOXOXOOXO]) got "none" expected "draw" Values: r.ResultJson = "\"none\"" exp = "\"draw\""	Failed	IN17 568 OUT172
Agentic - Game Backend: Toroidal wrap (Pac-Man) 1 passed	100.0%	IN28 003 OUT471
✓	GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN28 003 OUT471
Agentic - Game Backend: Tower targeting (nearest in range) 1 passed	100.0%	IN8 774 OUT1 495
✓	GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 774 OUT1 495
Agentic - Game Backend: Transaction ledger 1 passed	100.0%	IN19 396 OUT1 978
✓	GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN19 396 OUT1 978
Agentic - Game Backend: Trick-taking winner 1 passed	100.0%	IN18 756 OUT1 551
✓	GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN18 756 OUT1 551
Agentic - Game Backend: Turn-based duel resolution 1 passed	100.0%	IN8 837 OUT1 753
✓	GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 837 OUT1 753
Agentic - Game Backend: Typed armor mitigation 1 passed	100.0%	IN18 343 OUT1 020
✓	GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 343 OUT1 020
Agentic - Game Backend: UI menu (wrap/clamp/paginate) 1 failed	0.0%	IN8 853 OUT1 817
×	GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users ClampSlider did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: Entry point 'ClampSlider' was not found. Available functions: AskAi, RunScript, SkipTest, __CreateInMemoryDatabase, __DbExecute, __DbExecuteScript, __DbInsert, __DbQueryFirstJson, __DbQueryJson, __DbScalarJson, MenuMove.	Failed	IN8 853 OUT1 817
Agentic - Game Backend: Uno playable cards 1 passed	100.0%	IN19 267 OUT1 834
✓	GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN19 267 OUT1 834
Agentic - Game Backend: Velocity reflection 1 passed	100.0%	IN18 441 OUT1 072
✓	GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 441 OUT1 072
Agentic - Game Backend: Wall-slide fall 1 passed	100.0%	IN8 741 OUT628
✓	GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 741 OUT628
Agentic - Game Backend: War card compare 1 passed	100.0%	IN8 761 OUT1 373
✓	GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 761 OUT1 373
Agentic - Game Backend: Weighted loot table 1 passed	100.0%	IN8 764 OUT957
✓	GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 764 OUT957
Agentic - Game Backend: Weighted terrain pathfinding 1 failed	0.0%	IN45 939 OUT5 104
×	GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users MinCost did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 38:8: Expected a variable name, found '['.	Failed	IN45 939 OUT5 104
Agentic - Game Backend: Word-wrap line count 1 passed	100.0%	IN8 746 OUT332
✓	GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 746 OUT332
Agentic - Game Backend: XP to character level 1 passed	100.0%	IN8 754 OUT880
✓	GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 754 OUT880
Agentic - Idempotency (poison-retry / no double charge) 1 passed	100.0%	IN19 573 OUT375
✓	AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN19 573 OUT375
Agentic - Irreversible Commit (measure twice) 1 failed	0.0%	IN5 416 OUT158
×	AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query 1 tool callcategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: Globals.Get<bool>("irr.finalized", false) Context: Billing was never finalized.	Failed	IN5 416 OUT158
Agentic - Least Privilege (exact grants) 1 passed	100.0%	IN14 872 OUT424
✓	AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query 6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN14 872 OUT424
Agentic - Ledger (simulated DB read-modify-write) 1 passed	100.0%	IN13 695 OUT1 045
✓	AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN13 695 OUT1 045
Agentic - Ledger variants (5 RMW cases) 5 passed	100.0%	IN120 309 OUT5 904
✓	LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query 11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN28 327 OUT1 345
✓	LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN13 611 OUT928
✓	LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query 15 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN40 892 OUT2 217
✓	LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN15 760 OUT856
✓	LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN21 719 OUT558
Agentic - Lock Ordering (deadlock avoidance) 1 passed	100.0%	IN123 128 OUT2 144
✓	AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query 25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN123 128 OUT2 144
Agentic - Lock Ordering variants (5 cases) 4 passed1 failed	80.0%	IN344 374 OUT7 321
✓	LockVariantFiveResFourOps1 AI query 23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN70 607 OUT1 797
✓	LockVariantFourResThreeOps1 AI query 18 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN77 235 OUT1 450
×	LockVariantHeavyReuse1 AI query 1 tool callcategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: Globals.Get<bool>("op.done." + op, false) Context: Operation not performed: op_1	Failed	IN5 551 OUT148
✓	LockVariantReverseListed1 AI query 24 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN117 208 OUT2 170
✓	LockVariantThreeLockOp1 AI query 21 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN73 773 OUT1 756
Agentic - Migration (cascading rollback + reapply) 1 passed	100.0%	IN42 988 OUT1 024
✓	AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query 11 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN42 988 OUT1 024
Agentic - Requirements (detect contradiction) 1 passed	100.0%	IN7 620 OUT532
✓	AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query 2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN7 620 OUT532
Agentic - Retry/Circuit-breaker (transient vs hard-down) 1 passed	100.0%	IN35 512 OUT843
✓	AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query 10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN35 512 OUT843
Agentic - Saga (compensation / distributed rollback) 1 passed	100.0%	IN21 132 OUT375
✓	AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN21 132 OUT375
Agentic - Saga variants (5 compensation cases) 5 passed	100.0%	IN94 084 OUT1 895
✓	SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN13 407 OUT206
✓	SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN13 639 OUT292
✓	SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN19 736 OUT283
✓	SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN20 183 OUT491
✓	SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query 8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN27 119 OUT623
Agentic - Schema Compat (expand-contract) 1 passed	100.0%	IN25 135 OUT417
✓	AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query 7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN25 135 OUT417
Agentic - Scratchpad (self-repair, Dyck, tool-decision) 7 passed	100.0%	IN236 697 OUT9 435
✓	AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query 3 tool callscategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN46 369 OUT2 432
✓	AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query 4 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN57 487 OUT1 955
✓	ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN21 004 OUT120
✓	ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN21 421 OUT597
✓	ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN21 184 OUT382
✓	ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query 2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users	Passed	IN36 045 OUT2 848
✓	ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query 2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN33 187 OUT1 101
Agentic - Secret Rotation (zero-downtime ordering) 1 passed	100.0%	IN17 772 OUT721
✓	AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query 5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN17 772 OUT721
Agentic - StatefulOps (incident recovery, concurrency replan) 2 passed	100.0%	IN56 197 OUT1 851
✓	AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query 9 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN23 803 OUT906
✓	AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query 8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN32 394 OUT945
Agentic - Test Authoring (write a test that catches the bug) 1 passed	100.0%	IN11 752 OUT695
✓	AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query 6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN11 752 OUT695
Agentic - Topological Planning (deps + prereq) 1 passed	100.0%	IN30 478 OUT942
✓	AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN30 478 OUT942
Algorithms - 0/1 Knapsack (dynamic programming) 1 failed	0.0%	IN41 252 OUT2 608
×	KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Knapsack did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 10:15: Expected a variable name, found '['.	Failed	IN41 252 OUT2 608
Algorithms - 1D elastic collision 1 passed	100.0%	IN8 808 OUT137
✓	ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 808 OUT137
Algorithms - All-pairs shortest path pair 1 failed	0.0%	IN55 106 OUT7 982
×	FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Dist did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 23:22: Type 'List<string>' has no fields. Only struct values support '.Length'.	Failed	IN55 106 OUT7 982
Algorithms - Balanced brackets (stack) 1 passed	100.0%	IN18 697 OUT1 374
✓	ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN18 697 OUT1 374
Algorithms - Bellman-Ford shortest path (neg weights/cycles) 1 failed	0.0%	IN55 728 OUT7 793
×	BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users ShortestPath did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 11:30: Operator '!=' cannot compare 'string' and 'null'. Only numbers, bools, chars, strings, date/time values and recursively comparable containers/structs can be compared. 41:52: Operator '<' requires numeric, char, string or matching date/time operands, but got 'long' and 'long?'. 60:48: Operator '<' requires numeric, char, string or matching date/time operands, but got 'long' and 'long?'.	Failed	IN55 728 OUT7 793
Algorithms - Binary Gray code 1 passed	100.0%	IN18 282 OUT685
✓	GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN18 282 OUT685
Algorithms - Bipartite graph check 1 failed	0.0%	IN45 339 OUT4 391
×	BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users IsBipartite did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 20:11: Expected a variable name, found '['.	Failed	IN45 339 OUT4 391
Algorithms - Bit tricks (popcount/pow2/lowbit) 1 passed	100.0%	IN19 252 OUT1 767
✓	BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN19 252 OUT1 767
Algorithms - Caesar cipher 1 failed	0.0%	IN8 735 OUT1 103
×	CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: Caesar([Hello, World!],13) got "8511412112198, 7498101121113!" expected "Uryyb, Jbeyq!" Values: r.ResultJson = "\"8511412112198, 7498101121113!\"" exp = "\"Uryyb, Jbeyq!\""	Failed	IN8 735 OUT1 103
Algorithms - Catalan number 1 passed	100.0%	IN20 136 OUT2 546
✓	CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN20 136 OUT2 546
Algorithms - Chinese remainder theorem 1 passed	100.0%	IN28 269 OUT11 126
✓	CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN28 269 OUT11 126
Algorithms - Circle-circle MTV 1 passed	100.0%	IN8 852 OUT971
✓	CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 852 OUT971
Algorithms - Coin change combinations 1 passed	100.0%	IN18 248 OUT899
✓	CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 248 OUT899
Algorithms - Collatz step count 1 passed	100.0%	IN8 738 OUT806
✓	CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 738 OUT806
Algorithms - Connected components (union-find) 1 passed	100.0%	IN8 754 OUT1 847
✓	UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 754 OUT1 847
Algorithms - Container with most water 1 passed	100.0%	IN8 746 OUT483
✓	ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 746 OUT483
Algorithms - Count anagram groups 1 passed	100.0%	IN19 229 OUT1 847
✓	AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN19 229 OUT1 847
Algorithms - Count grid color regions 1 passed	100.0%	IN19 966 OUT2 816
✓	CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN19 966 OUT2 816
Algorithms - Count islands 1 failed	0.0%	IN44 106 OUT3 951
×	CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users CountIslands did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 41:24: Expected a variable name, found '['.	Failed	IN44 106 OUT3 951
Algorithms - Count overlapping occurrences 1 passed	100.0%	IN8 735 OUT1 131
✓	RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 735 OUT1 131
Algorithms - Count primes (Sieve of Eratosthenes) 1 passed	100.0%	IN8 714 OUT830
✓	CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 714 OUT830
Algorithms - Cumulative distribution 1 passed	100.0%	IN8 736 OUT1 332
✓	CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 736 OUT1 332
Algorithms - Dijkstra shortest path (weighted graph) 1 passed	100.0%	IN18 770 OUT1 705
✓	DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN18 770 OUT1 705
Algorithms - Directed cycle detection 1 passed	100.0%	IN18 728 OUT2 390
✓	DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN18 728 OUT2 390
Algorithms - Distinct subsequences 1 passed	100.0%	IN8 731 OUT490
✓	DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users	Passed	IN8 731 OUT490
Algorithms - Divisor count 1 passed	100.0%	IN8 719 OUT112
✓	DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 719 OUT112
Algorithms - Downsample by block average 1 passed	100.0%	IN29 036 OUT1 634
✓	ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN29 036 OUT1 634
Algorithms - Dutch national flag sort 1 passed	100.0%	IN8 727 OUT1 045
✓	DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 727 OUT1 045
Algorithms - Egg drop trials 1 passed	100.0%	IN8 788 OUT1 370
✓	EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 788 OUT1 370
Algorithms - Equal subset-sum partition 1 passed	100.0%	IN8 730 OUT1 658
✓	PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 730 OUT1 658
Algorithms - Euler totient 1 passed	100.0%	IN8 719 OUT305
✓	EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 719 OUT305
Algorithms - Eulerian path existence 1 passed	100.0%	IN8 778 OUT2 260
✓	EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 778 OUT2 260
Algorithms - Extended Euclidean 1 passed	100.0%	IN19 070 OUT1 570
✓	ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN19 070 OUT1 570
Algorithms - Format duration H:MM:SS 1 passed	100.0%	IN17 575 OUT97
✓	FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN17 575 OUT97
Algorithms - GCD and LCM 1 passed	100.0%	IN18 281 OUT895
✓	GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN18 281 OUT895
Algorithms - Grid transpose 1 failed	0.0%	IN19 420 OUT1 958
×	TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Transpose threw: String index 6 is out of range. The string has 6 character(s).	Failed	IN19 420 OUT1 958
Algorithms - House robber (max non-adjacent sum) 1 passed	100.0%	IN19 060 OUT1 683
✓	HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN19 060 OUT1 683
Algorithms - Integer linear interpolation 1 passed	100.0%	IN17 591 OUT61
✓	LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN17 591 OUT61
Algorithms - Integer square root 1 passed	100.0%	IN8 734 OUT1 306
✓	IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 734 OUT1 306
Algorithms - Integer to base-N string 1 passed	100.0%	IN17 933 OUT572
✓	BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN17 933 OUT572
Algorithms - Integer to Roman numeral 1 passed	100.0%	IN35 819 OUT494
✓	RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN35 819 OUT494
Algorithms - Josephus survivor 1 passed	100.0%	IN8 745 OUT1 217
✓	JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 745 OUT1 217
Algorithms - Jump game (min jumps) 1 failed	0.0%	IN37 432 OUT1 302
×	JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users MinJumps did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 4:18: Type 'List<string>' has no fields. Only struct values support '.Length'.	Failed	IN37 432 OUT1 302
Algorithms - Kth largest element 1 passed	100.0%	IN8 743 OUT647
✓	KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 743 OUT647
Algorithms - Least-squares line fit 1 passed	100.0%	IN31 608 OUT2 925
✓	LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN31 608 OUT2 925
Algorithms - Levenshtein edit distance (DP) 1 passed	100.0%	IN8 713 OUT645
✓	EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 713 OUT645
Algorithms - Log-scale bucket index 1 passed	100.0%	IN18 941 OUT1 435
✓	LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 941 OUT1 435
Algorithms - Logarithmic bin index 1 passed	100.0%	IN8 767 OUT1 553
✓	LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 767 OUT1 553
Algorithms - Longest common prefix 1 passed	100.0%	IN8 726 OUT1 359
✓	CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 726 OUT1 359
Algorithms - Longest common subsequence 1 passed	100.0%	IN8 711 OUT641
✓	LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 711 OUT641
Algorithms - Longest common substring 1 passed	100.0%	IN8 728 OUT1 110
✓	CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 728 OUT1 110
Algorithms - Longest increasing subsequence (DP) 1 passed	100.0%	IN8 745 OUT2 168
✓	LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 745 OUT2 168
Algorithms - Longest palindromic substring 1 passed	100.0%	IN18 339 OUT1 356
✓	LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 339 OUT1 356
Algorithms - Longest repeated substring 1 passed	100.0%	IN19 247 OUT1 862
✓	LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN19 247 OUT1 862
Algorithms - Lower bound (binary search) 1 passed	100.0%	IN8 756 OUT919
✓	LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 756 OUT919
Algorithms - Luhn checksum validation 1 passed	100.0%	IN18 887 OUT1 483
✓	LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 887 OUT1 483
Algorithms - Majority element 1 passed	100.0%	IN18 908 OUT1 671
✓	MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 908 OUT1 671
Algorithms - Matrix chain multiplication 1 failed	0.0%	IN8 756 OUT2 633
×	MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users MinMult threw: List index 5 is out of range. The list has 5 element(s).	Failed	IN8 756 OUT2 633
Algorithms - Maximum subarray sum (Kadane) 1 passed	100.0%	IN8 746 OUT1 409
✓	MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 746 OUT1 409
Algorithms - Merge K sorted lists 1 passed	100.0%	IN18 761 OUT1 401
✓	MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 761 OUT1 401
Algorithms - Merge overlapping intervals 1 passed	100.0%	IN19 081 OUT1 974
✓	MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN19 081 OUT1 974
Algorithms - Min-max normalization 1 passed	100.0%	IN8 748 OUT2 156
✓	MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 748 OUT2 156
Algorithms - Minimum coin change (DP) 1 passed	100.0%	IN8 739 OUT2 110
✓	CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 739 OUT2 110
Algorithms - Minimum knight moves (BFS) 1 failed	0.0%	IN41 015 OUT2 851
×	KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users MinMoves did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 8:8: Expected a variable name, found '['.	Failed	IN41 015 OUT2 851
Algorithms - Minimum path sum (grid DP) 1 passed	100.0%	IN8 752 OUT1 606
✓	MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 752 OUT1 606
Algorithms - Minimum spanning tree (Kruskal) 1 passed	100.0%	IN21 404 OUT4 352
✓	KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN21 404 OUT4 352
Algorithms - Minimum window substring 1 failed	0.0%	IN8 760 OUT1 964
×	MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: res.ResultJson == exp Context: Window(ADOBECODEBANC,ABC) got "" expected "BANC" Values: res.ResultJson = "\"\"" exp = "\"BANC\""	Failed	IN8 760 OUT1 964
Algorithms - Modular exponentiation (fast pow) 1 passed	100.0%	IN34 336 OUT6 771
✓	ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN34 336 OUT6 771
Algorithms - Modular inverse (extended Euclid) 1 passed	100.0%	IN20 332 OUT2 971
✓	ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN20 332 OUT2 971
Algorithms - Moving average 1 passed	100.0%	IN8 750 OUT1 497
✓	MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 750 OUT1 497
Algorithms - Next greater element 1 passed	100.0%	IN8 733 OUT290
✓	NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 733 OUT290
Algorithms - Next lexicographic permutation 1 passed	100.0%	IN8 741 OUT2 038
✓	NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 741 OUT2 038
Algorithms - Nth Fibonacci mod m 1 passed	100.0%	IN19 065 OUT1 950
✓	FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN19 065 OUT1 950
Algorithms - Overlapping substring count 1 passed	100.0%	IN8 725 OUT903
✓	SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 725 OUT903
Algorithms - Palindrome & anagram checks 1 passed	100.0%	IN33 095 OUT3 945
✓	PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN33 095 OUT3 945
Algorithms - Pascal triangle row 1 passed	100.0%	IN8 733 OUT1 757
✓	PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 733 OUT1 757
Algorithms - Point-to-segment distance 1 passed	100.0%	IN8 773 OUT273
✓	PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 773 OUT273
Algorithms - Prime factorization 1 passed	100.0%	IN8 744 OUT1 674
✓	PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 744 OUT1 674
Algorithms - Product of array except self 1 passed	100.0%	IN8 750 OUT2 166
✓	ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 750 OUT2 166
Algorithms - Quadratic Bezier point 1 passed	100.0%	IN18 596 OUT1 093
✓	BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN18 596 OUT1 093
Algorithms - Range sum via prefix sums 1 passed	100.0%	IN8 732 OUT890
✓	PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 732 OUT890
Algorithms - Reverse words & Hamming distance 1 failed	0.0%	IN37 939 OUT1 253

Agentic - Blue-Green Deploy (health-gated cutover) 1 passed

100.0%

IN14 575 OUT179

✓

AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query 4 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN14 575 OUT179

Agentic - Bugfix (root-cause vs regression) 1 passed

100.0%

IN30 876 OUT510

✓

AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query 2 tool callscategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN30 876 OUT510

Agentic - Cache Invalidation (exact set) 1 passed

100.0%

IN17 603 OUT665

✓

AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query 7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN17 603 OUT665

Agentic - Canary (metric-gated release) 1 passed

100.0%

IN46 354 OUT1 381

✓

AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN46 354 OUT1 381

Agentic - Canary variants (5 metric cases) 5 passed

100.0%

IN196 482 OUT5 394

✓

CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN31 055 OUT801

✓

CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query 11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN43 770 OUT1 276

✓

CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN31 067 OUT826

✓

CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query 14 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN56 584 OUT1 612

✓

CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN34 006 OUT879

Agentic - Code Review (find injected defect) 1 passed

100.0%

IN13 477 OUT869

✓

AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN13 477 OUT869

Agentic - Config Drift (remediate exact) 1 passed

100.0%

IN20 945 OUT523

✓

AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query 6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN20 945 OUT523

Agentic - Constraints (multi-section satisfaction) 1 passed

100.0%

IN22 097 OUT806

✓

AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN22 097 OUT806

Agentic - Constraints variants (5 cases) 5 passed

100.0%

IN50 869 OUT2 394

✓

ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN10 668 OUT684

✓

ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN9 833 OUT341

✓

ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN9 870 OUT341

✓

ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN10 428 OUT591

✓

ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN10 070 OUT437

Agentic - Data Reconciliation (per-field precedence) 1 passed

100.0%

IN14 188 OUT849

✓

AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN14 188 OUT849

Agentic - Diagnosis (root cause under budget) 1 failed

0.0%

IN7 256 OUT194

×

AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query 2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Assertion expression evaluated to false: Globals.Get<bool>("diag.has_report", false) Context: Agent did not report a root cause.

Failed

IN7 256 OUT194

Agentic - Diagnosis variants (5 cases) 5 passed

100.0%

IN100 341 OUT4 356

✓

DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN14 841 OUT668

✓

DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN15 258 OUT721

✓

DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN14 902 OUT538

✓

DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN33 162 OUT816

✓

DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query 16 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN22 178 OUT1 613

Agentic - ETL Repair (root-cause in stage 2) 1 passed

100.0%

IN45 548 OUT950

✓

AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN45 548 OUT950

Agentic - Feature Flag (staged rollout + kill-switch) 1 passed

100.0%

IN24 529 OUT925

✓

AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN24 529 OUT925

Agentic - Game Backend: 1D bouncing ball 1 passed

100.0%

IN8 783 OUT224

✓

GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 783 OUT224

Agentic - Game Backend: 1D swept anti-tunneling 1 passed

100.0%

IN19 654 OUT2 260

✓

GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 654 OUT2 260

Agentic - Game Backend: 2048 left-move (slide + merge) 1 passed

100.0%

IN8 803 OUT2 038

✓

GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 803 OUT2 038

Agentic - Game Backend: 8-direction from vector 1 passed

100.0%

IN8 810 OUT260

✓

GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 810 OUT260

Agentic - Game Backend: AABB collision resolution (MTV) 1 passed

100.0%

IN29 252 OUT1 597

✓

GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN29 252 OUT1 597

Agentic - Game Backend: AABB containment 1 passed

100.0%

IN17 557 OUT207

✓

GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN17 557 OUT207

Agentic - Game Backend: Acceleration to max speed 1 passed

100.0%

IN8 741 OUT615

✓

GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 741 OUT615

Agentic - Game Backend: Achievement thresholds 1 passed

100.0%

IN26 404 OUT7 872

✓

GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN26 404 OUT7 872

Agentic - Game Backend: Active status effects 1 failed

0.0%

IN38 557 OUT1 415

×

GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

ActiveAt did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 11:18: Type 'List<string>' has no fields. Only struct values support '.Length'.

Failed

IN38 557 OUT1 415

Agentic - Game Backend: Animation frame at time 1 passed

100.0%

IN8 773 OUT909

✓

GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 773 OUT909

Agentic - Game Backend: Army upkeep income 1 passed

100.0%

IN18 282 OUT901

✓

GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 282 OUT901

Agentic - Game Backend: Axis-aligned path follow 1 passed

100.0%

IN42 775 OUT3 205

✓

GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN42 775 OUT3 205

Agentic - Game Backend: Battleship shot resolution 1 failed

0.0%

IN43 098 OUT3 018

×

GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Fire did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 15:15: Expected a variable name, found '['.

Failed

IN43 098 OUT3 018

Agentic - Game Backend: Bounce restitution 1 passed

100.0%

IN8 760 OUT1 528

✓

GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 760 OUT1 528

Agentic - Game Backend: Bresenham line rasterization 1 passed

100.0%

IN19 826 OUT2 338

✓

GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 826 OUT2 338

Agentic - Game Backend: Buff/debuff stacking 1 passed

100.0%

IN18 354 OUT976

✓

GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 354 OUT976

Agentic - Game Backend: Build queue completions 1 passed

100.0%

IN8 754 OUT2 237

✓

GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 754 OUT2 237

Agentic - Game Backend: Camera-relative movement 1 passed

100.0%

IN8 841 OUT1 257

✓

GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 841 OUT1 257

Agentic - Game Backend: Card hand sort 1 passed

100.0%

IN20 222 OUT3 144

✓

GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN20 222 OUT3 144

Agentic - Game Backend: Card-game rules (blackjack/deck/compare) 1 passed

100.0%

IN18 709 OUT1 463

✓

GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 709 OUT1 463

Agentic - Game Backend: Chart scaling & histogram bins 1 passed

100.0%

IN8 826 OUT898

✓

GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 826 OUT898

Agentic - Game Backend: Checkers captures 1 failed

0.0%

IN8 796 OUT3 414

×

GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Captures threw: String index 24 is out of range. The string has 23 character(s).

Failed

IN8 796 OUT3 414

Agentic - Game Backend: Collision primitives (circle/aabb/segment) 1 failed

0.0%

IN8 999 OUT3 355

×

GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

AabbOverlap did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: Entry point 'AabbOverlap' was not found. Available functions: AskAi, RunScript, SkipTest, __CreateInMemoryDatabase, __DbExecute, __DbExecuteScript, __DbInsert, __DbQueryFirstJson, __DbQueryJson, __DbScalarJson, CircleCircle.

Failed

IN8 999 OUT3 355

Agentic - Game Backend: Combat turn order (initiative) 1 passed

100.0%

IN8 735 OUT2 690

✓

GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 735 OUT2 690

Agentic - Game Backend: Combo scoring (rising multiplier) 1 passed

100.0%

IN8 758 OUT823

✓

GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 758 OUT823

Agentic - Game Backend: Control-point capture 1 passed

100.0%

IN8 771 OUT1 259

✓

GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 771 OUT1 259

Agentic - Game Backend: Controls & Inventory (spec-driven function battery) 1 passed

100.0%

IN18 192 OUT604

✓

GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 192 OUT604

Agentic - Game Backend: Cooldowns & resource regen 1 passed

100.0%

IN8 829 OUT1 436

✓

GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 829 OUT1 436

Agentic - Game Backend: Coyote-time jump 1 passed

100.0%

IN17 584 OUT92

✓

GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN17 584 OUT92

Agentic - Game Backend: Crafting capacity 1 passed

100.0%

IN19 675 OUT2 444

✓

GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 675 OUT2 444

Agentic - Game Backend: Cribbage fifteens 1 passed

100.0%

IN8 751 OUT1 410

✓

GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 751 OUT1 410

Agentic - Game Backend: Critical-hit damage 1 passed

100.0%

IN8 741 OUT335

✓

GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 741 OUT335

Agentic - Game Backend: Damage over time 1 passed

100.0%

IN8 767 OUT1 279

✓

GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 767 OUT1 279

Agentic - Game Backend: Day/night cycle phase 1 passed

100.0%

IN8 753 OUT739

✓

GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 753 OUT739

Agentic - Game Backend: Dice notation max roll 1 passed

100.0%

IN19 962 OUT1 930

✓

GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 962 OUT1 930

Agentic - Game Backend: Enemy AI state machine 1 passed

100.0%

IN8 854 OUT1 206

✓

GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN8 854 OUT1 206

Agentic - Game Backend: Equipment slot validation 1 passed

100.0%

IN17 637 OUT217

✓

GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN17 637 OUT217

Agentic - Game Backend: Flood-fill region size 1 failed

0.0%

IN43 028 OUT3 372

×

GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

RegionSize did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 27:12: Expected a variable name, found '['.

Failed

IN43 028 OUT3 372

Agentic - Game Backend: Fog of war visibility 1 passed

100.0%

IN19 040 OUT1 786

✓

GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN19 040 OUT1 786

Agentic - Game Backend: Fog reveal (Chebyshev radius) 1 failed

0.0%

IN41 127 OUT2 482

×

GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Revealed did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 8:11: Expected a variable name, found '['.

Failed

IN41 127 OUT2 482

Agentic - Game Backend: Friction stop frame 1 passed

100.0%

IN8 749 OUT1 101

✓

GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 749 OUT1 101

Agentic - Game Backend: Game of Life (code-gen vs oracle) 1 failed

0.0%

IN8 808 OUT1 295

×

GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

candidate did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 1:53: Char literals must contain exactly one character and close with '''. 4:24: Unexpected character '`'. 4:61: Unexpected character '`'. 21:8: Char literals must contain exactly one character and close with '''. 26:69: Unexpected character '×'. 37:20: Unexpected character '#'. 39:1: Unexpected character '`'. 39:2: Unexpected character '`'. 39:3: Unexpected character '`'. 105:1: Unexpected character '`'. 105:2: Unexpected character '`'. 105:3: Unexpected character '`'. 107:47: Char literals must contain exactly one character and close with '''. 109:58: Char literals must contain exactly one character and close with '''. 111:1: Unexpected character '`'. 111:2: Unexpected character '`'. 111:3: Unexpected character '`'. 172:1: Unexpected character '`'. 172:2: Unexpected character '`'. 172:3: Unexpected character '`'. 176:84: Char literals must contain exactly one character and close with '''.

Failed

IN8 808 OUT1 295

Agentic - Game Backend: Greedy coin change 1 passed

100.0%

IN18 434 OUT1 061

✓

GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 434 OUT1 061

Agentic - Game Backend: Grid inventory & stacking (Diablo-style) 1 passed

100.0%

IN30 444 OUT2 319

✓

GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.3 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN30 444 OUT2 319

Agentic - Game Backend: Grid pathfinding (shortest path, BFS) 1 passed

100.0%

IN46 467 OUT5 385

✓

GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.4 AI queries 0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN46 467 OUT5 385

Agentic - Game Backend: Grid raycast 1 passed

100.0%

IN19 032 OUT1 612

✓

GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 032 OUT1 612

Agentic - Game Backend: Health bar segments 1 passed

100.0%

IN18 372 OUT904

✓

GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 372 OUT904

Agentic - Game Backend: High-score table insert 1 failed

0.0%

IN43 765 OUT3 349

×

GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Insert did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 7:15: Expected a variable name, found '['.

Failed

IN43 765 OUT3 349

Agentic - Game Backend: Hit chance 1 passed

100.0%

IN17 902 OUT437

✓

GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN17 902 OUT437

Agentic - Game Backend: Inventory encumbrance 1 passed

100.0%

IN29 102 OUT1 517

✓

GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN29 102 OUT1 517

Agentic - Game Backend: Inventory first-fit placement 1 passed

100.0%

IN27 662 OUT872

✓

GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN27 662 OUT872

Agentic - Game Backend: Inventory stack split 1 passed

100.0%

IN17 706 OUT181

✓

GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN17 706 OUT181

Agentic - Game Backend: Isometric projection 1 passed

100.0%

IN8 775 OUT909

✓

GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 775 OUT909

Agentic - Game Backend: Jump buffer 1 passed

100.0%

IN8 763 OUT1 399

✓

GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 763 OUT1 399

Agentic - Game Backend: Knockback with arena clamp 1 passed

100.0%

IN18 645 OUT1 187

✓

GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 645 OUT1 187

Agentic - Game Backend: Leaderboard rank 1 passed

100.0%

IN18 573 OUT1 148

✓

GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 573 OUT1 148

Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over) 1 passed

100.0%

IN18 272 OUT707

✓

GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 272 OUT707

Agentic - Game Backend: Lifesteal healing 1 passed

100.0%

IN8 746 OUT1 081

✓

GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 746 OUT1 081

Agentic - Game Backend: Lights Out toggle 1 failed

0.0%

IN40 224 OUT2 356

×

GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Toggle did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 5:8: Expected a variable name, found '['.

Failed

IN40 224 OUT2 356

Agentic - Game Backend: Match-3 cascade 1 passed

100.0%

IN8 820 OUT1 736

✓

GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN8 820 OUT1 736

Agentic - Game Backend: Match-3 detection 1 passed

100.0%

IN8 784 OUT724

✓

GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 784 OUT724

Agentic - Game Backend: Match-3 swap validity 1 passed

100.0%

IN29 785 OUT2 283

✓

GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN29 785 OUT2 283

Agentic - Game Backend: Memory match pairs 1 passed

100.0%

IN8 747 OUT1 972

✓

GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 747 OUT1 972

Agentic - Game Backend: Mercy invincibility (i-frames) 1 passed

100.0%

IN19 586 OUT2 105

✓

GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 586 OUT2 105

Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal) 1 passed

100.0%

IN21 533 OUT4 599

✓

GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN21 533 OUT4 599

Agentic - Game Backend: Minimap downsample 1 passed

100.0%

IN8 778 OUT457

✓

GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 778 OUT457

Agentic - Game Backend: Minimap marker position 1 passed

100.0%

IN18 893 OUT1 398

✓

GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 893 OUT1 398

Agentic - Game Backend: N-in-a-row win detection (Connect-4) 1 failed

0.0%

IN45 202 OUT3 912

×

GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.4 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Winner did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 3:8: Expected a variable name, found '['.

Failed

IN45 202 OUT3 912

Agentic - Game Backend: Nonogram row validation 1 passed

100.0%

IN19 407 OUT2 121

✓

GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 407 OUT2 121

Agentic - Game Backend: OBB collision (rotating rectangles, SAT) 1 passed

100.0%

IN8 905 OUT7 873

✓

GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN8 905 OUT7 873

Agentic - Game Backend: Particle lifetime & expiry 1 passed

100.0%

IN8 840 OUT1 296

✓

GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 840 OUT1 296

Agentic - Game Backend: Party XP share 1 passed

100.0%

IN8 758 OUT1 626

✓

GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 758 OUT1 626

Agentic - Game Backend: Patrol waypoints (ping-pong) 1 passed

100.0%

IN8 781 OUT1 316

✓

GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 781 OUT1 316

Agentic - Game Backend: Platformer jump arc 1 passed

100.0%

IN8 772 OUT109

✓

GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 772 OUT109

Agentic - Game Backend: Point on circle (trig) 1 passed

100.0%

IN28 208 OUT942

✓

GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN28 208 OUT942

Agentic - Game Backend: Poker hand comparison 1 failed

0.0%

IN69 550 OUT15 303

×

GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users

Compare did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 29:28: Expected '{' after the struct name. Structs are created with 'new Name { Field = value }', found '('.

Failed

IN69 550 OUT15 303

Agentic - Game Backend: Poker hand ranking 1 passed

100.0%

IN19 436 OUT2 218

✓

GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN19 436 OUT2 218

Agentic - Game Backend: Pop-cap training 1 passed

100.0%

IN8 759 OUT1 162

✓

GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 759 OUT1 162

Agentic - Game Backend: Progress bar fill 1 passed

100.0%

IN8 741 OUT709

✓

GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 741 OUT709

Agentic - Game Backend: Projectile range (physics/trig) 1 passed

100.0%

IN8 763 OUT588

✓

GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 763 OUT588

Agentic - Game Backend: Projectile with ground bounce 1 passed

100.0%

IN28 299 OUT753

✓

GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN28 299 OUT753

Agentic - Game Backend: Resource gathering 1 passed

100.0%

IN8 758 OUT463

✓

GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 758 OUT463

Agentic - Game Backend: Reversi flip count 1 failed

0.0%

IN38 702 OUT1 702

×

GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

FlipCount did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 11:8: Expected a variable name, found '['.

Failed

IN38 702 OUT1 702

Agentic - Game Backend: Rotate 2D vector (trig) 1 passed

100.0%

IN17 966 OUT465

✓

GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN17 966 OUT465

Agentic - Game Backend: Round-robin deal 1 passed

100.0%

IN18 512 OUT1 039

✓

GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 512 OUT1 039

Agentic - Game Backend: RTS resource economy 1 passed

100.0%

IN8 754 OUT201

✓

GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 754 OUT201

Agentic - Game Backend: RTS unit counters & damage 1 passed

100.0%

IN8 806 OUT1 019

✓

GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 806 OUT1 019

Agentic - Game Backend: Safest neighbour (danger map) 1 passed

100.0%

IN19 592 OUT2 413

✓

GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 592 OUT2 413

Agentic - Game Backend: Scroll viewport 1 passed

100.0%

IN8 776 OUT1 657

✓

GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 776 OUT1 657

Agentic - Game Backend: Set card validity 1 passed

100.0%

IN8 755 OUT673

✓

GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 755 OUT673

Agentic - Game Backend: Shop affordability 1 passed

100.0%

IN8 735 OUT787

✓

GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 735 OUT787

Agentic - Game Backend: Sliding puzzle move 1 passed

100.0%

IN19 864 OUT2 529

✓

GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 864 OUT2 529

Agentic - Game Backend: Snake head movement 1 passed

100.0%

IN19 234 OUT1 822

✓

GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 234 OUT1 822

Agentic - Game Backend: Snake tick 1 failed

0.0%

IN19 448 OUT1 901

×

GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Tick [1,0;0,0] R food(9,9) got "2,0" expected "2,0;1,0" Values: r.ResultJson = "\"2,0\"" exp = "\"2,0;1,0\""

Failed

IN19 448 OUT1 901

Agentic - Game Backend: Snap to grid 1 passed

100.0%

IN18 930 OUT1 459

✓

GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 930 OUT1 459

Agentic - Game Backend: Sokoban move/push legality 1 passed

100.0%

IN8 854 OUT662

✓

GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 854 OUT662

Agentic - Game Backend: Solitaire tableau move 1 passed

100.0%

IN8 793 OUT1 962

✓

GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 793 OUT1 962

Agentic - Game Backend: Spawn scheduler count 1 passed

100.0%

IN8 750 OUT779

✓

GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 750 OUT779

Agentic - Game Backend: Spawn-wave alive count 1 passed

100.0%

IN8 762 OUT1 125

✓

GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 762 OUT1 125

Agentic - Game Backend: Stat level scaling 1 passed

100.0%

IN19 412 OUT207

✓

GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query 1 tool callcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 412 OUT207

Agentic - Game Backend: Steering seek (greedy pursuit) 1 failed

0.0%

IN8 780 OUT1 208

×

GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Seek(0,0->5,2,3) got "3,3" expected "3,2" Values: r.ResultJson = "\"3,3\"" exp = "\"3,2\""

Failed

IN8 780 OUT1 208

Agentic - Game Backend: Tactical movement range 1 passed

100.0%

IN20 901 OUT3 743

✓

GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN20 901 OUT3 743

Agentic - Game Backend: Tech tree eligibility 1 failed

0.0%

IN44 788 OUT3 912

×

GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

CanResearch did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 4:11: Expected a variable name, found '['.

Failed

IN44 788 OUT3 912

Agentic - Game Backend: Tetris line clearing 1 passed

100.0%

IN20 168 OUT2 822

✓

GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN20 168 OUT2 822

Agentic - Game Backend: Threat table top target 1 passed

100.0%

IN8 759 OUT2 364

✓

GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 759 OUT2 364

Agentic - Game Backend: Tic-tac-toe result 1 failed

0.0%

IN17 568 OUT172

×

GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Winner([XOXOXOOXO]) got "none" expected "draw" Values: r.ResultJson = "\"none\"" exp = "\"draw\""

Failed

IN17 568 OUT172

Agentic - Game Backend: Toroidal wrap (Pac-Man) 1 passed

100.0%

IN28 003 OUT471

✓

GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN28 003 OUT471

Agentic - Game Backend: Tower targeting (nearest in range) 1 passed

100.0%

IN8 774 OUT1 495

✓

GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 774 OUT1 495

Agentic - Game Backend: Transaction ledger 1 passed

100.0%

IN19 396 OUT1 978

✓

GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN19 396 OUT1 978

Agentic - Game Backend: Trick-taking winner 1 passed

100.0%

IN18 756 OUT1 551

✓

GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN18 756 OUT1 551

Agentic - Game Backend: Turn-based duel resolution 1 passed

100.0%

IN8 837 OUT1 753

✓

GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 837 OUT1 753

Agentic - Game Backend: Typed armor mitigation 1 passed

100.0%

IN18 343 OUT1 020

✓

GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 343 OUT1 020

Agentic - Game Backend: UI menu (wrap/clamp/paginate) 1 failed

0.0%

IN8 853 OUT1 817

×

GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

ClampSlider did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: Entry point 'ClampSlider' was not found. Available functions: AskAi, RunScript, SkipTest, __CreateInMemoryDatabase, __DbExecute, __DbExecuteScript, __DbInsert, __DbQueryFirstJson, __DbQueryJson, __DbScalarJson, MenuMove.

Failed

IN8 853 OUT1 817

Agentic - Game Backend: Uno playable cards 1 passed

100.0%

IN19 267 OUT1 834

✓

GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN19 267 OUT1 834

Agentic - Game Backend: Velocity reflection 1 passed

100.0%

IN18 441 OUT1 072

✓

GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 441 OUT1 072

Agentic - Game Backend: Wall-slide fall 1 passed

100.0%

IN8 741 OUT628

✓

GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 741 OUT628

Agentic - Game Backend: War card compare 1 passed

100.0%

IN8 761 OUT1 373

✓

GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 761 OUT1 373

Agentic - Game Backend: Weighted loot table 1 passed

100.0%

IN8 764 OUT957

✓

GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 764 OUT957

Agentic - Game Backend: Weighted terrain pathfinding 1 failed

0.0%

IN45 939 OUT5 104

×

GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

MinCost did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 38:8: Expected a variable name, found '['.

Failed

IN45 939 OUT5 104

Agentic - Game Backend: Word-wrap line count 1 passed

100.0%

IN8 746 OUT332

✓

GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 746 OUT332

Agentic - Game Backend: XP to character level 1 passed

100.0%

IN8 754 OUT880

✓

GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 754 OUT880

Agentic - Idempotency (poison-retry / no double charge) 1 passed

100.0%

IN19 573 OUT375

✓

AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN19 573 OUT375

Agentic - Irreversible Commit (measure twice) 1 failed

0.0%

IN5 416 OUT158

×

AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query 1 tool callcategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Assertion expression evaluated to false: Globals.Get<bool>("irr.finalized", false) Context: Billing was never finalized.

Failed

IN5 416 OUT158

Agentic - Least Privilege (exact grants) 1 passed

100.0%

IN14 872 OUT424

✓

AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query 6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN14 872 OUT424

Agentic - Ledger (simulated DB read-modify-write) 1 passed

100.0%

IN13 695 OUT1 045

✓

AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN13 695 OUT1 045

Agentic - Ledger variants (5 RMW cases) 5 passed

100.0%

IN120 309 OUT5 904

✓

LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query 11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN28 327 OUT1 345

✓

LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN13 611 OUT928

✓

LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query 15 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN40 892 OUT2 217

✓

LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN15 760 OUT856

✓

LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN21 719 OUT558

Agentic - Lock Ordering (deadlock avoidance) 1 passed

100.0%

IN123 128 OUT2 144

✓

AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query 25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN123 128 OUT2 144

Agentic - Lock Ordering variants (5 cases) 4 passed1 failed

80.0%

IN344 374 OUT7 321

✓

LockVariantFiveResFourOps1 AI query 23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN70 607 OUT1 797

✓

LockVariantFourResThreeOps1 AI query 18 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN77 235 OUT1 450

×

LockVariantHeavyReuse1 AI query 1 tool callcategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Assertion expression evaluated to false: Globals.Get<bool>("op.done." + op, false) Context: Operation not performed: op_1

Failed

IN5 551 OUT148

✓

LockVariantReverseListed1 AI query 24 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN117 208 OUT2 170

✓

LockVariantThreeLockOp1 AI query 21 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN73 773 OUT1 756

Agentic - Migration (cascading rollback + reapply) 1 passed

100.0%

IN42 988 OUT1 024

✓

AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query 11 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN42 988 OUT1 024

Agentic - Requirements (detect contradiction) 1 passed

100.0%

IN7 620 OUT532

✓

AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query 2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN7 620 OUT532

Agentic - Retry/Circuit-breaker (transient vs hard-down) 1 passed

100.0%

IN35 512 OUT843

✓

AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query 10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN35 512 OUT843

Agentic - Saga (compensation / distributed rollback) 1 passed

100.0%

IN21 132 OUT375

✓

AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN21 132 OUT375

Agentic - Saga variants (5 compensation cases) 5 passed

100.0%

IN94 084 OUT1 895

✓

SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN13 407 OUT206

✓

SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN13 639 OUT292

✓

SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN19 736 OUT283

✓

SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN20 183 OUT491

✓

SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query 8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN27 119 OUT623

Agentic - Schema Compat (expand-contract) 1 passed

100.0%

IN25 135 OUT417

✓

AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query 7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN25 135 OUT417

Agentic - Scratchpad (self-repair, Dyck, tool-decision) 7 passed

100.0%

IN236 697 OUT9 435

✓

AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query 3 tool callscategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN46 369 OUT2 432

✓

AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query 4 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN57 487 OUT1 955

✓

ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN21 004 OUT120

✓

ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN21 421 OUT597

✓

ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN21 184 OUT382

✓

ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query 2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN36 045 OUT2 848

✓

ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query 2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN33 187 OUT1 101

Agentic - Secret Rotation (zero-downtime ordering) 1 passed

100.0%

IN17 772 OUT721

✓

AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query 5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN17 772 OUT721

Agentic - StatefulOps (incident recovery, concurrency replan) 2 passed

100.0%

IN56 197 OUT1 851

✓

AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query 9 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN23 803 OUT906

✓

AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query 8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN32 394 OUT945

Agentic - Test Authoring (write a test that catches the bug) 1 passed

100.0%

IN11 752 OUT695

✓

AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query 6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN11 752 OUT695

Agentic - Topological Planning (deps + prereq) 1 passed

100.0%

IN30 478 OUT942

✓

AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN30 478 OUT942

Algorithms - 0/1 Knapsack (dynamic programming) 1 failed

0.0%

IN41 252 OUT2 608

×

KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Knapsack did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 10:15: Expected a variable name, found '['.

Failed

IN41 252 OUT2 608

Algorithms - 1D elastic collision 1 passed

100.0%

IN8 808 OUT137

✓

ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 808 OUT137

Algorithms - All-pairs shortest path pair 1 failed

0.0%

IN55 106 OUT7 982

×

FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Dist did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 23:22: Type 'List<string>' has no fields. Only struct values support '.Length'.

Failed

IN55 106 OUT7 982

Algorithms - Balanced brackets (stack) 1 passed

100.0%

IN18 697 OUT1 374

✓

ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN18 697 OUT1 374

Algorithms - Bellman-Ford shortest path (neg weights/cycles) 1 failed

0.0%

IN55 728 OUT7 793

×

BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

ShortestPath did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 11:30: Operator '!=' cannot compare 'string' and 'null'. Only numbers, bools, chars, strings, date/time values and recursively comparable containers/structs can be compared. 41:52: Operator '<' requires numeric, char, string or matching date/time operands, but got 'long' and 'long?'. 60:48: Operator '<' requires numeric, char, string or matching date/time operands, but got 'long' and 'long?'.

Failed

IN55 728 OUT7 793

Algorithms - Binary Gray code 1 passed

100.0%

IN18 282 OUT685

✓

GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN18 282 OUT685

Algorithms - Bipartite graph check 1 failed

0.0%

IN45 339 OUT4 391

×

BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

IsBipartite did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 20:11: Expected a variable name, found '['.

Failed

IN45 339 OUT4 391

Algorithms - Bit tricks (popcount/pow2/lowbit) 1 passed

100.0%

IN19 252 OUT1 767

✓

BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN19 252 OUT1 767

Algorithms - Caesar cipher 1 failed

0.0%

IN8 735 OUT1 103

×

CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Caesar([Hello, World!],13) got "8511412112198, 7498101121113!" expected "Uryyb, Jbeyq!" Values: r.ResultJson = "\"8511412112198, 7498101121113!\"" exp = "\"Uryyb, Jbeyq!\""

Failed

IN8 735 OUT1 103

Algorithms - Catalan number 1 passed

100.0%

IN20 136 OUT2 546

✓

CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN20 136 OUT2 546

Algorithms - Chinese remainder theorem 1 passed

100.0%

IN28 269 OUT11 126

✓

CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN28 269 OUT11 126

Algorithms - Circle-circle MTV 1 passed

100.0%

IN8 852 OUT971

✓

CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 852 OUT971

Algorithms - Coin change combinations 1 passed

100.0%

IN18 248 OUT899

✓

CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 248 OUT899

Algorithms - Collatz step count 1 passed

100.0%

IN8 738 OUT806

✓

CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 738 OUT806

Algorithms - Connected components (union-find) 1 passed

100.0%

IN8 754 OUT1 847

✓

UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 754 OUT1 847

Algorithms - Container with most water 1 passed

100.0%

IN8 746 OUT483

✓

ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 746 OUT483

Algorithms - Count anagram groups 1 passed

100.0%

IN19 229 OUT1 847

✓

AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN19 229 OUT1 847

Algorithms - Count grid color regions 1 passed

100.0%

IN19 966 OUT2 816

✓

CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN19 966 OUT2 816

Algorithms - Count islands 1 failed

0.0%

IN44 106 OUT3 951

×

CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

CountIslands did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 41:24: Expected a variable name, found '['.

Failed

IN44 106 OUT3 951

Algorithms - Count overlapping occurrences 1 passed

100.0%

IN8 735 OUT1 131

✓

RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 735 OUT1 131

Algorithms - Count primes (Sieve of Eratosthenes) 1 passed

100.0%

IN8 714 OUT830

✓

CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 714 OUT830

Algorithms - Cumulative distribution 1 passed

100.0%

IN8 736 OUT1 332

✓

CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 736 OUT1 332

Algorithms - Dijkstra shortest path (weighted graph) 1 passed

100.0%

IN18 770 OUT1 705

✓

DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN18 770 OUT1 705

Algorithms - Directed cycle detection 1 passed

100.0%

IN18 728 OUT2 390

✓

DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN18 728 OUT2 390

Algorithms - Distinct subsequences 1 passed

100.0%

IN8 731 OUT490

✓

DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN8 731 OUT490

Algorithms - Divisor count 1 passed

100.0%

IN8 719 OUT112

✓

DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 719 OUT112

Algorithms - Downsample by block average 1 passed

100.0%

IN29 036 OUT1 634

✓

ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN29 036 OUT1 634

Algorithms - Dutch national flag sort 1 passed

100.0%

IN8 727 OUT1 045

✓

DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 727 OUT1 045

Algorithms - Egg drop trials 1 passed

100.0%

IN8 788 OUT1 370

✓

EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 788 OUT1 370

Algorithms - Equal subset-sum partition 1 passed

100.0%

IN8 730 OUT1 658

✓

PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 730 OUT1 658

Algorithms - Euler totient 1 passed

100.0%

IN8 719 OUT305

✓

EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 719 OUT305

Algorithms - Eulerian path existence 1 passed

100.0%

IN8 778 OUT2 260

✓

EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 778 OUT2 260

Algorithms - Extended Euclidean 1 passed

100.0%

IN19 070 OUT1 570

✓

ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN19 070 OUT1 570

Algorithms - Format duration H:MM:SS 1 passed

100.0%

IN17 575 OUT97

✓

FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN17 575 OUT97

Algorithms - GCD and LCM 1 passed

100.0%

IN18 281 OUT895

✓

GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN18 281 OUT895

Algorithms - Grid transpose 1 failed

0.0%

IN19 420 OUT1 958

×

TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Transpose threw: String index 6 is out of range. The string has 6 character(s).

Failed

IN19 420 OUT1 958

Algorithms - House robber (max non-adjacent sum) 1 passed

100.0%

IN19 060 OUT1 683

✓

HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN19 060 OUT1 683

Algorithms - Integer linear interpolation 1 passed

100.0%

IN17 591 OUT61

✓

LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN17 591 OUT61

Algorithms - Integer square root 1 passed

100.0%

IN8 734 OUT1 306

✓

IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 734 OUT1 306

Algorithms - Integer to base-N string 1 passed

100.0%

IN17 933 OUT572

✓

BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN17 933 OUT572

Algorithms - Integer to Roman numeral 1 passed

100.0%

IN35 819 OUT494

✓

RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN35 819 OUT494

Algorithms - Josephus survivor 1 passed

100.0%

IN8 745 OUT1 217

✓

JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 745 OUT1 217

Algorithms - Jump game (min jumps) 1 failed

0.0%

IN37 432 OUT1 302

×

JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

MinJumps did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 4:18: Type 'List<string>' has no fields. Only struct values support '.Length'.

Failed

IN37 432 OUT1 302

Algorithms - Kth largest element 1 passed

100.0%

IN8 743 OUT647

✓

KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 743 OUT647

Algorithms - Least-squares line fit 1 passed

100.0%

IN31 608 OUT2 925

✓

LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN31 608 OUT2 925

Algorithms - Levenshtein edit distance (DP) 1 passed

100.0%

IN8 713 OUT645

✓

EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 713 OUT645

Algorithms - Log-scale bucket index 1 passed

100.0%

IN18 941 OUT1 435

✓

LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 941 OUT1 435

Algorithms - Logarithmic bin index 1 passed

100.0%

IN8 767 OUT1 553

✓

LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 767 OUT1 553

Algorithms - Longest common prefix 1 passed

100.0%

IN8 726 OUT1 359

✓

CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 726 OUT1 359

Algorithms - Longest common subsequence 1 passed

100.0%

IN8 711 OUT641

✓

LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 711 OUT641

Algorithms - Longest common substring 1 passed

100.0%

IN8 728 OUT1 110

✓

CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 728 OUT1 110

Algorithms - Longest increasing subsequence (DP) 1 passed

100.0%

IN8 745 OUT2 168

✓

LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 745 OUT2 168

Algorithms - Longest palindromic substring 1 passed

100.0%

IN18 339 OUT1 356

✓

LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 339 OUT1 356

Algorithms - Longest repeated substring 1 passed

100.0%

IN19 247 OUT1 862

✓

LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN19 247 OUT1 862

Algorithms - Lower bound (binary search) 1 passed

100.0%

IN8 756 OUT919

✓

LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 756 OUT919

Algorithms - Luhn checksum validation 1 passed

100.0%

IN18 887 OUT1 483

✓

LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 887 OUT1 483

Algorithms - Majority element 1 passed

100.0%

IN18 908 OUT1 671

✓

MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 908 OUT1 671

Algorithms - Matrix chain multiplication 1 failed

0.0%

IN8 756 OUT2 633

×

MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

MinMult threw: List index 5 is out of range. The list has 5 element(s).

Failed

IN8 756 OUT2 633

Algorithms - Maximum subarray sum (Kadane) 1 passed

100.0%

IN8 746 OUT1 409

✓

MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 746 OUT1 409

Algorithms - Merge K sorted lists 1 passed

100.0%

IN18 761 OUT1 401

✓

MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 761 OUT1 401

Algorithms - Merge overlapping intervals 1 passed

100.0%

IN19 081 OUT1 974

✓

MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN19 081 OUT1 974

Algorithms - Min-max normalization 1 passed

100.0%

IN8 748 OUT2 156

✓

MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 748 OUT2 156

Algorithms - Minimum coin change (DP) 1 passed

100.0%

IN8 739 OUT2 110

✓

CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 739 OUT2 110

Algorithms - Minimum knight moves (BFS) 1 failed

0.0%

IN41 015 OUT2 851

×

KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

MinMoves did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 8:8: Expected a variable name, found '['.

Failed

IN41 015 OUT2 851

Algorithms - Minimum path sum (grid DP) 1 passed

100.0%

IN8 752 OUT1 606

✓

MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 752 OUT1 606

Algorithms - Minimum spanning tree (Kruskal) 1 passed

100.0%

IN21 404 OUT4 352

✓

KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN21 404 OUT4 352

Algorithms - Minimum window substring 1 failed

0.0%

IN8 760 OUT1 964

×

MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Assertion expression evaluated to false: res.ResultJson == exp Context: Window(ADOBECODEBANC,ABC) got "" expected "BANC" Values: res.ResultJson = "\"\"" exp = "\"BANC\""

Failed

IN8 760 OUT1 964

Algorithms - Modular exponentiation (fast pow) 1 passed

100.0%

IN34 336 OUT6 771

✓

ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN34 336 OUT6 771

Algorithms - Modular inverse (extended Euclid) 1 passed

100.0%

IN20 332 OUT2 971

✓

ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN20 332 OUT2 971

Algorithms - Moving average 1 passed

100.0%

IN8 750 OUT1 497

✓

MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 750 OUT1 497

Algorithms - Next greater element 1 passed

100.0%

IN8 733 OUT290

✓

NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 733 OUT290

Algorithms - Next lexicographic permutation 1 passed

100.0%

IN8 741 OUT2 038

✓

NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 741 OUT2 038

Algorithms - Nth Fibonacci mod m 1 passed

100.0%

IN19 065 OUT1 950

✓

FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN19 065 OUT1 950

Algorithms - Overlapping substring count 1 passed

100.0%

IN8 725 OUT903

✓

SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 725 OUT903

Algorithms - Palindrome & anagram checks 1 passed

100.0%

IN33 095 OUT3 945

✓

PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN33 095 OUT3 945

Algorithms - Pascal triangle row 1 passed

100.0%

IN8 733 OUT1 757

✓

PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 733 OUT1 757

Algorithms - Point-to-segment distance 1 passed

100.0%

IN8 773 OUT273

✓

PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 773 OUT273

Algorithms - Prime factorization 1 passed

100.0%

IN8 744 OUT1 674

✓

PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 744 OUT1 674

Algorithms - Product of array except self 1 passed

100.0%

IN8 750 OUT2 166

✓

ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 750 OUT2 166

Algorithms - Quadratic Bezier point 1 passed

100.0%

IN18 596 OUT1 093

✓

BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN18 596 OUT1 093

Algorithms - Range sum via prefix sums 1 passed

100.0%

IN8 732 OUT890

✓

PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 732 OUT890

Algorithms - Reverse words & Hamming distance 1 failed

0.0%

IN37 939 OUT1 253

×

ReverseWordsCodeGenCode-gen: model implements reverse-words + Hamming; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

ReverseWords did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 7:18: Type 'List<string>' has no fields. Only struct values support '.Length'.

Failed

IN37 939 OUT1 253

Algorithms - Rod cutting 1 passed

100.0%

IN8 766 OUT1 450

✓

RodCuttingCodeGenCode-gen: model implements rod cutting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 766 OUT1 450

Algorithms - Roman numeral to integer 1 passed

100.0%

IN8 755 OUT977

✓

RomanToIntCodeGenCode-gen: model implements Roman-to-int; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 755 OUT977

Algorithms - Rotate array right by k 1 failed

0.0%

IN40 281 OUT2 214

×

RotateArrayCodeGenCode-gen: model implements right rotation; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

RotateRight did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 8:11: Expected a variable name, found '['.

Failed

IN40 281 OUT2 214

Algorithms - Rotate grid 90 CW (discriminator) 1 passed

100.0%

IN25 551 OUT8 083

✓

RotateMatrixCodeGenCode-gen: model implements 90-degree CW rotation; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN25 551 OUT8 083

Algorithms - RPN evaluator 1 passed

100.0%

IN19 255 OUT1 916

✓

RpnCodeGenCode-gen: model implements an RPN evaluator; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN19 255 OUT1 916

Algorithms - Run-length compression (if shorter) 1 passed

100.0%

IN29 129 OUT1 660

✓

StringCompressCodeGenCode-gen: model implements string compression; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN29 129 OUT1 660

Algorithms - Run-length encoding (encode/decode) 1 passed

100.0%

IN8 782 OUT458

✓

RunLengthCodeGenCode-gen: model implements RLE encode+decode; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 782 OUT458

Algorithms - Semantic versioning (compare + bump) 1 passed

100.0%

IN18 179 OUT818

✓

SemVerCodeGenCode-gen: model implements semver compare + bump; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:InstructionFollowinglevel:Easysource:Ai4Users

Passed

IN18 179 OUT818

Algorithms - Single-pile Nim (optimal play) 1 passed

100.0%

IN8 744 OUT701

✓

NimCodeGenCode-gen: model implements optimal Nim; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 744 OUT701

Algorithms - Sliding window maximum 1 passed

100.0%

IN29 110 OUT1 863

✓

SlidingWindowMaxCodeGenCode-gen: model implements sliding window maximum; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN29 110 OUT1 863

Algorithms - Spiral matrix traversal 1 passed

100.0%

IN8 745 OUT433

✓

SpiralMatrixCodeGenCode-gen: model implements spiral traversal; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 745 OUT433

Algorithms - Square matrix multiply 1 passed

100.0%

IN8 737 OUT1 299

✓

MatMulCodeGenCode-gen: model implements square matrix multiply; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 737 OUT1 299

Algorithms - Strongly connected components 1 failed

0.0%

IN47 841 OUT5 536

×

SccCountCodeGenCode-gen: model implements SCC count; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Count did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 49:59: Expected ')' to close the argument list, found 'stack'.

Failed

IN47 841 OUT5 536

Algorithms - Subset sum reachability 1 passed

100.0%

IN19 873 OUT2 694

✓

SubsetSumCodeGenCode-gen: model implements subset-sum reachability; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN19 873 OUT2 694

Algorithms - Substring search (KMP) 1 passed

100.0%

IN19 588 OUT2 424

✓

KmpCodeGenCode-gen: model implements KMP search; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN19 588 OUT2 424

Algorithms - Sum of digits & digital root 1 passed

100.0%

IN17 517 OUT202

✓

DigitalRootCodeGenCode-gen: model implements sum-of-digits + digital root; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN17 517 OUT202

Algorithms - Temperature conversion 1 passed

100.0%

IN8 733 OUT366

✓

TemperatureCodeGenCode-gen: model implements temperature conversion; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 733 OUT366

Algorithms - Topological sort (lexicographically smallest) 1 passed

100.0%

IN8 774 OUT6 823

✓

TopoCodeGenCode-gen: model implements lexicographically-smallest topological sort.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 774 OUT6 823

Algorithms - Trailing zeros of factorial 1 passed

100.0%

IN8 734 OUT800

✓

TrailingZerosCodeGenCode-gen: model implements trailing zeros; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 734 OUT800

Algorithms - Trapping rain water 1 passed

100.0%

IN35 894 OUT5 300

✓

TrappingRainCodeGenCode-gen: model implements trapping rain water; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN35 894 OUT5 300

Algorithms - Two-sum indices 1 passed

100.0%

IN8 740 OUT1 316

✓

TwoSumCodeGenCode-gen: model implements two-sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 740 OUT1 316

Algorithms - Unique grid paths 1 passed

100.0%

IN8 715 OUT1 436

✓

UniquePathsCodeGenCode-gen: model implements unique paths; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 715 OUT1 436

Algorithms - Valid Sudoku unit 1 passed

100.0%

IN8 720 OUT902

✓

SudokuUnitCodeGenCode-gen: model implements Sudoku unit validation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 720 OUT902

Algorithms - Wildcard string matching 1 passed

100.0%

IN8 738 OUT3 368

✓

WildcardMatchCodeGenCode-gen: model implements wildcard match; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 738 OUT3 368

Algorithms - Z-function 1 passed

100.0%

IN8 739 OUT1 521

✓

ZFunctionCodeGenCode-gen: model implements the Z-function; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 739 OUT1 521

Database - Conditional aggregation pivot 1 passed

100.0%

IN947 OUT63

✓

PivotCountsCodeGenCode-gen: model writes the pivot query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN947 OUT63

Database - Count distinct opponents 1 passed

100.0%

IN927 OUT19

✓

DistinctCountCodeGenCode-gen: model writes the COUNT(DISTINCT) query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Easysource:Ai4Users

Passed

IN927 OUT19

Database - Date-range filter (BETWEEN) 1 passed

100.0%

IN957 OUT42

✓

DateRangeCodeGenCode-gen: model writes the date-range query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN957 OUT42

Database - EXISTS subquery (teams with matches) 1 passed

100.0%

IN934 OUT27

✓

ExistsSubqueryCodeGenCode-gen: model writes the EXISTS query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN934 OUT27

Database - JOIN player totals (two tables) 1 passed

100.0%

IN956 OUT36

✓

JoinScoresCodeGenCode-gen: model writes a JOIN + aggregate query; graded via QueryJson vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN956 OUT36

Database - LEFT JOIN anti-join (no items) 1 passed

100.0%

IN930 OUT29

✓

LeftJoinNullsCodeGenCode-gen: model writes the anti-join query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN930 OUT29

Database - Player stats SQL (multi-aggregate, QueryJson) 1 passed

100.0%

IN960 OUT36

✓

PlayerStatsCodeGenCode-gen: model writes a multi-aggregate stats query; graded by comparing its QueryJson output to the oracle query's (arbitrary shape, no struct).1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN960 OUT36

Database - Players above average score (subquery) 1 passed

100.0%

IN925 OUT26

✓

AboveAverageCodeGenCode-gen: model writes an above-average query; graded via QueryJson vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN925 OUT26

Database - Players who never played (anti-join) 1 passed

100.0%

IN922 OUT29

✓

NeverPlayedCodeGenCode-gen: model writes an anti-join query; graded via QueryJson vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN922 OUT29

Database - RANK window function 1 passed

100.0%

IN952 OUT27

✓

WindowRankCodeGenCode-gen: model writes the RANK() query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN952 OUT27

Database - Running total (window function) 1 passed

100.0%

IN949 OUT36

✓

RunningTotalCodeGenCode-gen: model writes the window running-total query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN949 OUT36

Database - Scoreboard SQL (players with >1 win, HAVING) 1 passed

100.0%

IN956 OUT33

✓

WinFilterCodeGenCode-gen: model writes a HAVING query; runs under try/catch with self-repair; result compared to the oracle query.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN956 OUT33

Database - Scoreboard SQL (top scorers query) 1 passed

100.0%

IN957 OUT27

✓

ScoreboardTopScorersCodeGenCode-gen: model writes a scoreboard SQL query; runs under try/catch with a self-repair loop on SQL errors; result compared to the oracle query.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN957 OUT27

Database - Self-join same-team pairs 1 passed

100.0%

IN940 OUT44

✓

SelfJoinCodeGenCode-gen: model writes the self-join query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN940 OUT44

Database - Top scorer per region 1 passed

100.0%

IN942 OUT51

✓

TopPerGroupCodeGenCode-gen: model writes the per-group top query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN942 OUT51

Database - UPDATE restock (SQL write) 1 passed

100.0%

IN934 OUT17

✓

UpdateInventoryCodeGenCode-gen: model writes an UPDATE; applied under try/catch, resulting state compared (via QueryJson) to the oracle update's state.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN934 OUT17

Database - Upsert with max merge 1 passed

100.0%

IN950 OUT35

✓

UpsertCodeGenCode-gen: model writes the upsert; executed on a fresh DB and the table compared.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN950 OUT35

SDLC - Acceptance criteria check 1 failed

0.0%

IN8 763 OUT1 638

×

SdlcAcceptanceCodeGenCode-gen: model implements acceptance-criteria check; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Failing([latency:<=:100;uptime:>=:99],[latency:80;uptime:99]) got "latency uptime" expected "none" Values: r.ResultJson = "\"latency uptime\"" exp = "\"none\""

Failed

IN8 763 OUT1 638

SDLC - Breaking API change detection 1 passed

100.0%

IN19 846 OUT2 678

✓

SdlcApiBreakingCodeGenCode-gen: model implements API-breaking detection; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN19 846 OUT2 678

SDLC - Condense without losing facts 1 passed

100.0%

IN1 048 OUT55

✓

NoInfoLossCodeGenRedaction test: model condenses keeping facts; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 048 OUT55

SDLC - Config merge with override 1 passed

100.0%

IN18 471 OUT1 196

✓

ConfigMergeCodeGenCode-gen: model implements config merge; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN18 471 OUT1 196

SDLC - Extract fields into fixed template 1 passed

100.0%

IN1 004 OUT22

✓

FieldExtractCodeGenRedaction test: model extracts fields into the template; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 004 OUT22

SDLC - Locate conflicting requirement pair 1 passed

100.0%

IN57 009 OUT8 652

✓

SdlcConstraintConflictCodeGenCode-gen: model locates the conflicting requirement pair; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN57 009 OUT8 652

SDLC - Locate redundant requirement 1 passed

100.0%

IN8 801 OUT5 753

✓

SdlcRedundantRequirementCodeGenCode-gen: model locates a redundant requirement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN8 801 OUT5 753

SDLC - Mask PII to placeholder 1 passed

100.0%

IN972 OUT42

✓

MaskPiiCodeGenRedaction test: model masks PII to a placeholder; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN972 OUT42

SDLC - Modules on a dependency cycle 1 passed

100.0%

IN21 243 OUT4 217

✓

DependencyCycleModulesCodeGenCode-gen: model implements dependency-cycle detection; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN21 243 OUT4 217

SDLC - Remove duplicate requirements 1 passed

100.0%

IN945 OUT10

✓

DeduplicateCodeGenRedaction test: model removes duplicate requirements; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN945 OUT10

SDLC - Renumber requirements sequentially 1 passed

100.0%

IN1 012 OUT42

✓

RenumberCodeGenRedaction test: model renumbers sequentially; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 012 OUT42

SDLC - Reorganize notes into section template 1 passed

100.0%

IN1 126 OUT94

✓

RewriteSectionsCodeGenRedaction test: model reorganizes the notes; deterministic checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 126 OUT94

SDLC - Required semver bump from changelog 1 passed

100.0%

IN17 620 OUT1 238

✓

SemverBumpCodeGenCode-gen: model implements the semver-bump rule; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Easysource:Ai4Users

Passed

IN17 620 OUT1 238

SDLC - Requirement-to-test traceability 1 passed

100.0%

IN8 781 OUT1 085

✓

SdlcTraceabilityCodeGenCode-gen: model implements traceability gap; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 781 OUT1 085

SDLC - Requirements completeness (spec coverage gap) 1 passed

100.0%

IN31 270 OUT4 671

✓

SdlcSpecCoverageCodeGenCode-gen: model implements spec-gap detection; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN31 270 OUT4 671

SDLC - Requirements consistency (contradiction detection) 1 passed

100.0%

IN35 645 OUT5 689

✓

ConsistencyCodeGenCode-gen: model implements requirements-consistency checking; graded vs oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN35 645 OUT5 689

SDLC - Reshape prose into JSON template 1 passed

100.0%

IN1 014 OUT28

✓

JsonShapeCodeGenRedaction test: model reshapes prose into JSON; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 014 OUT28

SDLC - Reshape rows into Markdown table 1 passed

100.0%

IN968 OUT31

✓

TableFormatCodeGenRedaction test: model builds a Markdown table; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN968 OUT31

SDLC - Rewrite in imperative requirement style 1 passed

100.0%

IN991 OUT50

✓

StyleRewriteCodeGenRedaction test: model rewrites notes in imperative style; checker + self-review loop.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN991 OUT50

SDLC - Roll up estimates over task tree 1 passed

100.0%

IN19 378 OUT1 989

✓

EstimateRollupCodeGenCode-gen: model implements estimate roll-up over a task tree; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN19 378 OUT1 989

SDLC - State-machine reachability 1 passed

100.0%

IN19 489 OUT2 287

✓

SdlcStateReachableCodeGenCode-gen: model implements state reachability; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN19 489 OUT2 287

SDLC - Ticket priority sort 1 passed

100.0%

IN18 972 OUT1 860

✓

SdlcPrioritySortCodeGenCode-gen: model implements priority sort; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN18 972 OUT1 860

SDLC - Unify glossary term consistency 1 passed

100.0%

IN972 OUT38

✓

GlossaryCodeGenRedaction test: model unifies synonyms to the canonical term; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN972 OUT38

Qwen 3.6 35B FP8 Instruct | Benchmark #11

SC# Coding & Agentic Benchmark by Ai4Users

Results by difficulty

Results by category