Ai4Users - Qwen 3.6 27B FP8 Instruct

Benchmark items
Agentic - Blue-Green Deploy (health-gated cutover) 1 passed	100.0%	IN19 101 OUT417
✓	AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query 6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN19 101 OUT417
Agentic - Bugfix (root-cause vs regression) 1 passed	100.0%	IN20 366 OUT218
✓	AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query 1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users	Passed	IN20 366 OUT218
Agentic - Cache Invalidation (exact set) 1 passed	100.0%	IN14 851 OUT606
✓	AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query 7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN14 851 OUT606
Agentic - Canary (metric-gated release) 1 passed	100.0%	IN19 144 OUT571
✓	AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query 6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN19 144 OUT571
Agentic - Canary variants (5 metric cases) 5 passed	100.0%	IN165 972 OUT5 181
✓	CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN23 826 OUT909
✓	CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN32 006 OUT986
✓	CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN27 278 OUT776
✓	CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query 14 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN54 828 OUT1 491
✓	CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN28 034 OUT1 019
Agentic - Code Review (find injected defect) 1 passed	100.0%	IN13 446 OUT759
✓	AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN13 446 OUT759
Agentic - Config Drift (remediate exact) 1 passed	100.0%	IN12 162 OUT434
✓	AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query 6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN12 162 OUT434
Agentic - Constraints (multi-section satisfaction) 1 passed	100.0%	IN11 486 OUT533
✓	AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN11 486 OUT533
Agentic - Constraints variants (5 cases) 5 passed	100.0%	IN50 723 OUT2 343
✓	ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN10 572 OUT626
✓	ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN9 922 OUT382
✓	ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN9 980 OUT452
✓	ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN10 277 OUT480
✓	ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN9 972 OUT403
Agentic - Data Reconciliation (per-field precedence) 1 passed	100.0%	IN14 087 OUT785
✓	AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN14 087 OUT785
Agentic - Diagnosis (root cause under budget) 1 passed	100.0%	IN12 628 OUT778
✓	AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN12 628 OUT778
Agentic - Diagnosis variants (5 cases) 5 passed	100.0%	IN78 006 OUT4 324
✓	DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN15 409 OUT930
✓	DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN15 433 OUT870
✓	DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN15 172 OUT633
✓	DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN15 593 OUT876
✓	DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN16 399 OUT1 015
Agentic - ETL Repair (root-cause in stage 2) 1 passed	100.0%	IN39 612 OUT1 066
✓	AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query 15 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN39 612 OUT1 066
Agentic - Feature Flag (staged rollout + kill-switch) 1 passed	100.0%	IN28 058 OUT832
✓	AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN28 058 OUT832
Agentic - Game Backend: 1D bouncing ball 1 passed	100.0%	IN8 913 OUT136
✓	GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 913 OUT136
Agentic - Game Backend: 1D swept anti-tunneling 1 passed	100.0%	IN8 924 OUT221
✓	GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 924 OUT221
Agentic - Game Backend: 2048 left-move (slide + merge) 1 passed	100.0%	IN8 933 OUT318
✓	GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 933 OUT318
Agentic - Game Backend: 8-direction from vector 1 passed	100.0%	IN18 137 OUT496
✓	GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 137 OUT496
Agentic - Game Backend: AABB collision resolution (MTV) 1 passed	100.0%	IN9 029 OUT306
✓	GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN9 029 OUT306
Agentic - Game Backend: AABB containment 1 passed	100.0%	IN8 869 OUT58
✓	GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 869 OUT58
Agentic - Game Backend: Acceleration to max speed 1 passed	100.0%	IN8 871 OUT80
✓	GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 871 OUT80
Agentic - Game Backend: Achievement thresholds 1 passed	100.0%	IN8 863 OUT132
✓	GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 863 OUT132
Agentic - Game Backend: Active status effects 1 passed	100.0%	IN8 885 OUT171
✓	GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 885 OUT171
Agentic - Game Backend: Animation frame at time 1 passed	100.0%	IN8 903 OUT94
✓	GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 903 OUT94
Agentic - Game Backend: Army upkeep income 1 passed	100.0%	IN8 851 OUT109
✓	GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 851 OUT109
Agentic - Game Backend: Axis-aligned path follow 1 passed	100.0%	IN8 894 OUT370
✓	GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN8 894 OUT370
Agentic - Game Backend: Battleship shot resolution 1 passed	100.0%	IN8 953 OUT292
✓	GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 953 OUT292
Agentic - Game Backend: Bounce restitution 1 passed	100.0%	IN8 890 OUT73
✓	GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 890 OUT73
Agentic - Game Backend: Bresenham line rasterization 1 passed	100.0%	IN27 833 OUT738
✓	GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN27 833 OUT738
Agentic - Game Backend: Buff/debuff stacking 1 passed	100.0%	IN8 916 OUT259
✓	GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 916 OUT259
Agentic - Game Backend: Build queue completions 1 passed	100.0%	IN8 884 OUT106
✓	GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 884 OUT106
Agentic - Game Backend: Camera-relative movement 1 passed	100.0%	IN8 971 OUT289
✓	GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 971 OUT289
Agentic - Game Backend: Card hand sort 1 passed	100.0%	IN8 892 OUT439
✓	GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 892 OUT439
Agentic - Game Backend: Card-game rules (blackjack/deck/compare) 1 passed	100.0%	IN9 067 OUT604
✓	GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN9 067 OUT604
Agentic - Game Backend: Chart scaling & histogram bins 1 passed	100.0%	IN8 956 OUT240
✓	GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 956 OUT240
Agentic - Game Backend: Checkers captures 1 passed	100.0%	IN37 735 OUT1 566
✓	GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN37 735 OUT1 566
Agentic - Game Backend: Collision primitives (circle/aabb/segment) 1 failed	0.0%	IN9 129 OUT624
×	GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == BoolJson(RefSegmentsIntersect(ax, ay, bx, by, cx, cy, dx, dy)) Context: SegmentsIntersect mismatch got true Values: r.ResultJson = "true"	Failed	IN9 129 OUT624
Agentic - Game Backend: Combat turn order (initiative) 1 passed	100.0%	IN8 865 OUT357
✓	GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 865 OUT357
Agentic - Game Backend: Combo scoring (rising multiplier) 1 passed	100.0%	IN8 888 OUT136
✓	GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 888 OUT136
Agentic - Game Backend: Control-point capture 1 passed	100.0%	IN8 901 OUT177
✓	GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 901 OUT177
Agentic - Game Backend: Controls & Inventory (spec-driven function battery) 1 passed	100.0%	IN9 042 OUT257
✓	GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN9 042 OUT257
Agentic - Game Backend: Cooldowns & resource regen 1 passed	100.0%	IN8 959 OUT176
✓	GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 959 OUT176
Agentic - Game Backend: Coyote-time jump 1 passed	100.0%	IN8 880 OUT70
✓	GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 880 OUT70
Agentic - Game Backend: Crafting capacity 1 passed	100.0%	IN8 904 OUT340
✓	GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 904 OUT340
Agentic - Game Backend: Cribbage fifteens 1 passed	100.0%	IN8 881 OUT218
✓	GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 881 OUT218
Agentic - Game Backend: Critical-hit damage 1 passed	100.0%	IN8 871 OUT57
✓	GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 871 OUT57
Agentic - Game Backend: Damage over time 1 passed	100.0%	IN8 897 OUT166
✓	GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 897 OUT166
Agentic - Game Backend: Day/night cycle phase 1 passed	100.0%	IN8 883 OUT91
✓	GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 883 OUT91
Agentic - Game Backend: Dice notation max roll 1 passed	100.0%	IN8 898 OUT259
✓	GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 898 OUT259
Agentic - Game Backend: Enemy AI state machine 1 passed	100.0%	IN8 984 OUT475
✓	GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN8 984 OUT475
Agentic - Game Backend: Equipment slot validation 1 passed	100.0%	IN8 898 OUT163
✓	GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 898 OUT163
Agentic - Game Backend: Flood-fill region size 1 passed	100.0%	IN18 236 OUT766
✓	GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 236 OUT766
Agentic - Game Backend: Fog of war visibility 1 passed	100.0%	IN8 926 OUT188
✓	GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 926 OUT188
Agentic - Game Backend: Fog reveal (Chebyshev radius) 1 passed	100.0%	IN8 899 OUT247
✓	GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 899 OUT247
Agentic - Game Backend: Friction stop frame 1 passed	100.0%	IN8 879 OUT49
✓	GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 879 OUT49
Agentic - Game Backend: Game of Life (code-gen vs oracle) 1 passed	100.0%	IN8 938 OUT320
✓	GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 938 OUT320
Agentic - Game Backend: Greedy coin change 1 failed	0.0%	IN17 929 OUT253
×	GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: MakeChange(87,[25 10 5 1]) got "3102" expected "3 1 0 2" Values: r.ResultJson = "\"3102\"" exp = "\"3 1 0 2\""	Failed	IN17 929 OUT253
Agentic - Game Backend: Grid inventory & stacking (Diablo-style) 1 passed	100.0%	IN9 013 OUT342
✓	GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN9 013 OUT342
Agentic - Game Backend: Grid pathfinding (shortest path, BFS) 1 failed	0.0%	IN30 476 OUT3 071
×	GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.3 AI queries 0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString() Context: ShortestPath mismatch grid 4x4 (0,0)->(3,3) got -1 expected 6 Values: r.ResultJson = "-1"	Failed	IN30 476 OUT3 071
Agentic - Game Backend: Grid raycast 1 passed	100.0%	IN8 936 OUT262
✓	GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 936 OUT262
Agentic - Game Backend: Health bar segments 1 passed	100.0%	IN8 937 OUT289
✓	GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 937 OUT289
Agentic - Game Backend: High-score table insert 1 failed	0.0%	IN37 607 OUT1 359
×	GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Insert did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 1:39: Unexpected character '#'. 1:75: Unexpected character '`'. 1:81: Unexpected character '`'. 1:91: Unexpected character '`'. 1:101: Unexpected character '`'. 1:118: Unexpected character '`'. 1:125: Unexpected character '`'. 1:130: Unexpected character '`'. 1:135: Unexpected character '`'. 1:182: Unexpected character '`'. 1:189: Unexpected character '`'. 1:210: Unexpected character '`'. 1:236: Unexpected character '`'. 1:248: Unexpected character '`'. 1:256: Unexpected character '`'. 1:398: Unexpected character '`'. 1:405: Unexpected character '`'. 3:80: Unexpected character '`'. 3:115: Unexpected character '`'. 3:138: Unexpected character '`'. 3:145: Unexpected character '`'. 5:34: Unexpected character '#'. 5:103: Unexpected character '`'. 5:120: Unexpected character '`'. 5:177: Unexpected character '#'. 5:184: Char literals must contain exactly one character and close with '''.	Failed	IN37 607 OUT1 359
Agentic - Game Backend: Hit chance 1 passed	100.0%	IN8 875 OUT59
✓	GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 875 OUT59
Agentic - Game Backend: Inventory encumbrance 1 passed	100.0%	IN8 872 OUT149
✓	GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 872 OUT149
Agentic - Game Backend: Inventory first-fit placement 1 passed	100.0%	IN8 938 OUT213
✓	GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 938 OUT213
Agentic - Game Backend: Inventory stack split 1 passed	100.0%	IN8 887 OUT46
✓	GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 887 OUT46
Agentic - Game Backend: Isometric projection 1 passed	100.0%	IN8 905 OUT96
✓	GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 905 OUT96
Agentic - Game Backend: Jump buffer 1 passed	100.0%	IN8 893 OUT137
✓	GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 893 OUT137
Agentic - Game Backend: Knockback with arena clamp 1 passed	100.0%	IN8 932 OUT212
✓	GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 932 OUT212
Agentic - Game Backend: Leaderboard rank 1 passed	100.0%	IN8 865 OUT103
✓	GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 865 OUT103
Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over) 1 passed	100.0%	IN9 016 OUT210
✓	GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN9 016 OUT210
Agentic - Game Backend: Lifesteal healing 1 passed	100.0%	IN8 876 OUT127
✓	GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 876 OUT127
Agentic - Game Backend: Lights Out toggle 1 passed	100.0%	IN8 892 OUT238
✓	GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 892 OUT238
Agentic - Game Backend: Match-3 cascade 1 failed	0.0%	IN18 950 OUT1 826
×	GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString() Context: Resolve 5x4 [33113133323222333331] got 10 expected 13 Values: r.ResultJson = "10"	Failed	IN18 950 OUT1 826
Agentic - Game Backend: Match-3 detection 1 passed	100.0%	IN8 914 OUT511
✓	GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 914 OUT511
Agentic - Game Backend: Match-3 swap validity 1 passed	100.0%	IN8 923 OUT493
✓	GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN8 923 OUT493
Agentic - Game Backend: Memory match pairs 1 passed	100.0%	IN8 877 OUT90
✓	GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 877 OUT90
Agentic - Game Backend: Mercy invincibility (i-frames) 1 passed	100.0%	IN8 932 OUT210
✓	GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 932 OUT210
Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal) 1 passed	100.0%	IN18 697 OUT1 197
✓	GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN18 697 OUT1 197
Agentic - Game Backend: Minimap downsample 1 passed	100.0%	IN8 908 OUT221
✓	GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 908 OUT221
Agentic - Game Backend: Minimap marker position 1 passed	100.0%	IN8 911 OUT140
✓	GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 911 OUT140
Agentic - Game Backend: N-in-a-row win detection (Connect-4) 1 failed	0.0%	IN39 992 OUT2 479
×	GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.4 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Winner did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 1:21: Unexpected character '#'. 1:28: Char literals must contain exactly one character and close with '''.	Failed	IN39 992 OUT2 479
Agentic - Game Backend: Nonogram row validation 1 passed	100.0%	IN8 888 OUT289
✓	GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 888 OUT289
Agentic - Game Backend: OBB collision (rotating rectangles, SAT) 1 passed	100.0%	IN19 394 OUT2 328
✓	GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN19 394 OUT2 328
Agentic - Game Backend: Particle lifetime & expiry 1 passed	100.0%	IN8 970 OUT190
✓	GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 970 OUT190
Agentic - Game Backend: Party XP share 1 passed	100.0%	IN8 888 OUT111
✓	GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 888 OUT111
Agentic - Game Backend: Patrol waypoints (ping-pong) 1 passed	100.0%	IN8 911 OUT128
✓	GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 911 OUT128
Agentic - Game Backend: Platformer jump arc 1 passed	100.0%	IN8 902 OUT102
✓	GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 902 OUT102
Agentic - Game Backend: Point on circle (trig) 1 passed	100.0%	IN8 935 OUT124
✓	GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 935 OUT124
Agentic - Game Backend: Poker hand comparison 1 passed	100.0%	IN46 847 OUT6 971
✓	GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users	Passed	IN46 847 OUT6 971
Agentic - Game Backend: Poker hand ranking 1 passed	100.0%	IN18 624 OUT1 371
✓	GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN18 624 OUT1 371
Agentic - Game Backend: Pop-cap training 1 passed	100.0%	IN8 889 OUT95
✓	GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 889 OUT95
Agentic - Game Backend: Progress bar fill 1 passed	100.0%	IN8 871 OUT58
✓	GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 871 OUT58
Agentic - Game Backend: Projectile range (physics/trig) 1 passed	100.0%	IN8 893 OUT92
✓	GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 893 OUT92
Agentic - Game Backend: Projectile with ground bounce 1 passed	100.0%	IN8 927 OUT131
✓	GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 927 OUT131
Agentic - Game Backend: Resource gathering 1 passed	100.0%	IN8 888 OUT40
✓	GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 888 OUT40
Agentic - Game Backend: Reversi flip count 1 passed	100.0%	IN18 459 OUT818
✓	GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN18 459 OUT818
Agentic - Game Backend: Rotate 2D vector (trig) 1 passed	100.0%	IN8 904 OUT115
✓	GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 904 OUT115
Agentic - Game Backend: Round-robin deal 1 passed	100.0%	IN8 893 OUT85
✓	GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 893 OUT85
Agentic - Game Backend: RTS resource economy 1 passed	100.0%	IN8 884 OUT100
✓	GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 884 OUT100
Agentic - Game Backend: RTS unit counters & damage 1 passed	100.0%	IN8 936 OUT171
✓	GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 936 OUT171
Agentic - Game Backend: Safest neighbour (danger map) 1 passed	100.0%	IN18 117 OUT807
✓	GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 117 OUT807
Agentic - Game Backend: Scroll viewport 1 passed	100.0%	IN8 906 OUT73
✓	GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 906 OUT73
Agentic - Game Backend: Set card validity 1 passed	100.0%	IN8 885 OUT164
✓	GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 885 OUT164
Agentic - Game Backend: Shop affordability 1 passed	100.0%	IN8 865 OUT62
✓	GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 865 OUT62
Agentic - Game Backend: Sliding puzzle move 1 passed	100.0%	IN8 896 OUT286
✓	GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 896 OUT286
Agentic - Game Backend: Snake head movement 1 passed	100.0%	IN8 953 OUT277
✓	GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 953 OUT277
Agentic - Game Backend: Snake tick 1 failed	0.0%	IN18 450 OUT705
×	GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: Tick [1,0;0,0] R food(9,9) got "2,0;1,0;0,0" expected "2,0;1,0" Values: r.ResultJson = "\"2,0;1,0;0,0\"" exp = "\"2,0;1,0\""	Failed	IN18 450 OUT705
Agentic - Game Backend: Snap to grid 1 passed	100.0%	IN8 881 OUT63
✓	GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 881 OUT63
Agentic - Game Backend: Sokoban move/push legality 1 passed	100.0%	IN8 984 OUT244
✓	GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 984 OUT244
Agentic - Game Backend: Solitaire tableau move 1 passed	100.0%	IN8 923 OUT210
✓	GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 923 OUT210
Agentic - Game Backend: Spawn scheduler count 1 passed	100.0%	IN8 880 OUT82
✓	GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 880 OUT82
Agentic - Game Backend: Spawn-wave alive count 1 passed	100.0%	IN8 892 OUT139
✓	GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 892 OUT139
Agentic - Game Backend: Stat level scaling 1 passed	100.0%	IN8 884 OUT65
✓	GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 884 OUT65
Agentic - Game Backend: Steering seek (greedy pursuit) 1 passed	100.0%	IN8 910 OUT135
✓	GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 910 OUT135
Agentic - Game Backend: Tactical movement range 1 passed	100.0%	IN38 848 OUT1 989
✓	GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN38 848 OUT1 989
Agentic - Game Backend: Tech tree eligibility 1 passed	100.0%	IN8 904 OUT210
✓	GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 904 OUT210
Agentic - Game Backend: Tetris line clearing 1 passed	100.0%	IN8 895 OUT289
✓	GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 895 OUT289
Agentic - Game Backend: Threat table top target 1 passed	100.0%	IN8 889 OUT302
✓	GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 889 OUT302
Agentic - Game Backend: Tic-tac-toe result 1 passed	100.0%	IN8 880 OUT291
✓	GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 880 OUT291
Agentic - Game Backend: Toroidal wrap (Pac-Man) 1 passed	100.0%	IN8 916 OUT60
✓	GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 916 OUT60
Agentic - Game Backend: Tower targeting (nearest in range) 1 passed	100.0%	IN8 904 OUT203
✓	GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 904 OUT203
Agentic - Game Backend: Transaction ledger 1 passed	100.0%	IN8 907 OUT213
✓	GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 907 OUT213
Agentic - Game Backend: Trick-taking winner 1 passed	100.0%	IN8 935 OUT503
✓	GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN8 935 OUT503
Agentic - Game Backend: Turn-based duel resolution 1 passed	100.0%	IN8 967 OUT287
✓	GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 967 OUT287
Agentic - Game Backend: Typed armor mitigation 1 passed	100.0%	IN8 891 OUT364
✓	GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 891 OUT364
Agentic - Game Backend: UI menu (wrap/clamp/paginate) 1 passed	100.0%	IN8 983 OUT266
✓	GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 983 OUT266
Agentic - Game Backend: Uno playable cards 1 passed	100.0%	IN8 919 OUT195
✓	GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 919 OUT195
Agentic - Game Backend: Velocity reflection 1 passed	100.0%	IN8 863 OUT114
✓	GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 863 OUT114
Agentic - Game Backend: Wall-slide fall 1 passed	100.0%	IN8 871 OUT73
✓	GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 871 OUT73
Agentic - Game Backend: War card compare 1 passed	100.0%	IN8 891 OUT232
✓	GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 891 OUT232
Agentic - Game Backend: Weighted loot table 1 passed	100.0%	IN8 894 OUT99
✓	GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 894 OUT99
Agentic - Game Backend: Weighted terrain pathfinding 1 passed	100.0%	IN40 408 OUT3 257
✓	GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN40 408 OUT3 257
Agentic - Game Backend: Word-wrap line count 1 passed	100.0%	IN8 876 OUT168
✓	GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 876 OUT168
Agentic - Game Backend: XP to character level 1 passed	100.0%	IN8 884 OUT88
✓	GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 884 OUT88
Agentic - Idempotency (poison-retry / no double charge) 1 passed	100.0%	IN14 365 OUT403
✓	AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN14 365 OUT403
Agentic - Irreversible Commit (measure twice) 1 passed	100.0%	IN15 966 OUT413
✓	AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query 7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN15 966 OUT413
Agentic - Least Privilege (exact grants) 1 passed	100.0%	IN11 652 OUT304
✓	AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query 5 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN11 652 OUT304
Agentic - Ledger (simulated DB read-modify-write) 1 failed	0.0%	IN39 877 OUT1 648
×	AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query 16 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: actual == expected Context: Account alice ended at 600 but expected 0 (in-order apply with overdraft-skip). Values: actual = 600 expected = 0	Failed	IN39 877 OUT1 648
Agentic - Ledger variants (5 RMW cases) 5 passed	100.0%	IN162 972 OUT6 996
✓	LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN16 080 OUT919
✓	LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query 17 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN48 961 OUT1 933
✓	LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN17 242 OUT1 403
✓	LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query 18 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN60 238 OUT1 700
✓	LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN20 451 OUT1 041
Agentic - Lock Ordering (deadlock avoidance) 1 passed	100.0%	IN67 223 OUT1 762
✓	AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query 25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN67 223 OUT1 762
Agentic - Lock Ordering variants (5 cases) 5 passed	100.0%	IN298 746 OUT8 258
✓	LockVariantFiveResFourOps1 AI query 23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN67 092 OUT1 728
✓	LockVariantFourResThreeOps1 AI query 18 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN49 075 OUT1 312
✓	LockVariantHeavyReuse1 AI query 23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN67 093 OUT1 722
✓	LockVariantReverseListed1 AI query 25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN64 423 OUT1 939
✓	LockVariantThreeLockOp1 AI query 20 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN51 063 OUT1 557
Agentic - Migration (cascading rollback + reapply) 1 passed	100.0%	IN21 240 OUT853
✓	AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query 11 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN21 240 OUT853
Agentic - Requirements (detect contradiction) 1 passed	100.0%	IN7 457 OUT376
✓	AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query 2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN7 457 OUT376
Agentic - Retry/Circuit-breaker (transient vs hard-down) 1 passed	100.0%	IN26 529 OUT800
✓	AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query 10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN26 529 OUT800
Agentic - Saga (compensation / distributed rollback) 1 passed	100.0%	IN21 324 OUT304
✓	AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN21 324 OUT304
Agentic - Saga variants (5 compensation cases) 5 passed	100.0%	IN102 077 OUT2 101
✓	SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN13 553 OUT228
✓	SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN13 749 OUT318
✓	SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query 7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN23 658 OUT530
✓	SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN20 060 OUT377
✓	SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query 9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN31 057 OUT648
Agentic - Schema Compat (expand-contract) 1 passed	100.0%	IN21 563 OUT511
✓	AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN21 563 OUT511
Agentic - Scratchpad (self-repair, Dyck, tool-decision) 7 passed	100.0%	IN227 086 OUT8 839
✓	AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query 1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN22 187 OUT909
✓	AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query 4 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN55 297 OUT779
✓	ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query 0 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN10 658 OUT27
✓	ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN21 664 OUT470
✓	ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN21 480 OUT269
✓	ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query 4 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users	Passed	IN62 437 OUT5 581
✓	ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query 2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN33 363 OUT804
Agentic - Secret Rotation (zero-downtime ordering) 1 passed	100.0%	IN16 479 OUT259
✓	AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query 5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN16 479 OUT259
Agentic - StatefulOps (incident recovery, concurrency replan) 2 passed	100.0%	IN54 395 OUT1 442
✓	AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query 7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN22 504 OUT662
✓	AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query 8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN31 891 OUT780
Agentic - Test Authoring (write a test that catches the bug) 1 passed	100.0%	IN16 854 OUT471
✓	AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query 5 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN16 854 OUT471
Agentic - Topological Planning (deps + prereq) 1 passed	100.0%	IN25 457 OUT760
✓	AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN25 457 OUT760
Algorithms - 0/1 Knapsack (dynamic programming) 1 passed	100.0%	IN18 133 OUT650
✓	KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN18 133 OUT650
Algorithms - 1D elastic collision 1 passed	100.0%	IN8 938 OUT179
✓	ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 938 OUT179
Algorithms - All-pairs shortest path pair 1 passed	100.0%	IN8 906 OUT480
✓	FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 906 OUT480
Algorithms - Balanced brackets (stack) 1 passed	100.0%	IN8 869 OUT217
✓	ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 869 OUT217
Algorithms - Bellman-Ford shortest path (neg weights/cycles) 1 passed	100.0%	IN8 915 OUT1 094
✓	BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 915 OUT1 094
Algorithms - Binary Gray code 1 passed	100.0%	IN8 842 OUT22
✓	GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 842 OUT22
Algorithms - Bipartite graph check 1 failed	0.0%	IN8 888 OUT386
×	BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: IsBipartite(4,[0,1;1,2;2,3;3,0]) got false expected true Values: r.ResultJson = "false" exp = "true"	Failed	IN8 888 OUT386
Algorithms - Bit tricks (popcount/pow2/lowbit) 1 passed	100.0%	IN8 916 OUT124
✓	BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 916 OUT124
Algorithms - Caesar cipher 1 passed	100.0%	IN8 865 OUT202
✓	CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 865 OUT202
Algorithms - Catalan number 1 passed	100.0%	IN8 887 OUT84
✓	CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 887 OUT84
Algorithms - Chinese remainder theorem 1 passed	100.0%	IN8 905 OUT757
✓	CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 905 OUT757
Algorithms - Circle-circle MTV 1 passed	100.0%	IN8 982 OUT157
✓	CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 982 OUT157
Algorithms - Coin change combinations 1 passed	100.0%	IN18 074 OUT631
✓	CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 074 OUT631
Algorithms - Collatz step count 1 passed	100.0%	IN8 868 OUT92
✓	CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 868 OUT92
Algorithms - Connected components (union-find) 1 failed	0.0%	IN37 711 OUT1 338
×	UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users CountComponents did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 3:8: Expected a variable name, found '['.	Failed	IN37 711 OUT1 338
Algorithms - Container with most water 1 passed	100.0%	IN8 876 OUT225
✓	ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 876 OUT225
Algorithms - Count anagram groups 1 passed	100.0%	IN8 859 OUT144
✓	AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 859 OUT144
Algorithms - Count grid color regions 1 passed	100.0%	IN8 879 OUT444
✓	CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 879 OUT444
Algorithms - Count islands 1 passed	100.0%	IN8 882 OUT501
✓	CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 882 OUT501
Algorithms - Count overlapping occurrences 1 passed	100.0%	IN8 865 OUT102
✓	RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 865 OUT102
Algorithms - Count primes (Sieve of Eratosthenes) 1 passed	100.0%	IN8 844 OUT134
✓	CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 844 OUT134
Algorithms - Cumulative distribution 1 passed	100.0%	IN8 866 OUT246
✓	CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 866 OUT246
Algorithms - Dijkstra shortest path (weighted graph) 1 passed	100.0%	IN8 918 OUT526
✓	DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 918 OUT526
Algorithms - Directed cycle detection 1 failed	0.0%	IN38 122 OUT1 576
×	DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users HasCycle did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 3:40: Expected '(' after the List type. Lists are created with 'new List<T>()' or 'new List<T>(items)', found '['.	Failed	IN38 122 OUT1 576
Algorithms - Distinct subsequences 1 passed	100.0%	IN8 861 OUT314
✓	DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users	Passed	IN8 861 OUT314
Algorithms - Divisor count 1 passed	100.0%	IN8 849 OUT68
✓	DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 849 OUT68
Algorithms - Downsample by block average 1 passed	100.0%	IN8 890 OUT209
✓	ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 890 OUT209
Algorithms - Dutch national flag sort 1 passed	100.0%	IN8 857 OUT272
✓	DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 857 OUT272
Algorithms - Egg drop trials 1 failed	0.0%	IN8 918 OUT238
×	EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString() Context: MinTrials(2,10) got 10 expected 4 Values: r.ResultJson = "10"	Failed	IN8 918 OUT238
Algorithms - Equal subset-sum partition 1 passed	100.0%	IN8 860 OUT257
✓	PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 860 OUT257
Algorithms - Euler totient 1 passed	100.0%	IN8 849 OUT205
✓	EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 849 OUT205
Algorithms - Eulerian path existence 1 passed	100.0%	IN18 341 OUT947
✓	EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN18 341 OUT947
Algorithms - Extended Euclidean 1 passed	100.0%	IN8 961 OUT189
✓	ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 961 OUT189
Algorithms - Format duration H:MM:SS 1 passed	100.0%	IN8 876 OUT184
✓	FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 876 OUT184
Algorithms - GCD and LCM 1 passed	100.0%	IN17 876 OUT229
✓	GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN17 876 OUT229
Algorithms - Grid transpose 1 passed	100.0%	IN8 884 OUT86
✓	TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 884 OUT86
Algorithms - House robber (max non-adjacent sum) 1 passed	100.0%	IN8 848 OUT217
✓	HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 848 OUT217
Algorithms - Integer linear interpolation 1 passed	100.0%	IN8 882 OUT35
✓	LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 882 OUT35
Algorithms - Integer square root 1 passed	100.0%	IN8 864 OUT153
✓	IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 864 OUT153
Algorithms - Integer to base-N string 1 passed	100.0%	IN8 870 OUT107
✓	BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 870 OUT107
Algorithms - Integer to Roman numeral 1 passed	100.0%	IN8 899 OUT515
✓	RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 899 OUT515
Algorithms - Josephus survivor 1 passed	100.0%	IN8 875 OUT237
✓	JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 875 OUT237
Algorithms - Jump game (min jumps) 1 passed	100.0%	IN8 883 OUT238
✓	JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 883 OUT238
Algorithms - Kth largest element 1 passed	100.0%	IN8 873 OUT94
✓	KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 873 OUT94
Algorithms - Least-squares line fit 1 passed	100.0%	IN8 915 OUT263
✓	LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 915 OUT263
Algorithms - Levenshtein edit distance (DP) 1 passed	100.0%	IN8 843 OUT307
✓	EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 843 OUT307
Algorithms - Log-scale bucket index 1 passed	100.0%	IN8 908 OUT116
✓	LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 908 OUT116
Algorithms - Logarithmic bin index 1 passed	100.0%	IN8 897 OUT86
✓	LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 897 OUT86
Algorithms - Longest common prefix 1 passed	100.0%	IN8 856 OUT175
✓	CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 856 OUT175
Algorithms - Longest common subsequence 1 passed	100.0%	IN8 841 OUT246
✓	LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 841 OUT246
Algorithms - Longest common substring 1 passed	100.0%	IN8 858 OUT179
✓	CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 858 OUT179
Algorithms - Longest increasing subsequence (DP) 1 passed	100.0%	IN8 875 OUT302
✓	LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 875 OUT302
Algorithms - Longest palindromic substring 1 passed	100.0%	IN8 853 OUT258
✓	LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 853 OUT258
Algorithms - Longest repeated substring 1 failed	0.0%	IN8 858 OUT180
×	LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString() Context: Length([aaaa]) got 2 expected 3 Values: r.ResultJson = "2"	Failed	IN8 858 OUT180
Algorithms - Lower bound (binary search) 1 passed	100.0%	IN8 886 OUT150
✓	LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 886 OUT150
Algorithms - Luhn checksum validation 1 passed	100.0%	IN8 887 OUT188
✓	LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 887 OUT188
Algorithms - Majority element 1 passed	100.0%	IN8 851 OUT254
✓	MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 851 OUT254
Algorithms - Matrix chain multiplication 1 passed	100.0%	IN8 886 OUT364
✓	MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 886 OUT364
Algorithms - Maximum subarray sum (Kadane) 1 passed	100.0%	IN8 876 OUT131
✓	MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 876 OUT131
Algorithms - Merge K sorted lists 1 passed	100.0%	IN8 869 OUT225
✓	MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 869 OUT225
Algorithms - Merge overlapping intervals 1 passed	100.0%	IN18 284 OUT905
✓	MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 284 OUT905
Algorithms - Min-max normalization 1 passed	100.0%	IN8 878 OUT281
✓	MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 878 OUT281
Algorithms - Minimum coin change (DP) 1 passed	100.0%	IN8 869 OUT288
✓	CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 869 OUT288
Algorithms - Minimum knight moves (BFS) 1 passed	100.0%	IN28 126 OUT1 274
✓	KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN28 126 OUT1 274
Algorithms - Minimum path sum (grid DP) 1 passed	100.0%	IN8 882 OUT350
✓	MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 882 OUT350
Algorithms - Minimum spanning tree (Kruskal) 1 passed	100.0%	IN18 406 OUT1 229
✓	KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN18 406 OUT1 229
Algorithms - Minimum window substring 1 passed	100.0%	IN8 890 OUT462
✓	MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users	Passed	IN8 890 OUT462
Algorithms - Modular exponentiation (fast pow) 1 passed	100.0%	IN8 901 OUT142
✓	ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 901 OUT142
Algorithms - Modular inverse (extended Euclid) 1 passed	100.0%	IN27 770 OUT976
✓	ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN27 770 OUT976
Algorithms - Moving average 1 passed	100.0%	IN8 880 OUT230
✓	MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 880 OUT230
Algorithms - Next greater element 1 passed	100.0%	IN8 863 OUT323
✓	NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 863 OUT323
Algorithms - Next lexicographic permutation 1 passed	100.0%	IN8 871 OUT324
✓	NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 871 OUT324
Algorithms - Nth Fibonacci mod m 1 passed	100.0%	IN8 870 OUT119
✓	FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 870 OUT119
Algorithms - Overlapping substring count 1 passed	100.0%	IN8 855 OUT96
✓	SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 855 OUT96
Algorithms - Palindrome & anagram checks 1 passed	100.0%	IN8 885 OUT642
✓	PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 885 OUT642
Algorithms - Pascal triangle row 1 passed	100.0%	IN8 863 OUT148
✓	PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 863 OUT148
Algorithms - Point-to-segment distance 1 passed	100.0%	IN27 791 OUT953
✓	PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN27 791 OUT953
Algorithms - Prime factorization 1 passed	100.0%	IN8 874 OUT131
✓	PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 874 OUT131
Algorithms - Product of array except self 1 passed	100.0%	IN18 117 OUT652
✓	ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 117 OUT652
Algorithms - Quadratic Bezier point 1 passed	100.0%	IN8 940 OUT149
✓	BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 940 OUT149
Algorithms - Range sum via prefix sums 1 passed	100.0%	IN8 862 OUT69
✓	PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 862 OUT69
Algorithms - Reverse words & Hamming distance 1 passed	100.0%	IN8 866 OUT110

Agentic - Blue-Green Deploy (health-gated cutover) 1 passed

100.0%

IN19 101 OUT417

✓

AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query 6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN19 101 OUT417

Agentic - Bugfix (root-cause vs regression) 1 passed

100.0%

IN20 366 OUT218

✓

AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query 1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN20 366 OUT218

Agentic - Cache Invalidation (exact set) 1 passed

100.0%

IN14 851 OUT606

✓

AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query 7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN14 851 OUT606

Agentic - Canary (metric-gated release) 1 passed

100.0%

IN19 144 OUT571

✓

AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query 6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN19 144 OUT571

Agentic - Canary variants (5 metric cases) 5 passed

100.0%

IN165 972 OUT5 181

✓

CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN23 826 OUT909

✓

CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN32 006 OUT986

✓

CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN27 278 OUT776

✓

CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query 14 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN54 828 OUT1 491

✓

CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN28 034 OUT1 019

Agentic - Code Review (find injected defect) 1 passed

100.0%

IN13 446 OUT759

✓

AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN13 446 OUT759

Agentic - Config Drift (remediate exact) 1 passed

100.0%

IN12 162 OUT434

✓

AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query 6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN12 162 OUT434

Agentic - Constraints (multi-section satisfaction) 1 passed

100.0%

IN11 486 OUT533

✓

AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN11 486 OUT533

Agentic - Constraints variants (5 cases) 5 passed

100.0%

IN50 723 OUT2 343

✓

ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN10 572 OUT626

✓

ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN9 922 OUT382

✓

ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN9 980 OUT452

✓

ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN10 277 OUT480

✓

ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN9 972 OUT403

Agentic - Data Reconciliation (per-field precedence) 1 passed

100.0%

IN14 087 OUT785

✓

AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN14 087 OUT785

Agentic - Diagnosis (root cause under budget) 1 passed

100.0%

IN12 628 OUT778

✓

AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN12 628 OUT778

Agentic - Diagnosis variants (5 cases) 5 passed

100.0%

IN78 006 OUT4 324

✓

DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN15 409 OUT930

✓

DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN15 433 OUT870

✓

DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN15 172 OUT633

✓

DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN15 593 OUT876

✓

DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN16 399 OUT1 015

Agentic - ETL Repair (root-cause in stage 2) 1 passed

100.0%

IN39 612 OUT1 066

✓

AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query 15 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN39 612 OUT1 066

Agentic - Feature Flag (staged rollout + kill-switch) 1 passed

100.0%

IN28 058 OUT832

✓

AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN28 058 OUT832

Agentic - Game Backend: 1D bouncing ball 1 passed

100.0%

IN8 913 OUT136

✓

GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 913 OUT136

Agentic - Game Backend: 1D swept anti-tunneling 1 passed

100.0%

IN8 924 OUT221

✓

GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 924 OUT221

Agentic - Game Backend: 2048 left-move (slide + merge) 1 passed

100.0%

IN8 933 OUT318

✓

GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 933 OUT318

Agentic - Game Backend: 8-direction from vector 1 passed

100.0%

IN18 137 OUT496

✓

GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 137 OUT496

Agentic - Game Backend: AABB collision resolution (MTV) 1 passed

100.0%

IN9 029 OUT306

✓

GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN9 029 OUT306

Agentic - Game Backend: AABB containment 1 passed

100.0%

IN8 869 OUT58

✓

GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 869 OUT58

Agentic - Game Backend: Acceleration to max speed 1 passed

100.0%

IN8 871 OUT80

✓

GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 871 OUT80

Agentic - Game Backend: Achievement thresholds 1 passed

100.0%

IN8 863 OUT132

✓

GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 863 OUT132

Agentic - Game Backend: Active status effects 1 passed

100.0%

IN8 885 OUT171

✓

GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 885 OUT171

Agentic - Game Backend: Animation frame at time 1 passed

100.0%

IN8 903 OUT94

✓

GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 903 OUT94

Agentic - Game Backend: Army upkeep income 1 passed

100.0%

IN8 851 OUT109

✓

GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 851 OUT109

Agentic - Game Backend: Axis-aligned path follow 1 passed

100.0%

IN8 894 OUT370

✓

GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN8 894 OUT370

Agentic - Game Backend: Battleship shot resolution 1 passed

100.0%

IN8 953 OUT292

✓

GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 953 OUT292

Agentic - Game Backend: Bounce restitution 1 passed

100.0%

IN8 890 OUT73

✓

GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 890 OUT73

Agentic - Game Backend: Bresenham line rasterization 1 passed

100.0%

IN27 833 OUT738

✓

GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN27 833 OUT738

Agentic - Game Backend: Buff/debuff stacking 1 passed

100.0%

IN8 916 OUT259

✓

GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 916 OUT259

Agentic - Game Backend: Build queue completions 1 passed

100.0%

IN8 884 OUT106

✓

GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 884 OUT106

Agentic - Game Backend: Camera-relative movement 1 passed

100.0%

IN8 971 OUT289

✓

GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 971 OUT289

Agentic - Game Backend: Card hand sort 1 passed

100.0%

IN8 892 OUT439

✓

GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 892 OUT439

Agentic - Game Backend: Card-game rules (blackjack/deck/compare) 1 passed

100.0%

IN9 067 OUT604

✓

GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN9 067 OUT604

Agentic - Game Backend: Chart scaling & histogram bins 1 passed

100.0%

IN8 956 OUT240

✓

GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 956 OUT240

Agentic - Game Backend: Checkers captures 1 passed

100.0%

IN37 735 OUT1 566

✓

GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN37 735 OUT1 566

Agentic - Game Backend: Collision primitives (circle/aabb/segment) 1 failed

0.0%

IN9 129 OUT624

×

GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == BoolJson(RefSegmentsIntersect(ax, ay, bx, by, cx, cy, dx, dy)) Context: SegmentsIntersect mismatch got true Values: r.ResultJson = "true"

Failed

IN9 129 OUT624

Agentic - Game Backend: Combat turn order (initiative) 1 passed

100.0%

IN8 865 OUT357

✓

GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 865 OUT357

Agentic - Game Backend: Combo scoring (rising multiplier) 1 passed

100.0%

IN8 888 OUT136

✓

GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 888 OUT136

Agentic - Game Backend: Control-point capture 1 passed

100.0%

IN8 901 OUT177

✓

GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 901 OUT177

Agentic - Game Backend: Controls & Inventory (spec-driven function battery) 1 passed

100.0%

IN9 042 OUT257

✓

GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN9 042 OUT257

Agentic - Game Backend: Cooldowns & resource regen 1 passed

100.0%

IN8 959 OUT176

✓

GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 959 OUT176

Agentic - Game Backend: Coyote-time jump 1 passed

100.0%

IN8 880 OUT70

✓

GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 880 OUT70

Agentic - Game Backend: Crafting capacity 1 passed

100.0%

IN8 904 OUT340

✓

GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 904 OUT340

Agentic - Game Backend: Cribbage fifteens 1 passed

100.0%

IN8 881 OUT218

✓

GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 881 OUT218

Agentic - Game Backend: Critical-hit damage 1 passed

100.0%

IN8 871 OUT57

✓

GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 871 OUT57

Agentic - Game Backend: Damage over time 1 passed

100.0%

IN8 897 OUT166

✓

GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 897 OUT166

Agentic - Game Backend: Day/night cycle phase 1 passed

100.0%

IN8 883 OUT91

✓

GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 883 OUT91

Agentic - Game Backend: Dice notation max roll 1 passed

100.0%

IN8 898 OUT259

✓

GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 898 OUT259

Agentic - Game Backend: Enemy AI state machine 1 passed

100.0%

IN8 984 OUT475

✓

GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN8 984 OUT475

Agentic - Game Backend: Equipment slot validation 1 passed

100.0%

IN8 898 OUT163

✓

GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 898 OUT163

Agentic - Game Backend: Flood-fill region size 1 passed

100.0%

IN18 236 OUT766

✓

GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 236 OUT766

Agentic - Game Backend: Fog of war visibility 1 passed

100.0%

IN8 926 OUT188

✓

GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 926 OUT188

Agentic - Game Backend: Fog reveal (Chebyshev radius) 1 passed

100.0%

IN8 899 OUT247

✓

GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 899 OUT247

Agentic - Game Backend: Friction stop frame 1 passed

100.0%

IN8 879 OUT49

✓

GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 879 OUT49

Agentic - Game Backend: Game of Life (code-gen vs oracle) 1 passed

100.0%

IN8 938 OUT320

✓

GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 938 OUT320

Agentic - Game Backend: Greedy coin change 1 failed

0.0%

IN17 929 OUT253

×

GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: MakeChange(87,[25 10 5 1]) got "3102" expected "3 1 0 2" Values: r.ResultJson = "\"3102\"" exp = "\"3 1 0 2\""

Failed

IN17 929 OUT253

Agentic - Game Backend: Grid inventory & stacking (Diablo-style) 1 passed

100.0%

IN9 013 OUT342

✓

GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN9 013 OUT342

Agentic - Game Backend: Grid pathfinding (shortest path, BFS) 1 failed

0.0%

IN30 476 OUT3 071

×

GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.3 AI queries 0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp.ToString() Context: ShortestPath mismatch grid 4x4 (0,0)->(3,3) got -1 expected 6 Values: r.ResultJson = "-1"

Failed

IN30 476 OUT3 071

Agentic - Game Backend: Grid raycast 1 passed

100.0%

IN8 936 OUT262

✓

GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 936 OUT262

Agentic - Game Backend: Health bar segments 1 passed

100.0%

IN8 937 OUT289

✓

GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 937 OUT289

Agentic - Game Backend: High-score table insert 1 failed

0.0%

IN37 607 OUT1 359

×

GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Insert did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 1:39: Unexpected character '#'. 1:75: Unexpected character '`'. 1:81: Unexpected character '`'. 1:91: Unexpected character '`'. 1:101: Unexpected character '`'. 1:118: Unexpected character '`'. 1:125: Unexpected character '`'. 1:130: Unexpected character '`'. 1:135: Unexpected character '`'. 1:182: Unexpected character '`'. 1:189: Unexpected character '`'. 1:210: Unexpected character '`'. 1:236: Unexpected character '`'. 1:248: Unexpected character '`'. 1:256: Unexpected character '`'. 1:398: Unexpected character '`'. 1:405: Unexpected character '`'. 3:80: Unexpected character '`'. 3:115: Unexpected character '`'. 3:138: Unexpected character '`'. 3:145: Unexpected character '`'. 5:34: Unexpected character '#'. 5:103: Unexpected character '`'. 5:120: Unexpected character '`'. 5:177: Unexpected character '#'. 5:184: Char literals must contain exactly one character and close with '''.

Failed

IN37 607 OUT1 359

Agentic - Game Backend: Hit chance 1 passed

100.0%

IN8 875 OUT59

✓

GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 875 OUT59

Agentic - Game Backend: Inventory encumbrance 1 passed

100.0%

IN8 872 OUT149

✓

GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 872 OUT149

Agentic - Game Backend: Inventory first-fit placement 1 passed

100.0%

IN8 938 OUT213

✓

GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 938 OUT213

Agentic - Game Backend: Inventory stack split 1 passed

100.0%

IN8 887 OUT46

✓

GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 887 OUT46

Agentic - Game Backend: Isometric projection 1 passed

100.0%

IN8 905 OUT96

✓

GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 905 OUT96

Agentic - Game Backend: Jump buffer 1 passed

100.0%

IN8 893 OUT137

✓

GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 893 OUT137

Agentic - Game Backend: Knockback with arena clamp 1 passed

100.0%

IN8 932 OUT212

✓

GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 932 OUT212

Agentic - Game Backend: Leaderboard rank 1 passed

100.0%

IN8 865 OUT103

✓

GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 865 OUT103

Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over) 1 passed

100.0%

IN9 016 OUT210

✓

GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN9 016 OUT210

Agentic - Game Backend: Lifesteal healing 1 passed

100.0%

IN8 876 OUT127

✓

GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 876 OUT127

Agentic - Game Backend: Lights Out toggle 1 passed

100.0%

IN8 892 OUT238

✓

GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 892 OUT238

Agentic - Game Backend: Match-3 cascade 1 failed

0.0%

IN18 950 OUT1 826

×

GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp.ToString() Context: Resolve 5x4 [33113133323222333331] got 10 expected 13 Values: r.ResultJson = "10"

Failed

IN18 950 OUT1 826

Agentic - Game Backend: Match-3 detection 1 passed

100.0%

IN8 914 OUT511

✓

GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 914 OUT511

Agentic - Game Backend: Match-3 swap validity 1 passed

100.0%

IN8 923 OUT493

✓

GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN8 923 OUT493

Agentic - Game Backend: Memory match pairs 1 passed

100.0%

IN8 877 OUT90

✓

GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 877 OUT90

Agentic - Game Backend: Mercy invincibility (i-frames) 1 passed

100.0%

IN8 932 OUT210

✓

GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 932 OUT210

Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal) 1 passed

100.0%

IN18 697 OUT1 197

✓

GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN18 697 OUT1 197

Agentic - Game Backend: Minimap downsample 1 passed

100.0%

IN8 908 OUT221

✓

GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 908 OUT221

Agentic - Game Backend: Minimap marker position 1 passed

100.0%

IN8 911 OUT140

✓

GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 911 OUT140

Agentic - Game Backend: N-in-a-row win detection (Connect-4) 1 failed

0.0%

IN39 992 OUT2 479

×

GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.4 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Winner did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 1:21: Unexpected character '#'. 1:28: Char literals must contain exactly one character and close with '''.

Failed

IN39 992 OUT2 479

Agentic - Game Backend: Nonogram row validation 1 passed

100.0%

IN8 888 OUT289

✓

GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 888 OUT289

Agentic - Game Backend: OBB collision (rotating rectangles, SAT) 1 passed

100.0%

IN19 394 OUT2 328

✓

GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN19 394 OUT2 328

Agentic - Game Backend: Particle lifetime & expiry 1 passed

100.0%

IN8 970 OUT190

✓

GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 970 OUT190

Agentic - Game Backend: Party XP share 1 passed

100.0%

IN8 888 OUT111

✓

GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 888 OUT111

Agentic - Game Backend: Patrol waypoints (ping-pong) 1 passed

100.0%

IN8 911 OUT128

✓

GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 911 OUT128

Agentic - Game Backend: Platformer jump arc 1 passed

100.0%

IN8 902 OUT102

✓

GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 902 OUT102

Agentic - Game Backend: Point on circle (trig) 1 passed

100.0%

IN8 935 OUT124

✓

GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 935 OUT124

Agentic - Game Backend: Poker hand comparison 1 passed

100.0%

IN46 847 OUT6 971

✓

GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users

Passed

IN46 847 OUT6 971

Agentic - Game Backend: Poker hand ranking 1 passed

100.0%

IN18 624 OUT1 371

✓

GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN18 624 OUT1 371

Agentic - Game Backend: Pop-cap training 1 passed

100.0%

IN8 889 OUT95

✓

GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 889 OUT95

Agentic - Game Backend: Progress bar fill 1 passed

100.0%

IN8 871 OUT58

✓

GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 871 OUT58

Agentic - Game Backend: Projectile range (physics/trig) 1 passed

100.0%

IN8 893 OUT92

✓

GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 893 OUT92

Agentic - Game Backend: Projectile with ground bounce 1 passed

100.0%

IN8 927 OUT131

✓

GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 927 OUT131

Agentic - Game Backend: Resource gathering 1 passed

100.0%

IN8 888 OUT40

✓

GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 888 OUT40

Agentic - Game Backend: Reversi flip count 1 passed

100.0%

IN18 459 OUT818

✓

GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN18 459 OUT818

Agentic - Game Backend: Rotate 2D vector (trig) 1 passed

100.0%

IN8 904 OUT115

✓

GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 904 OUT115

Agentic - Game Backend: Round-robin deal 1 passed

100.0%

IN8 893 OUT85

✓

GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 893 OUT85

Agentic - Game Backend: RTS resource economy 1 passed

100.0%

IN8 884 OUT100

✓

GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 884 OUT100

Agentic - Game Backend: RTS unit counters & damage 1 passed

100.0%

IN8 936 OUT171

✓

GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 936 OUT171

Agentic - Game Backend: Safest neighbour (danger map) 1 passed

100.0%

IN18 117 OUT807

✓

GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 117 OUT807

Agentic - Game Backend: Scroll viewport 1 passed

100.0%

IN8 906 OUT73

✓

GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 906 OUT73

Agentic - Game Backend: Set card validity 1 passed

100.0%

IN8 885 OUT164

✓

GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 885 OUT164

Agentic - Game Backend: Shop affordability 1 passed

100.0%

IN8 865 OUT62

✓

GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 865 OUT62

Agentic - Game Backend: Sliding puzzle move 1 passed

100.0%

IN8 896 OUT286

✓

GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 896 OUT286

Agentic - Game Backend: Snake head movement 1 passed

100.0%

IN8 953 OUT277

✓

GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 953 OUT277

Agentic - Game Backend: Snake tick 1 failed

0.0%

IN18 450 OUT705

×

GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Tick [1,0;0,0] R food(9,9) got "2,0;1,0;0,0" expected "2,0;1,0" Values: r.ResultJson = "\"2,0;1,0;0,0\"" exp = "\"2,0;1,0\""

Failed

IN18 450 OUT705

Agentic - Game Backend: Snap to grid 1 passed

100.0%

IN8 881 OUT63

✓

GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 881 OUT63

Agentic - Game Backend: Sokoban move/push legality 1 passed

100.0%

IN8 984 OUT244

✓

GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 984 OUT244

Agentic - Game Backend: Solitaire tableau move 1 passed

100.0%

IN8 923 OUT210

✓

GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 923 OUT210

Agentic - Game Backend: Spawn scheduler count 1 passed

100.0%

IN8 880 OUT82

✓

GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 880 OUT82

Agentic - Game Backend: Spawn-wave alive count 1 passed

100.0%

IN8 892 OUT139

✓

GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 892 OUT139

Agentic - Game Backend: Stat level scaling 1 passed

100.0%

IN8 884 OUT65

✓

GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 884 OUT65

Agentic - Game Backend: Steering seek (greedy pursuit) 1 passed

100.0%

IN8 910 OUT135

✓

GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 910 OUT135

Agentic - Game Backend: Tactical movement range 1 passed

100.0%

IN38 848 OUT1 989

✓

GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN38 848 OUT1 989

Agentic - Game Backend: Tech tree eligibility 1 passed

100.0%

IN8 904 OUT210

✓

GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 904 OUT210

Agentic - Game Backend: Tetris line clearing 1 passed

100.0%

IN8 895 OUT289

✓

GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 895 OUT289

Agentic - Game Backend: Threat table top target 1 passed

100.0%

IN8 889 OUT302

✓

GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 889 OUT302

Agentic - Game Backend: Tic-tac-toe result 1 passed

100.0%

IN8 880 OUT291

✓

GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 880 OUT291

Agentic - Game Backend: Toroidal wrap (Pac-Man) 1 passed

100.0%

IN8 916 OUT60

✓

GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 916 OUT60

Agentic - Game Backend: Tower targeting (nearest in range) 1 passed

100.0%

IN8 904 OUT203

✓

GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 904 OUT203

Agentic - Game Backend: Transaction ledger 1 passed

100.0%

IN8 907 OUT213

✓

GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 907 OUT213

Agentic - Game Backend: Trick-taking winner 1 passed

100.0%

IN8 935 OUT503

✓

GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN8 935 OUT503

Agentic - Game Backend: Turn-based duel resolution 1 passed

100.0%

IN8 967 OUT287

✓

GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 967 OUT287

Agentic - Game Backend: Typed armor mitigation 1 passed

100.0%

IN8 891 OUT364

✓

GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 891 OUT364

Agentic - Game Backend: UI menu (wrap/clamp/paginate) 1 passed

100.0%

IN8 983 OUT266

✓

GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 983 OUT266

Agentic - Game Backend: Uno playable cards 1 passed

100.0%

IN8 919 OUT195

✓

GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 919 OUT195

Agentic - Game Backend: Velocity reflection 1 passed

100.0%

IN8 863 OUT114

✓

GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 863 OUT114

Agentic - Game Backend: Wall-slide fall 1 passed

100.0%

IN8 871 OUT73

✓

GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 871 OUT73

Agentic - Game Backend: War card compare 1 passed

100.0%

IN8 891 OUT232

✓

GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 891 OUT232

Agentic - Game Backend: Weighted loot table 1 passed

100.0%

IN8 894 OUT99

✓

GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 894 OUT99

Agentic - Game Backend: Weighted terrain pathfinding 1 passed

100.0%

IN40 408 OUT3 257

✓

GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN40 408 OUT3 257

Agentic - Game Backend: Word-wrap line count 1 passed

100.0%

IN8 876 OUT168

✓

GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 876 OUT168

Agentic - Game Backend: XP to character level 1 passed

100.0%

IN8 884 OUT88

✓

GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 884 OUT88

Agentic - Idempotency (poison-retry / no double charge) 1 passed

100.0%

IN14 365 OUT403

✓

AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN14 365 OUT403

Agentic - Irreversible Commit (measure twice) 1 passed

100.0%

IN15 966 OUT413

✓

AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query 7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN15 966 OUT413

Agentic - Least Privilege (exact grants) 1 passed

100.0%

IN11 652 OUT304

✓

AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query 5 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN11 652 OUT304

Agentic - Ledger (simulated DB read-modify-write) 1 failed

0.0%

IN39 877 OUT1 648

×

AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query 16 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Assertion expression evaluated to false: actual == expected Context: Account alice ended at 600 but expected 0 (in-order apply with overdraft-skip). Values: actual = 600 expected = 0

Failed

IN39 877 OUT1 648

Agentic - Ledger variants (5 RMW cases) 5 passed

100.0%

IN162 972 OUT6 996

✓

LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN16 080 OUT919

✓

LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query 17 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN48 961 OUT1 933

✓

LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN17 242 OUT1 403

✓

LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query 18 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN60 238 OUT1 700

✓

LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN20 451 OUT1 041

Agentic - Lock Ordering (deadlock avoidance) 1 passed

100.0%

IN67 223 OUT1 762

✓

AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query 25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN67 223 OUT1 762

Agentic - Lock Ordering variants (5 cases) 5 passed

100.0%

IN298 746 OUT8 258

✓

LockVariantFiveResFourOps1 AI query 23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN67 092 OUT1 728

✓

LockVariantFourResThreeOps1 AI query 18 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN49 075 OUT1 312

✓

LockVariantHeavyReuse1 AI query 23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN67 093 OUT1 722

✓

LockVariantReverseListed1 AI query 25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN64 423 OUT1 939

✓

LockVariantThreeLockOp1 AI query 20 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN51 063 OUT1 557

Agentic - Migration (cascading rollback + reapply) 1 passed

100.0%

IN21 240 OUT853

✓

AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query 11 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN21 240 OUT853

Agentic - Requirements (detect contradiction) 1 passed

100.0%

IN7 457 OUT376

✓

AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query 2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN7 457 OUT376

Agentic - Retry/Circuit-breaker (transient vs hard-down) 1 passed

100.0%

IN26 529 OUT800

✓

AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query 10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN26 529 OUT800

Agentic - Saga (compensation / distributed rollback) 1 passed

100.0%

IN21 324 OUT304

✓

AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN21 324 OUT304

Agentic - Saga variants (5 compensation cases) 5 passed

100.0%

IN102 077 OUT2 101

✓

SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN13 553 OUT228

✓

SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN13 749 OUT318

✓

SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query 7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN23 658 OUT530

✓

SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN20 060 OUT377

✓

SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query 9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN31 057 OUT648

Agentic - Schema Compat (expand-contract) 1 passed

100.0%

IN21 563 OUT511

✓

AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN21 563 OUT511

Agentic - Scratchpad (self-repair, Dyck, tool-decision) 7 passed

100.0%

IN227 086 OUT8 839

✓

AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query 1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN22 187 OUT909

✓

AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query 4 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN55 297 OUT779

✓

ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query 0 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN10 658 OUT27

✓

ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN21 664 OUT470

✓

ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN21 480 OUT269

✓

ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query 4 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN62 437 OUT5 581

✓

ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query 2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN33 363 OUT804

Agentic - Secret Rotation (zero-downtime ordering) 1 passed

100.0%

IN16 479 OUT259

✓

AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query 5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN16 479 OUT259

Agentic - StatefulOps (incident recovery, concurrency replan) 2 passed

100.0%

IN54 395 OUT1 442

✓

AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query 7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN22 504 OUT662

✓

AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query 8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN31 891 OUT780

Agentic - Test Authoring (write a test that catches the bug) 1 passed

100.0%

IN16 854 OUT471

✓

AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query 5 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN16 854 OUT471

Agentic - Topological Planning (deps + prereq) 1 passed

100.0%

IN25 457 OUT760

✓

AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN25 457 OUT760

Algorithms - 0/1 Knapsack (dynamic programming) 1 passed

100.0%

IN18 133 OUT650

✓

KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN18 133 OUT650

Algorithms - 1D elastic collision 1 passed

100.0%

IN8 938 OUT179

✓

ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 938 OUT179

Algorithms - All-pairs shortest path pair 1 passed

100.0%

IN8 906 OUT480

✓

FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 906 OUT480

Algorithms - Balanced brackets (stack) 1 passed

100.0%

IN8 869 OUT217

✓

ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 869 OUT217

Algorithms - Bellman-Ford shortest path (neg weights/cycles) 1 passed

100.0%

IN8 915 OUT1 094

✓

BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 915 OUT1 094

Algorithms - Binary Gray code 1 passed

100.0%

IN8 842 OUT22

✓

GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 842 OUT22

Algorithms - Bipartite graph check 1 failed

0.0%

IN8 888 OUT386

×

BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: IsBipartite(4,[0,1;1,2;2,3;3,0]) got false expected true Values: r.ResultJson = "false" exp = "true"

Failed

IN8 888 OUT386

Algorithms - Bit tricks (popcount/pow2/lowbit) 1 passed

100.0%

IN8 916 OUT124

✓

BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 916 OUT124

Algorithms - Caesar cipher 1 passed

100.0%

IN8 865 OUT202

✓

CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 865 OUT202

Algorithms - Catalan number 1 passed

100.0%

IN8 887 OUT84

✓

CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 887 OUT84

Algorithms - Chinese remainder theorem 1 passed

100.0%

IN8 905 OUT757

✓

CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 905 OUT757

Algorithms - Circle-circle MTV 1 passed

100.0%

IN8 982 OUT157

✓

CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 982 OUT157

Algorithms - Coin change combinations 1 passed

100.0%

IN18 074 OUT631

✓

CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 074 OUT631

Algorithms - Collatz step count 1 passed

100.0%

IN8 868 OUT92

✓

CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 868 OUT92

Algorithms - Connected components (union-find) 1 failed

0.0%

IN37 711 OUT1 338

×

UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

CountComponents did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 3:8: Expected a variable name, found '['.

Failed

IN37 711 OUT1 338

Algorithms - Container with most water 1 passed

100.0%

IN8 876 OUT225

✓

ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 876 OUT225

Algorithms - Count anagram groups 1 passed

100.0%

IN8 859 OUT144

✓

AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 859 OUT144

Algorithms - Count grid color regions 1 passed

100.0%

IN8 879 OUT444

✓

CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 879 OUT444

Algorithms - Count islands 1 passed

100.0%

IN8 882 OUT501

✓

CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 882 OUT501

Algorithms - Count overlapping occurrences 1 passed

100.0%

IN8 865 OUT102

✓

RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 865 OUT102

Algorithms - Count primes (Sieve of Eratosthenes) 1 passed

100.0%

IN8 844 OUT134

✓

CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 844 OUT134

Algorithms - Cumulative distribution 1 passed

100.0%

IN8 866 OUT246

✓

CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 866 OUT246

Algorithms - Dijkstra shortest path (weighted graph) 1 passed

100.0%

IN8 918 OUT526

✓

DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 918 OUT526

Algorithms - Directed cycle detection 1 failed

0.0%

IN38 122 OUT1 576

×

DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

HasCycle did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 3:40: Expected '(' after the List type. Lists are created with 'new List<T>()' or 'new List<T>(items)', found '['.

Failed

IN38 122 OUT1 576

Algorithms - Distinct subsequences 1 passed

100.0%

IN8 861 OUT314

✓

DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN8 861 OUT314

Algorithms - Divisor count 1 passed

100.0%

IN8 849 OUT68

✓

DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 849 OUT68

Algorithms - Downsample by block average 1 passed

100.0%

IN8 890 OUT209

✓

ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 890 OUT209

Algorithms - Dutch national flag sort 1 passed

100.0%

IN8 857 OUT272

✓

DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 857 OUT272

Algorithms - Egg drop trials 1 failed

0.0%

IN8 918 OUT238

×

EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp.ToString() Context: MinTrials(2,10) got 10 expected 4 Values: r.ResultJson = "10"

Failed

IN8 918 OUT238

Algorithms - Equal subset-sum partition 1 passed

100.0%

IN8 860 OUT257

✓

PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 860 OUT257

Algorithms - Euler totient 1 passed

100.0%

IN8 849 OUT205

✓

EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 849 OUT205

Algorithms - Eulerian path existence 1 passed

100.0%

IN18 341 OUT947

✓

EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN18 341 OUT947

Algorithms - Extended Euclidean 1 passed

100.0%

IN8 961 OUT189

✓

ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 961 OUT189

Algorithms - Format duration H:MM:SS 1 passed

100.0%

IN8 876 OUT184

✓

FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 876 OUT184

Algorithms - GCD and LCM 1 passed

100.0%

IN17 876 OUT229

✓

GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN17 876 OUT229

Algorithms - Grid transpose 1 passed

100.0%

IN8 884 OUT86

✓

TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 884 OUT86

Algorithms - House robber (max non-adjacent sum) 1 passed

100.0%

IN8 848 OUT217

✓

HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 848 OUT217

Algorithms - Integer linear interpolation 1 passed

100.0%

IN8 882 OUT35

✓

LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 882 OUT35

Algorithms - Integer square root 1 passed

100.0%

IN8 864 OUT153

✓

IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 864 OUT153

Algorithms - Integer to base-N string 1 passed

100.0%

IN8 870 OUT107

✓

BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 870 OUT107

Algorithms - Integer to Roman numeral 1 passed

100.0%

IN8 899 OUT515

✓

RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 899 OUT515

Algorithms - Josephus survivor 1 passed

100.0%

IN8 875 OUT237

✓

JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 875 OUT237

Algorithms - Jump game (min jumps) 1 passed

100.0%

IN8 883 OUT238

✓

JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 883 OUT238

Algorithms - Kth largest element 1 passed

100.0%

IN8 873 OUT94

✓

KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 873 OUT94

Algorithms - Least-squares line fit 1 passed

100.0%

IN8 915 OUT263

✓

LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 915 OUT263

Algorithms - Levenshtein edit distance (DP) 1 passed

100.0%

IN8 843 OUT307

✓

EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 843 OUT307

Algorithms - Log-scale bucket index 1 passed

100.0%

IN8 908 OUT116

✓

LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 908 OUT116

Algorithms - Logarithmic bin index 1 passed

100.0%

IN8 897 OUT86

✓

LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 897 OUT86

Algorithms - Longest common prefix 1 passed

100.0%

IN8 856 OUT175

✓

CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 856 OUT175

Algorithms - Longest common subsequence 1 passed

100.0%

IN8 841 OUT246

✓

LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 841 OUT246

Algorithms - Longest common substring 1 passed

100.0%

IN8 858 OUT179

✓

CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 858 OUT179

Algorithms - Longest increasing subsequence (DP) 1 passed

100.0%

IN8 875 OUT302

✓

LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 875 OUT302

Algorithms - Longest palindromic substring 1 passed

100.0%

IN8 853 OUT258

✓

LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 853 OUT258

Algorithms - Longest repeated substring 1 failed

0.0%

IN8 858 OUT180

×

LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp.ToString() Context: Length([aaaa]) got 2 expected 3 Values: r.ResultJson = "2"

Failed

IN8 858 OUT180

Algorithms - Lower bound (binary search) 1 passed

100.0%

IN8 886 OUT150

✓

LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 886 OUT150

Algorithms - Luhn checksum validation 1 passed

100.0%

IN8 887 OUT188

✓

LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 887 OUT188

Algorithms - Majority element 1 passed

100.0%

IN8 851 OUT254

✓

MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 851 OUT254

Algorithms - Matrix chain multiplication 1 passed

100.0%

IN8 886 OUT364

✓

MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 886 OUT364

Algorithms - Maximum subarray sum (Kadane) 1 passed

100.0%

IN8 876 OUT131

✓

MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 876 OUT131

Algorithms - Merge K sorted lists 1 passed

100.0%

IN8 869 OUT225

✓

MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 869 OUT225

Algorithms - Merge overlapping intervals 1 passed

100.0%

IN18 284 OUT905

✓

MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 284 OUT905

Algorithms - Min-max normalization 1 passed

100.0%

IN8 878 OUT281

✓

MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 878 OUT281

Algorithms - Minimum coin change (DP) 1 passed

100.0%

IN8 869 OUT288

✓

CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 869 OUT288

Algorithms - Minimum knight moves (BFS) 1 passed

100.0%

IN28 126 OUT1 274

✓

KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN28 126 OUT1 274

Algorithms - Minimum path sum (grid DP) 1 passed

100.0%

IN8 882 OUT350

✓

MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 882 OUT350

Algorithms - Minimum spanning tree (Kruskal) 1 passed

100.0%

IN18 406 OUT1 229

✓

KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN18 406 OUT1 229

Algorithms - Minimum window substring 1 passed

100.0%

IN8 890 OUT462

✓

MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN8 890 OUT462

Algorithms - Modular exponentiation (fast pow) 1 passed

100.0%

IN8 901 OUT142

✓

ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 901 OUT142

Algorithms - Modular inverse (extended Euclid) 1 passed

100.0%

IN27 770 OUT976

✓

ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN27 770 OUT976

Algorithms - Moving average 1 passed

100.0%

IN8 880 OUT230

✓

MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 880 OUT230

Algorithms - Next greater element 1 passed

100.0%

IN8 863 OUT323

✓

NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 863 OUT323

Algorithms - Next lexicographic permutation 1 passed

100.0%

IN8 871 OUT324

✓

NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 871 OUT324

Algorithms - Nth Fibonacci mod m 1 passed

100.0%

IN8 870 OUT119

✓

FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 870 OUT119

Algorithms - Overlapping substring count 1 passed

100.0%

IN8 855 OUT96

✓

SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 855 OUT96

Algorithms - Palindrome & anagram checks 1 passed

100.0%

IN8 885 OUT642

✓

PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 885 OUT642

Algorithms - Pascal triangle row 1 passed

100.0%

IN8 863 OUT148

✓

PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 863 OUT148

Algorithms - Point-to-segment distance 1 passed

100.0%

IN27 791 OUT953

✓

PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN27 791 OUT953

Algorithms - Prime factorization 1 passed

100.0%

IN8 874 OUT131

✓

PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 874 OUT131

Algorithms - Product of array except self 1 passed

100.0%

IN18 117 OUT652

✓

ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 117 OUT652

Algorithms - Quadratic Bezier point 1 passed

100.0%

IN8 940 OUT149

✓

BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 940 OUT149

Algorithms - Range sum via prefix sums 1 passed

100.0%

IN8 862 OUT69

✓

PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 862 OUT69

Algorithms - Reverse words & Hamming distance 1 passed

100.0%

IN8 866 OUT110

✓

ReverseWordsCodeGenCode-gen: model implements reverse-words + Hamming; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 866 OUT110

Algorithms - Rod cutting 1 passed

100.0%

IN8 896 OUT224

✓

RodCuttingCodeGenCode-gen: model implements rod cutting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 896 OUT224

Algorithms - Roman numeral to integer 1 passed

100.0%

IN8 885 OUT220

✓

RomanToIntCodeGenCode-gen: model implements Roman-to-int; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 885 OUT220

Algorithms - Rotate array right by k 1 passed

100.0%

IN8 889 OUT215

✓

RotateArrayCodeGenCode-gen: model implements right rotation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 889 OUT215

Algorithms - Rotate grid 90 CW (discriminator) 1 failed

0.0%

IN8 892 OUT112

×

RotateMatrixCodeGenCode-gen: model implements 90-degree CW rotation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Rotate90 2x3 [abcdef] got "badcfe" expected "ecafdb" Values: r.ResultJson = "\"badcfe\"" exp = "\"ecafdb\""

Failed

IN8 892 OUT112

Algorithms - RPN evaluator 1 passed

100.0%

IN8 894 OUT233

✓

RpnCodeGenCode-gen: model implements an RPN evaluator; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 894 OUT233

Algorithms - Run-length compression (if shorter) 1 passed

100.0%

IN8 866 OUT168

✓

StringCompressCodeGenCode-gen: model implements string compression; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 866 OUT168

Algorithms - Run-length encoding (encode/decode) 1 passed

100.0%

IN8 912 OUT299

✓

RunLengthCodeGenCode-gen: model implements RLE encode+decode; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 912 OUT299

Algorithms - Semantic versioning (compare + bump) 1 passed

100.0%

IN8 942 OUT363

✓

SemVerCodeGenCode-gen: model implements semver compare + bump; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:InstructionFollowinglevel:Easysource:Ai4Users

Passed

IN8 942 OUT363

Algorithms - Single-pile Nim (optimal play) 1 passed

100.0%

IN8 874 OUT33

✓

NimCodeGenCode-gen: model implements optimal Nim; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 874 OUT33

Algorithms - Sliding window maximum 1 passed

100.0%

IN8 866 OUT251

✓

SlidingWindowMaxCodeGenCode-gen: model implements sliding window maximum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 866 OUT251

Algorithms - Spiral matrix traversal 1 passed

100.0%

IN8 875 OUT276

✓

SpiralMatrixCodeGenCode-gen: model implements spiral traversal; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 875 OUT276

Algorithms - Square matrix multiply 1 passed

100.0%

IN8 867 OUT294

✓

MatMulCodeGenCode-gen: model implements square matrix multiply; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 867 OUT294

Algorithms - Strongly connected components 1 passed

100.0%

IN8 878 OUT562

✓

SccCountCodeGenCode-gen: model implements SCC count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN8 878 OUT562

Algorithms - Subset sum reachability 1 passed

100.0%

IN18 041 OUT463

✓

SubsetSumCodeGenCode-gen: model implements subset-sum reachability; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 041 OUT463

Algorithms - Substring search (KMP) 1 passed

100.0%

IN27 657 OUT902

✓

KmpCodeGenCode-gen: model implements KMP search; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN27 657 OUT902

Algorithms - Sum of digits & digital root 1 passed

100.0%

IN8 852 OUT143

✓

DigitalRootCodeGenCode-gen: model implements sum-of-digits + digital root; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 852 OUT143

Algorithms - Temperature conversion 1 passed

100.0%

IN8 863 OUT55

✓

TemperatureCodeGenCode-gen: model implements temperature conversion; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 863 OUT55

Algorithms - Topological sort (lexicographically smallest) 1 passed

100.0%

IN8 904 OUT429

✓

TopoCodeGenCode-gen: model implements lexicographically-smallest topological sort.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 904 OUT429

Algorithms - Trailing zeros of factorial 1 passed

100.0%

IN8 864 OUT79

✓

TrailingZerosCodeGenCode-gen: model implements trailing zeros; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 864 OUT79

Algorithms - Trapping rain water 1 passed

100.0%

IN8 863 OUT364

✓

TrappingRainCodeGenCode-gen: model implements trapping rain water; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 863 OUT364

Algorithms - Two-sum indices 1 passed

100.0%

IN8 870 OUT132

✓

TwoSumCodeGenCode-gen: model implements two-sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 870 OUT132

Algorithms - Unique grid paths 1 passed

100.0%

IN8 845 OUT292

✓

UniquePathsCodeGenCode-gen: model implements unique paths; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 845 OUT292

Algorithms - Valid Sudoku unit 1 passed

100.0%

IN8 850 OUT70

✓

SudokuUnitCodeGenCode-gen: model implements Sudoku unit validation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 850 OUT70

Algorithms - Wildcard string matching 1 passed

100.0%

IN8 868 OUT256

✓

WildcardMatchCodeGenCode-gen: model implements wildcard match; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 868 OUT256

Algorithms - Z-function 1 passed

100.0%

IN8 869 OUT299

✓

ZFunctionCodeGenCode-gen: model implements the Z-function; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 869 OUT299

Database - Conditional aggregation pivot 1 passed

100.0%

IN947 OUT64

✓

PivotCountsCodeGenCode-gen: model writes the pivot query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN947 OUT64

Database - Count distinct opponents 1 passed

100.0%

IN927 OUT19

✓

DistinctCountCodeGenCode-gen: model writes the COUNT(DISTINCT) query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Easysource:Ai4Users

Passed

IN927 OUT19

Database - Date-range filter (BETWEEN) 1 passed

100.0%

IN957 OUT43

✓

DateRangeCodeGenCode-gen: model writes the date-range query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN957 OUT43

Database - EXISTS subquery (teams with matches) 1 passed

100.0%

IN934 OUT32

✓

ExistsSubqueryCodeGenCode-gen: model writes the EXISTS query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN934 OUT32

Database - JOIN player totals (two tables) 1 passed

100.0%

IN956 OUT38

✓

JoinScoresCodeGenCode-gen: model writes a JOIN + aggregate query; graded via QueryJson vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN956 OUT38

Database - LEFT JOIN anti-join (no items) 1 passed

100.0%

IN930 OUT28

✓

LeftJoinNullsCodeGenCode-gen: model writes the anti-join query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN930 OUT28

Database - Player stats SQL (multi-aggregate, QueryJson) 1 passed

100.0%

IN960 OUT36

✓

PlayerStatsCodeGenCode-gen: model writes a multi-aggregate stats query; graded by comparing its QueryJson output to the oracle query's (arbitrary shape, no struct).1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN960 OUT36

Database - Players above average score (subquery) 1 passed

100.0%

IN925 OUT27

✓

AboveAverageCodeGenCode-gen: model writes an above-average query; graded via QueryJson vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN925 OUT27

Database - Players who never played (anti-join) 1 passed

100.0%

IN922 OUT30

✓

NeverPlayedCodeGenCode-gen: model writes an anti-join query; graded via QueryJson vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN922 OUT30

Database - RANK window function 1 passed

100.0%

IN952 OUT27

✓

WindowRankCodeGenCode-gen: model writes the RANK() query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN952 OUT27

Database - Running total (window function) 1 passed

100.0%

IN949 OUT24

✓

RunningTotalCodeGenCode-gen: model writes the window running-total query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN949 OUT24

Database - Scoreboard SQL (players with >1 win, HAVING) 1 passed

100.0%

IN956 OUT34

✓

WinFilterCodeGenCode-gen: model writes a HAVING query; runs under try/catch with self-repair; result compared to the oracle query.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN956 OUT34

Database - Scoreboard SQL (top scorers query) 1 passed

100.0%

IN957 OUT25

✓

ScoreboardTopScorersCodeGenCode-gen: model writes a scoreboard SQL query; runs under try/catch with a self-repair loop on SQL errors; result compared to the oracle query.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN957 OUT25

Database - Self-join same-team pairs 1 passed

100.0%

IN940 OUT44

✓

SelfJoinCodeGenCode-gen: model writes the self-join query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN940 OUT44

Database - Top scorer per region 1 passed

100.0%

IN942 OUT34

✓

TopPerGroupCodeGenCode-gen: model writes the per-group top query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN942 OUT34

Database - UPDATE restock (SQL write) 1 passed

100.0%

IN934 OUT17

✓

UpdateInventoryCodeGenCode-gen: model writes an UPDATE; applied under try/catch, resulting state compared (via QueryJson) to the oracle update's state.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN934 OUT17

Database - Upsert with max merge 1 passed

100.0%

IN1 988 OUT72

✓

UpsertCodeGenCode-gen: model writes the upsert; executed on a fresh DB and the table compared.2 AI queries 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN1 988 OUT72

SDLC - Acceptance criteria check 1 passed

100.0%

IN8 893 OUT550

✓

SdlcAcceptanceCodeGenCode-gen: model implements acceptance-criteria check; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 893 OUT550

SDLC - Breaking API change detection 1 passed

100.0%

IN8 890 OUT346

✓

SdlcApiBreakingCodeGenCode-gen: model implements API-breaking detection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 890 OUT346

SDLC - Condense without losing facts 1 passed

100.0%

IN1 048 OUT60

✓

NoInfoLossCodeGenRedaction test: model condenses keeping facts; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 048 OUT60

SDLC - Config merge with override 1 passed

100.0%

IN8 884 OUT306

✓

ConfigMergeCodeGenCode-gen: model implements config merge; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 884 OUT306

SDLC - Extract fields into fixed template 1 passed

100.0%

IN1 004 OUT22

✓

FieldExtractCodeGenRedaction test: model extracts fields into the template; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 004 OUT22

SDLC - Locate conflicting requirement pair 1 failed

0.0%

IN18 799 OUT1 546

×

SdlcConstraintConflictCodeGenCode-gen: model locates the conflicting requirement pair; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users

FindConflict threw: unsupported operator

Failed

IN18 799 OUT1 546

SDLC - Locate redundant requirement 1 failed

0.0%

IN8 931 OUT868

×

SdlcRedundantRequirementCodeGenCode-gen: model locates a redundant requirement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users

FindRedundant threw: Cannot parse '= 7' as int.

Failed

IN8 931 OUT868

SDLC - Mask PII to placeholder 1 passed

100.0%

IN972 OUT42

✓

MaskPiiCodeGenRedaction test: model masks PII to a placeholder; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN972 OUT42

SDLC - Modules on a dependency cycle 1 passed

100.0%

IN8 896 OUT489

✓

DependencyCycleModulesCodeGenCode-gen: model implements dependency-cycle detection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN8 896 OUT489

SDLC - Remove duplicate requirements 1 passed

100.0%

IN945 OUT10

✓

DeduplicateCodeGenRedaction test: model removes duplicate requirements; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN945 OUT10

SDLC - Renumber requirements sequentially 1 passed

100.0%

IN1 012 OUT42

✓

RenumberCodeGenRedaction test: model renumbers sequentially; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 012 OUT42

SDLC - Reorganize notes into section template 1 passed

100.0%

IN1 126 OUT94

✓

RewriteSectionsCodeGenRedaction test: model reorganizes the notes; deterministic checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 126 OUT94

SDLC - Required semver bump from changelog 1 passed

100.0%

IN8 889 OUT205

✓

SemverBumpCodeGenCode-gen: model implements the semver-bump rule; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Easysource:Ai4Users

Passed

IN8 889 OUT205

SDLC - Requirement-to-test traceability 1 passed

100.0%

IN8 911 OUT188

✓

SdlcTraceabilityCodeGenCode-gen: model implements traceability gap; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 911 OUT188

SDLC - Requirements completeness (spec coverage gap) 1 passed

100.0%

IN8 906 OUT293

✓

SdlcSpecCoverageCodeGenCode-gen: model implements spec-gap detection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 906 OUT293

SDLC - Requirements consistency (contradiction detection) 1 failed

0.0%

IN39 412 OUT2 625

×

ConsistencyCodeGenCode-gen: model implements requirements-consistency checking; graded vs oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Consistent did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 48:14: Expected a variable name, found '['.

Failed

IN39 412 OUT2 625

SDLC - Reshape prose into JSON template 1 passed

100.0%

IN1 014 OUT28

✓

JsonShapeCodeGenRedaction test: model reshapes prose into JSON; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 014 OUT28

SDLC - Reshape rows into Markdown table 1 passed

100.0%

IN968 OUT31

✓

TableFormatCodeGenRedaction test: model builds a Markdown table; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN968 OUT31

SDLC - Rewrite in imperative requirement style 1 passed

100.0%

IN991 OUT50

✓

StyleRewriteCodeGenRedaction test: model rewrites notes in imperative style; checker + self-review loop.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN991 OUT50

SDLC - Roll up estimates over task tree 1 passed

100.0%

IN8 902 OUT222

✓

EstimateRollupCodeGenCode-gen: model implements estimate roll-up over a task tree; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 902 OUT222

SDLC - State-machine reachability 1 passed

100.0%

IN8 888 OUT337

✓

SdlcStateReachableCodeGenCode-gen: model implements state reachability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 888 OUT337

SDLC - Ticket priority sort 1 passed

100.0%

IN8 881 OUT404

✓

SdlcPrioritySortCodeGenCode-gen: model implements priority sort; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 881 OUT404

SDLC - Unify glossary term consistency 1 passed

100.0%

IN972 OUT38

✓

GlossaryCodeGenRedaction test: model unifies synonyms to the canonical term; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN972 OUT38

Qwen 3.6 27B FP8 Instruct | Benchmark #3

SC# Coding & Agentic Benchmark by Ai4Users

Results by difficulty

Results by category