Ai4Users - Qwen 3.6 35B NVFP4 Instruct

Benchmark items
Agentic - Blue-Green Deploy (health-gated cutover) 1 passed	100.0%	IN14 575 OUT228
✓	AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query 4 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN14 575 OUT228
Agentic - Bugfix (root-cause vs regression) 1 passed	100.0%	IN20 113 OUT256
✓	AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query 1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users	Passed	IN20 113 OUT256
Agentic - Cache Invalidation (exact set) 1 passed	100.0%	IN24 056 OUT737
✓	AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query 7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN24 056 OUT737
Agentic - Canary (metric-gated release) 1 passed	100.0%	IN45 087 OUT1 106
✓	AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN45 087 OUT1 106
Agentic - Canary variants (5 metric cases) 5 passed	100.0%	IN187 771 OUT5 593
✓	CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN31 734 OUT1 105
✓	CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN36 466 OUT1 073
✓	CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN31 463 OUT854
✓	CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query 14 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN56 469 OUT1 620
✓	CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN31 639 OUT941
Agentic - Code Review (find injected defect) 1 passed	100.0%	IN13 557 OUT939
✓	AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN13 557 OUT939
Agentic - Config Drift (remediate exact) 1 passed	100.0%	IN12 198 OUT425
✓	AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query 6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN12 198 OUT425
Agentic - Constraints (multi-section satisfaction) 1 passed	100.0%	IN11 676 OUT665
✓	AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN11 676 OUT665
Agentic - Constraints variants (5 cases) 5 passed	100.0%	IN53 476 OUT2 394
✓	ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN10 610 OUT673
✓	ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query 4 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN12 698 OUT380
✓	ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN9 969 OUT397
✓	ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN10 257 OUT581
✓	ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN9 942 OUT363
Agentic - Data Reconciliation (per-field precedence) 1 passed	100.0%	IN14 564 OUT1 030
✓	AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN14 564 OUT1 030
Agentic - Diagnosis (root cause under budget) 1 passed	100.0%	IN12 607 OUT676
✓	AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN12 607 OUT676
Agentic - Diagnosis variants (5 cases) 5 passed	100.0%	IN106 168 OUT5 832
✓	DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN15 512 OUT916
✓	DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN32 868 OUT1 417
✓	DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN15 340 OUT830
✓	DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN16 265 OUT1 129
✓	DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query 16 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN26 183 OUT1 540
Agentic - ETL Repair (root-cause in stage 2) 1 passed	100.0%	IN33 759 OUT1 117
✓	AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN33 759 OUT1 117
Agentic - Feature Flag (staged rollout + kill-switch) 1 passed	100.0%	IN24 381 OUT771
✓	AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN24 381 OUT771
Agentic - Game Backend: 1D bouncing ball 1 passed	100.0%	IN8 783 OUT2 977
✓	GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 783 OUT2 977
Agentic - Game Backend: 1D swept anti-tunneling 1 passed	100.0%	IN8 794 OUT235
✓	GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 794 OUT235
Agentic - Game Backend: 2048 left-move (slide + merge) 1 passed	100.0%	IN8 803 OUT1 616
✓	GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 803 OUT1 616
Agentic - Game Backend: 8-direction from vector 1 passed	100.0%	IN8 810 OUT266
✓	GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 810 OUT266
Agentic - Game Backend: AABB collision resolution (MTV) 1 failed	0.0%	IN28 008 OUT1 053
×	GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: ResolveAabb got "2,4" expected "4,2" Values: r.ResultJson = "\"2,4\"" exp = "\"4,2\""	Failed	IN28 008 OUT1 053
Agentic - Game Backend: AABB containment 1 passed	100.0%	IN8 739 OUT1 246
✓	GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 739 OUT1 246
Agentic - Game Backend: Acceleration to max speed 1 passed	100.0%	IN8 741 OUT118
✓	GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 741 OUT118
Agentic - Game Backend: Achievement thresholds 1 passed	100.0%	IN8 733 OUT789
✓	GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 733 OUT789
Agentic - Game Backend: Active status effects 1 passed	100.0%	IN8 755 OUT1 194
✓	GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 755 OUT1 194
Agentic - Game Backend: Animation frame at time 1 passed	100.0%	IN8 773 OUT90
✓	GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 773 OUT90
Agentic - Game Backend: Army upkeep income 1 passed	100.0%	IN8 721 OUT790
✓	GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 721 OUT790
Agentic - Game Backend: Axis-aligned path follow 1 passed	100.0%	IN20 343 OUT3 116
✓	GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN20 343 OUT3 116
Agentic - Game Backend: Battleship shot resolution 1 passed	100.0%	IN33 633 OUT4 108
✓	GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN33 633 OUT4 108
Agentic - Game Backend: Bounce restitution 1 passed	100.0%	IN8 760 OUT77
✓	GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 760 OUT77
Agentic - Game Backend: Bresenham line rasterization 1 failed	0.0%	IN43 539 OUT3 290
×	GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: Line (0,0)->(2,0) got "0,01,02,0" expected "0,0;1,0;2,0" Values: r.ResultJson = "\"0,01,02,0\"" exp = "\"0,0;1,0;2,0\""	Failed	IN43 539 OUT3 290
Agentic - Game Backend: Buff/debuff stacking 1 failed	0.0%	IN37 154 OUT1 162
×	GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users ActivePower did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 22:43: Expected '{' after the struct name. Structs are created with 'new Name { Field = value }', found '('.	Failed	IN37 154 OUT1 162
Agentic - Game Backend: Build queue completions 1 passed	100.0%	IN8 754 OUT1 048
✓	GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 754 OUT1 048
Agentic - Game Backend: Camera-relative movement 1 passed	100.0%	IN18 009 OUT352
✓	GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 009 OUT352
Agentic - Game Backend: Card hand sort 1 failed	0.0%	IN38 420 OUT2 052
×	GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users SortHand did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 26:5: Expected an expression, found 'struct'.	Failed	IN38 420 OUT2 052
Agentic - Game Backend: Card-game rules (blackjack/deck/compare) 1 passed	100.0%	IN18 577 OUT1 242
✓	GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 577 OUT1 242
Agentic - Game Backend: Chart scaling & histogram bins 1 passed	100.0%	IN8 826 OUT1 139
✓	GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 826 OUT1 139
Agentic - Game Backend: Checkers captures 1 passed	100.0%	IN8 796 OUT3 318
✓	GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 796 OUT3 318
Agentic - Game Backend: Collision primitives (circle/aabb/segment) 1 passed	100.0%	IN18 782 OUT1 336
✓	GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 782 OUT1 336
Agentic - Game Backend: Combat turn order (initiative) 1 failed	0.0%	IN37 526 OUT1 414
×	GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users TurnOrder did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 46:20: Unknown variable 'initiatives'. 47:21: Unknown variable 'initiatives'. 52:20: Unknown variable 'names'. 53:21: Unknown variable 'names'.	Failed	IN37 526 OUT1 414
Agentic - Game Backend: Combo scoring (rising multiplier) 1 passed	100.0%	IN8 758 OUT1 551
✓	GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 758 OUT1 551
Agentic - Game Backend: Control-point capture 1 passed	100.0%	IN8 771 OUT216
✓	GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 771 OUT216
Agentic - Game Backend: Controls & Inventory (spec-driven function battery) 1 passed	100.0%	IN18 176 OUT511
✓	GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 176 OUT511
Agentic - Game Backend: Cooldowns & resource regen 1 failed	0.0%	IN8 829 OUT2 843
×	GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users CooldownRemaining did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: Entry point 'CooldownRemaining' was not found. Available functions: AskAi, RunScript, SkipTest, __CreateInMemoryDatabase, __DbExecute, __DbExecuteScript, __DbInsert, __DbQueryFirstJson, __DbQueryJson, __DbScalarJson, AbilityReady.	Failed	IN8 829 OUT2 843
Agentic - Game Backend: Coyote-time jump 1 passed	100.0%	IN17 629 OUT141
✓	GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN17 629 OUT141
Agentic - Game Backend: Crafting capacity 1 failed	0.0%	IN38 486 OUT2 131
×	GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users MaxCrafts did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 68:19: Expected ')' to close the parenthesized expression, found '='.	Failed	IN38 486 OUT2 131
Agentic - Game Backend: Cribbage fifteens 1 passed	100.0%	IN8 751 OUT1 790
✓	GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 751 OUT1 790
Agentic - Game Backend: Critical-hit damage 1 passed	100.0%	IN8 741 OUT465
✓	GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 741 OUT465
Agentic - Game Backend: Damage over time 1 passed	100.0%	IN8 767 OUT164
✓	GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 767 OUT164
Agentic - Game Backend: Day/night cycle phase 1 passed	100.0%	IN8 753 OUT1 392
✓	GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 753 OUT1 392
Agentic - Game Backend: Dice notation max roll 1 passed	100.0%	IN8 768 OUT227
✓	GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 768 OUT227
Agentic - Game Backend: Enemy AI state machine 1 passed	100.0%	IN18 211 OUT844
✓	GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN18 211 OUT844
Agentic - Game Backend: Equipment slot validation 1 passed	100.0%	IN8 768 OUT178
✓	GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 768 OUT178
Agentic - Game Backend: Flood-fill region size 1 passed	100.0%	IN38 317 OUT1 908
✓	GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN38 317 OUT1 908
Agentic - Game Backend: Fog of war visibility 1 passed	100.0%	IN8 796 OUT1 173
✓	GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 796 OUT1 173
Agentic - Game Backend: Fog reveal (Chebyshev radius) 1 failed	0.0%	IN37 500 OUT1 468
×	GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Revealed did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 10:11: Expected a variable name, found '['.	Failed	IN37 500 OUT1 468
Agentic - Game Backend: Friction stop frame 1 passed	100.0%	IN8 749 OUT1 430
✓	GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 749 OUT1 430
Agentic - Game Backend: Game of Life (code-gen vs oracle) 1 failed	0.0%	IN8 808 OUT1 536
×	GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users candidate did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 1:24: Char literals must contain exactly one character and close with '''. 10:5: Char literals must contain exactly one character and close with '''. 15:8: Char literals must contain exactly one character and close with '''. 17:1: Unexpected character '`'. 17:2: Unexpected character '`'. 17:3: Unexpected character '`'. 69:1: Unexpected character '`'. 69:2: Unexpected character '`'. 69:3: Unexpected character '`'. 71:26: Unexpected character '#'. 74:28: Unexpected character '`'. 74:36: Unexpected character '`'. 75:29: Unexpected character '`'. 75:32: Unexpected character '`'. 79:84: Unexpected character '`'. 79:92: Unexpected character '`'. 79:104: Unexpected character '`'. 79:109: Unexpected character '`'. 79:115: Unexpected character '`'. 79:136: Unexpected character '`'. 79:150: Unexpected character '`'. 79:155: Unexpected character '`'. 79:176: Unexpected character '`'. 79:180: Unexpected character '`'. 82:44: Unexpected character '→'. 82:67: Unexpected character '✓'. 83:52: Unexpected character '→'. 83:75: Unexpected character '✓'. 85:49: Unexpected character '#'. 87:50: Unexpected character '\'. 89:8: Char literals must contain exactly one character and close with '''. 93:44: Unexpected character '`'. 93:58: Unexpected character '`'. 93:128: Unexpected character '`'. 93:142: Unexpected character '`'. 93:270: Unexpected character '`'. 93:284: Unexpected character '`'. 97:1: Unexpected character '`'. 97:2: Unexpected character '`'. 97:3: Unexpected character '`'. 149:1: Unexpected character '`'. 149:2: Unexpected character '`'. 149:3: Unexpected character '`'. 153:7: Char literals must contain exactly one character and close with '''. 155:1: Unexpected character '`'. 155:2: Unexpected character '`'. 155:3: Unexpected character '`'. 207:1: Unexpected character '`'. 207:2: Unexpected character '`'. 207:3: Unexpected character '`'.	Failed	IN8 808 OUT1 536
Agentic - Game Backend: Greedy coin change 1 passed	100.0%	IN18 145 OUT701
✓	GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 145 OUT701
Agentic - Game Backend: Grid inventory & stacking (Diablo-style) 1 passed	100.0%	IN8 883 OUT1 063
✓	GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 883 OUT1 063
Agentic - Game Backend: Grid pathfinding (shortest path, BFS) 1 passed	100.0%	IN32 082 OUT3 543
✓	GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.3 AI queries 0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN32 082 OUT3 543
Agentic - Game Backend: Grid raycast 1 passed	100.0%	IN17 981 OUT613
✓	GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN17 981 OUT613
Agentic - Game Backend: Health bar segments 1 failed	0.0%	IN8 807 OUT2 497
×	GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users RenderBar did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: Entry point 'RenderBar' was not found. Available functions: AskAi, RunScript, SkipTest, __CreateInMemoryDatabase, __DbExecute, __DbExecuteScript, __DbInsert, __DbQueryFirstJson, __DbQueryJson, __DbScalarJson, FilledSegments.	Failed	IN8 807 OUT2 497
Agentic - Game Backend: High-score table insert 1 passed	100.0%	IN8 761 OUT3 705
✓	GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 761 OUT3 705
Agentic - Game Backend: Hit chance 1 passed	100.0%	IN8 745 OUT78
✓	GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 745 OUT78
Agentic - Game Backend: Inventory encumbrance 1 passed	100.0%	IN8 742 OUT767
✓	GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 742 OUT767
Agentic - Game Backend: Inventory first-fit placement 1 passed	100.0%	IN31 563 OUT2 916
✓	GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN31 563 OUT2 916
Agentic - Game Backend: Inventory stack split 1 passed	100.0%	IN8 757 OUT65
✓	GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 757 OUT65
Agentic - Game Backend: Isometric projection 1 passed	100.0%	IN17 854 OUT260
✓	GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN17 854 OUT260
Agentic - Game Backend: Jump buffer 1 failed	0.0%	IN8 763 OUT155
×	GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: Jumped([PL],0) got true expected false Values: r.ResultJson = "true" exp = "false"	Failed	IN8 763 OUT155
Agentic - Game Backend: Knockback with arena clamp 1 passed	100.0%	IN8 802 OUT865
✓	GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 802 OUT865
Agentic - Game Backend: Leaderboard rank 1 failed	0.0%	IN38 337 OUT1 323
×	GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users Rank did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 6:15: Expected a variable name, found '['.	Failed	IN38 337 OUT1 323
Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over) 1 passed	100.0%	IN8 886 OUT210
✓	GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 886 OUT210
Agentic - Game Backend: Lifesteal healing 1 passed	100.0%	IN8 746 OUT1 026
✓	GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 746 OUT1 026
Agentic - Game Backend: Lights Out toggle 1 passed	100.0%	IN20 321 OUT3 019
✓	GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN20 321 OUT3 019
Agentic - Game Backend: Match-3 cascade 1 passed	100.0%	IN8 820 OUT4 249
✓	GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN8 820 OUT4 249
Agentic - Game Backend: Match-3 detection 1 failed	0.0%	IN38 079 OUT1 736
×	GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users CountMatched did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 4:10: Expected ';' after the variable declaration, found ','.	Failed	IN38 079 OUT1 736
Agentic - Game Backend: Match-3 swap validity 1 passed	100.0%	IN18 379 OUT1 197
✓	GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN18 379 OUT1 197
Agentic - Game Backend: Memory match pairs 1 passed	100.0%	IN8 747 OUT1 039
✓	GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 747 OUT1 039
Agentic - Game Backend: Mercy invincibility (i-frames) 1 passed	100.0%	IN17 908 OUT413
✓	GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN17 908 OUT413
Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal) 1 passed	100.0%	IN18 671 OUT1 644
✓	GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN18 671 OUT1 644
Agentic - Game Backend: Minimap downsample 1 passed	100.0%	IN8 778 OUT1 514
✓	GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 778 OUT1 514
Agentic - Game Backend: Minimap marker position 1 passed	100.0%	IN18 677 OUT1 199
✓	GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN18 677 OUT1 199
Agentic - Game Backend: N-in-a-row win detection (Connect-4) 1 failed	0.0%	IN37 518 OUT1 423
×	GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.4 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: Winner mismatch 4x4 board=[....XXXX........] got "none" expected "X" Values: r.ResultJson = "\"none\"" exp = "\"X\""	Failed	IN37 518 OUT1 423
Agentic - Game Backend: Nonogram row validation 1 passed	100.0%	IN8 758 OUT3 695
✓	GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 758 OUT3 695
Agentic - Game Backend: OBB collision (rotating rectangles, SAT) 1 passed	100.0%	IN8 905 OUT2 155
✓	GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN8 905 OUT2 155
Agentic - Game Backend: Particle lifetime & expiry 1 passed	100.0%	IN8 840 OUT190
✓	GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 840 OUT190
Agentic - Game Backend: Party XP share 1 passed	100.0%	IN26 630 OUT279
✓	GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN26 630 OUT279
Agentic - Game Backend: Patrol waypoints (ping-pong) 1 passed	100.0%	IN8 781 OUT1 616
✓	GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 781 OUT1 616
Agentic - Game Backend: Platformer jump arc 1 passed	100.0%	IN8 772 OUT109
✓	GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 772 OUT109
Agentic - Game Backend: Point on circle (trig) 1 passed	100.0%	IN8 805 OUT1 149
✓	GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 805 OUT1 149
Agentic - Game Backend: Poker hand comparison 1 failed	0.0%	IN56 531 OUT10 005
×	GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users Compare did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 94:27: Expected '{' after the struct name. Structs are created with 'new Name { Field = value }', found '('.	Failed	IN56 531 OUT10 005
Agentic - Game Backend: Poker hand ranking 1 passed	100.0%	IN8 824 OUT3 992
✓	GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN8 824 OUT3 992
Agentic - Game Backend: Pop-cap training 1 passed	100.0%	IN8 759 OUT91
✓	GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 759 OUT91
Agentic - Game Backend: Progress bar fill 1 passed	100.0%	IN8 741 OUT109
✓	GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 741 OUT109
Agentic - Game Backend: Projectile range (physics/trig) 1 passed	100.0%	IN17 641 OUT126
✓	GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN17 641 OUT126
Agentic - Game Backend: Projectile with ground bounce 1 passed	100.0%	IN17 788 OUT291
✓	GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN17 788 OUT291
Agentic - Game Backend: Resource gathering 1 passed	100.0%	IN8 758 OUT49
✓	GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 758 OUT49
Agentic - Game Backend: Reversi flip count 1 passed	100.0%	IN8 829 OUT410
✓	GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN8 829 OUT410
Agentic - Game Backend: Rotate 2D vector (trig) 1 passed	100.0%	IN17 727 OUT266
✓	GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN17 727 OUT266
Agentic - Game Backend: Round-robin deal 1 passed	100.0%	IN18 929 OUT1 481
✓	GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 929 OUT1 481
Agentic - Game Backend: RTS resource economy 1 passed	100.0%	IN8 754 OUT1 071
✓	GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 754 OUT1 071
Agentic - Game Backend: RTS unit counters & damage 1 passed	100.0%	IN8 806 OUT177
✓	GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 806 OUT177
Agentic - Game Backend: Safest neighbour (danger map) 1 passed	100.0%	IN36 778 OUT1 592
✓	GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN36 778 OUT1 592
Agentic - Game Backend: Scroll viewport 1 passed	100.0%	IN8 776 OUT111
✓	GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 776 OUT111
Agentic - Game Backend: Set card validity 1 passed	100.0%	IN8 755 OUT918
✓	GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 755 OUT918
Agentic - Game Backend: Shop affordability 1 passed	100.0%	IN8 735 OUT252
✓	GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 735 OUT252
Agentic - Game Backend: Sliding puzzle move 1 passed	100.0%	IN8 766 OUT409
✓	GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 766 OUT409
Agentic - Game Backend: Snake head movement 1 passed	100.0%	IN17 943 OUT509
✓	GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN17 943 OUT509
Agentic - Game Backend: Snake tick 1 failed	0.0%	IN18 219 OUT753
×	GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: Tick [1,0;0,0] R food(9,9) got "2,0" expected "2,0;1,0" Values: r.ResultJson = "\"2,0\"" exp = "\"2,0;1,0\""	Failed	IN18 219 OUT753
Agentic - Game Backend: Snap to grid 1 passed	100.0%	IN18 473 OUT662
✓	GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN18 473 OUT662
Agentic - Game Backend: Sokoban move/push legality 1 passed	100.0%	IN8 854 OUT7 390
✓	GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 854 OUT7 390
Agentic - Game Backend: Solitaire tableau move 1 passed	100.0%	IN8 793 OUT1 941
✓	GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 793 OUT1 941
Agentic - Game Backend: Spawn scheduler count 1 passed	100.0%	IN8 750 OUT2 528
✓	GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 750 OUT2 528
Agentic - Game Backend: Spawn-wave alive count 1 passed	100.0%	IN8 762 OUT139
✓	GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 762 OUT139
Agentic - Game Backend: Stat level scaling 1 passed	100.0%	IN8 754 OUT65
✓	GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 754 OUT65
Agentic - Game Backend: Steering seek (greedy pursuit) 1 passed	100.0%	IN17 795 OUT383
✓	GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN17 795 OUT383
Agentic - Game Backend: Tactical movement range 1 failed	0.0%	IN38 642 OUT1 962
×	GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Reachable did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 10:26: Expected '{' after the struct name. Structs are created with 'new Name { Field = value }', found '<'.	Failed	IN38 642 OUT1 962
Agentic - Game Backend: Tech tree eligibility 1 passed	100.0%	IN8 774 OUT2 627
✓	GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 774 OUT2 627
Agentic - Game Backend: Tetris line clearing 1 passed	100.0%	IN8 765 OUT429
✓	GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 765 OUT429
Agentic - Game Backend: Threat table top target 1 passed	100.0%	IN8 759 OUT3 029
✓	GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 759 OUT3 029
Agentic - Game Backend: Tic-tac-toe result 1 failed	0.0%	IN8 750 OUT224
×	GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: Winner([XX.OOOXXX]) got "O" expected "X" Values: r.ResultJson = "\"O\"" exp = "\"X\""	Failed	IN8 750 OUT224
Agentic - Game Backend: Toroidal wrap (Pac-Man) 1 passed	100.0%	IN8 786 OUT1 368
✓	GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 786 OUT1 368
Agentic - Game Backend: Tower targeting (nearest in range) 1 passed	100.0%	IN29 752 OUT2 003
✓	GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN29 752 OUT2 003
Agentic - Game Backend: Transaction ledger 1 passed	100.0%	IN8 777 OUT1 185
✓	GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 777 OUT1 185
Agentic - Game Backend: Trick-taking winner 1 passed	100.0%	IN18 162 OUT1 074
✓	GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users	Passed	IN18 162 OUT1 074
Agentic - Game Backend: Turn-based duel resolution 1 passed	100.0%	IN8 837 OUT596
✓	GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 837 OUT596
Agentic - Game Backend: Typed armor mitigation 1 passed	100.0%	IN8 761 OUT2 227
✓	GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 761 OUT2 227
Agentic - Game Backend: UI menu (wrap/clamp/paginate) 1 passed	100.0%	IN8 853 OUT224
✓	GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 853 OUT224
Agentic - Game Backend: Uno playable cards 1 passed	100.0%	IN8 789 OUT1 530
✓	GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 789 OUT1 530
Agentic - Game Backend: Velocity reflection 1 passed	100.0%	IN8 733 OUT746
✓	GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 733 OUT746
Agentic - Game Backend: Wall-slide fall 1 passed	100.0%	IN8 741 OUT83
✓	GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 741 OUT83
Agentic - Game Backend: War card compare 1 passed	100.0%	IN8 761 OUT202
✓	GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN8 761 OUT202
Agentic - Game Backend: Weighted loot table 1 passed	100.0%	IN19 296 OUT1 782
✓	GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users	Passed	IN19 296 OUT1 782
Agentic - Game Backend: Weighted terrain pathfinding 1 failed	0.0%	IN55 884 OUT8 148
×	GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users MinCost did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 25:8: Expected a variable name, found '['.	Failed	IN55 884 OUT8 148
Agentic - Game Backend: Word-wrap line count 1 failed	0.0%	IN8 746 OUT265
×	GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString() Context: LineCount([toolongword],5) got 2 expected 1 Values: r.ResultJson = "2"	Failed	IN8 746 OUT265
Agentic - Game Backend: XP to character level 1 passed	100.0%	IN8 754 OUT2 257
✓	GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users	Passed	IN8 754 OUT2 257
Agentic - Idempotency (poison-retry / no double charge) 1 passed	100.0%	IN18 788 OUT657
✓	AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query 9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN18 788 OUT657
Agentic - Irreversible Commit (measure twice) 1 passed	100.0%	IN20 121 OUT582
✓	AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query 9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN20 121 OUT582
Agentic - Least Privilege (exact grants) 1 passed	100.0%	IN14 871 OUT422
✓	AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query 6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN14 871 OUT422
Agentic - Ledger (simulated DB read-modify-write) 1 passed	100.0%	IN17 585 OUT1 632
✓	AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN17 585 OUT1 632
Agentic - Ledger variants (5 RMW cases) 5 passed	100.0%	IN101 620 OUT5 929
✓	LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query 11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN26 018 OUT1 466
✓	LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN14 036 OUT1 140
✓	LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN14 593 OUT1 311
✓	LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN13 212 OUT861
✓	LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN33 761 OUT1 151
Agentic - Lock Ordering (deadlock avoidance) 1 passed	100.0%	IN69 128 OUT2 051
✓	AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query 25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN69 128 OUT2 051
Agentic - Lock Ordering variants (5 cases) 5 passed	100.0%	IN365 598 OUT8 862
✓	LockVariantFiveResFourOps1 AI query 23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN86 313 OUT1 904
✓	LockVariantFourResThreeOps1 AI query 18 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN62 485 OUT1 545
✓	LockVariantHeavyReuse1 AI query 23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN86 645 OUT1 934
✓	LockVariantReverseListed1 AI query 24 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN40 215 OUT1 818
✓	LockVariantThreeLockOp1 AI query 20 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN89 940 OUT1 661
Agentic - Migration (cascading rollback + reapply) 1 passed	100.0%	IN34 078 OUT1 035
✓	AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query 11 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN34 078 OUT1 035
Agentic - Requirements (detect contradiction) 1 passed	100.0%	IN7 569 OUT483
✓	AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query 2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN7 569 OUT483
Agentic - Retry/Circuit-breaker (transient vs hard-down) 1 passed	100.0%	IN35 542 OUT864
✓	AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query 10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN35 542 OUT864
Agentic - Saga (compensation / distributed rollback) 1 passed	100.0%	IN21 437 OUT429
✓	AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN21 437 OUT429
Agentic - Saga variants (5 compensation cases) 5 passed	100.0%	IN94 880 OUT2 259
✓	SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN13 601 OUT316
✓	SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN13 647 OUT269
✓	SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN20 208 OUT530
✓	SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN20 218 OUT500
✓	SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query 8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN27 206 OUT644
Agentic - Schema Compat (expand-contract) 1 passed	100.0%	IN21 844 OUT570
✓	AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN21 844 OUT570
Agentic - Scratchpad (self-repair, Dyck, tool-decision) 4 passed3 failed	57.1%	IN250 422 OUT14 899
×	AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query 1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: passed Context: Agent failed to produce a passing implementation within 10 attempts. Values: passed = false	Failed	IN21 882 OUT1 193
×	AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query 2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: got != "" Context: Model did not provide an ANSWER marker. Values: got = ""	Failed	IN32 173 OUT712
✓	ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN21 004 OUT122
✓	ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN21 392 OUT610
×	ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: answer != "" Context: ToolDecisionFibonacciModulo: missing ANSWER marker. Values: answer = ""	Failed	IN21 675 OUT1 080
✓	ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query 2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users	Passed	IN47 567 OUT8 267
✓	ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query 6 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN84 729 OUT2 915
Agentic - Secret Rotation (zero-downtime ordering) 1 passed	100.0%	IN18 016 OUT719
✓	AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query 5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN18 016 OUT719
Agentic - StatefulOps (incident recovery, concurrency replan) 2 passed	100.0%	IN65 514 OUT1 847
✓	AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query 9 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users	Passed	IN33 366 OUT977
✓	AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query 8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users	Passed	IN32 148 OUT870
Agentic - Test Authoring (write a test that catches the bug) 1 passed	100.0%	IN16 933 OUT523
✓	AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query 5 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users	Passed	IN16 933 OUT523
Agentic - Topological Planning (deps + prereq) 1 passed	100.0%	IN33 787 OUT986
✓	AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users	Passed	IN33 787 OUT986
Algorithms - 0/1 Knapsack (dynamic programming) 1 passed	100.0%	IN37 824 OUT1 576
✓	KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN37 824 OUT1 576
Algorithms - 1D elastic collision 1 passed	100.0%	IN8 808 OUT145
✓	ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 808 OUT145
Algorithms - All-pairs shortest path pair 1 passed	100.0%	IN35 849 OUT5 941
✓	FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN35 849 OUT5 941
Algorithms - Balanced brackets (stack) 1 passed	100.0%	IN8 739 OUT199
✓	ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 739 OUT199
Algorithms - Bellman-Ford shortest path (neg weights/cycles) 1 failed	0.0%	IN51 790 OUT6 982
×	BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users ShortestPath did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 3:12: Expected a field name, found 'int'.	Failed	IN51 790 OUT6 982
Algorithms - Binary Gray code 1 passed	100.0%	IN8 712 OUT22
✓	GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 712 OUT22
Algorithms - Bipartite graph check 1 passed	100.0%	IN18 047 OUT964
✓	BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 047 OUT964
Algorithms - Bit tricks (popcount/pow2/lowbit) 1 passed	100.0%	IN8 786 OUT1 177
✓	BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 786 OUT1 177
Algorithms - Caesar cipher 1 passed	100.0%	IN8 735 OUT1 502
✓	CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 735 OUT1 502
Algorithms - Catalan number 1 passed	100.0%	IN8 757 OUT89
✓	CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 757 OUT89
Algorithms - Chinese remainder theorem 1 passed	100.0%	IN20 993 OUT3 731
✓	CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN20 993 OUT3 731
Algorithms - Circle-circle MTV 1 passed	100.0%	IN17 987 OUT398
✓	CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN17 987 OUT398
Algorithms - Coin change combinations 1 passed	100.0%	IN8 742 OUT200
✓	CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 742 OUT200
Algorithms - Collatz step count 1 passed	100.0%	IN8 738 OUT99
✓	CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 738 OUT99
Algorithms - Connected components (union-find) 1 passed	100.0%	IN47 531 OUT5 166
✓	UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN47 531 OUT5 166
Algorithms - Container with most water 1 passed	100.0%	IN8 746 OUT2 505
✓	ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 746 OUT2 505
Algorithms - Count anagram groups 1 passed	100.0%	IN8 729 OUT1 368
✓	AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 729 OUT1 368
Algorithms - Count grid color regions 1 passed	100.0%	IN8 749 OUT1 898
✓	CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 749 OUT1 898
Algorithms - Count islands 1 passed	100.0%	IN20 018 OUT2 918
✓	CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN20 018 OUT2 918
Algorithms - Count overlapping occurrences 1 passed	100.0%	IN8 735 OUT99
✓	RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 735 OUT99
Algorithms - Count primes (Sieve of Eratosthenes) 1 passed	100.0%	IN18 530 OUT1 257
✓	CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN18 530 OUT1 257
Algorithms - Cumulative distribution 1 passed	100.0%	IN8 736 OUT500
✓	CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 736 OUT500
Algorithms - Dijkstra shortest path (weighted graph) 1 passed	100.0%	IN36 868 OUT5 976
✓	DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN36 868 OUT5 976
Algorithms - Directed cycle detection 1 passed	100.0%	IN27 841 OUT1 426
✓	DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN27 841 OUT1 426
Algorithms - Distinct subsequences 1 passed	100.0%	IN8 731 OUT196
✓	DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users	Passed	IN8 731 OUT196
Algorithms - Divisor count 1 passed	100.0%	IN8 719 OUT117
✓	DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 719 OUT117
Algorithms - Downsample by block average 1 passed	100.0%	IN8 760 OUT178
✓	ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 760 OUT178
Algorithms - Dutch national flag sort 1 passed	100.0%	IN8 727 OUT298
✓	DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 727 OUT298
Algorithms - Egg drop trials 1 passed	100.0%	IN8 788 OUT2 572
✓	EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 788 OUT2 572
Algorithms - Equal subset-sum partition 1 passed	100.0%	IN8 730 OUT2 830
✓	PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 730 OUT2 830
Algorithms - Euler totient 1 passed	100.0%	IN8 719 OUT1 935
✓	EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 719 OUT1 935
Algorithms - Eulerian path existence 1 passed	100.0%	IN21 109 OUT4 129
✓	EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN21 109 OUT4 129
Algorithms - Extended Euclidean 1 passed	100.0%	IN17 904 OUT399
✓	ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN17 904 OUT399
Algorithms - Format duration H:MM:SS 1 passed	100.0%	IN17 675 OUT255
✓	FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN17 675 OUT255
Algorithms - GCD and LCM 1 passed	100.0%	IN8 723 OUT127
✓	GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 723 OUT127
Algorithms - Grid transpose 1 failed	0.0%	IN8 754 OUT2 937
×	TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Transpose threw: String index 6 is out of range. The string has 6 character(s).	Failed	IN8 754 OUT2 937
Algorithms - House robber (max non-adjacent sum) 1 passed	100.0%	IN8 718 OUT221
✓	HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 718 OUT221
Algorithms - Integer linear interpolation 1 passed	100.0%	IN8 752 OUT35
✓	LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 752 OUT35
Algorithms - Integer square root 1 passed	100.0%	IN8 734 OUT3 323
✓	IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 734 OUT3 323
Algorithms - Integer to base-N string 1 passed	100.0%	IN26 978 OUT607
✓	BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN26 978 OUT607
Algorithms - Integer to Roman numeral 1 passed	100.0%	IN8 769 OUT209
✓	RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 769 OUT209
Algorithms - Josephus survivor 1 passed	100.0%	IN8 745 OUT4 957
✓	JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 745 OUT4 957
Algorithms - Jump game (min jumps) 1 passed	100.0%	IN8 753 OUT282
✓	JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 753 OUT282
Algorithms - Kth largest element 1 passed	100.0%	IN8 743 OUT2 134
✓	KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 743 OUT2 134
Algorithms - Least-squares line fit 1 passed	100.0%	IN8 785 OUT217
✓	LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 785 OUT217
Algorithms - Levenshtein edit distance (DP) 1 passed	100.0%	IN8 713 OUT417
✓	EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 713 OUT417
Algorithms - Log-scale bucket index 1 passed	100.0%	IN23 414 OUT5 885
✓	LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN23 414 OUT5 885
Algorithms - Logarithmic bin index 1 passed	100.0%	IN8 767 OUT111
✓	LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 767 OUT111
Algorithms - Longest common prefix 1 passed	100.0%	IN8 726 OUT231
✓	CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users	Passed	IN8 726 OUT231
Algorithms - Longest common subsequence 1 passed	100.0%	IN8 711 OUT299
✓	LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 711 OUT299
Algorithms - Longest common substring 1 passed	100.0%	IN8 728 OUT502
✓	CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 728 OUT502
Algorithms - Longest increasing subsequence (DP) 1 failed	0.0%	IN37 397 OUT1 353
×	LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users LIS did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 3:14: Operator '==' cannot compare 'string' and 'null'. Only numbers, bools, chars, strings, date/time values and recursively comparable containers/structs can be compared. 3:36: Type 'string' has no supported method named 'Equals'. 9:21: Type 'List<string>' has no fields. Only struct values support '.Length'. 12:20: Type 'string' has no supported method named 'Equals'.	Failed	IN37 397 OUT1 353
Algorithms - Longest palindromic substring 1 passed	100.0%	IN30 785 OUT2 596
✓	LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN30 785 OUT2 596
Algorithms - Longest repeated substring 1 passed	100.0%	IN8 728 OUT1 586
✓	LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 728 OUT1 586
Algorithms - Lower bound (binary search) 1 passed	100.0%	IN8 756 OUT168
✓	LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 756 OUT168
Algorithms - Luhn checksum validation 1 passed	100.0%	IN8 757 OUT1 479
✓	LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 757 OUT1 479
Algorithms - Majority element 1 passed	100.0%	IN18 649 OUT1 412
✓	MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 649 OUT1 412
Algorithms - Matrix chain multiplication 1 failed	0.0%	IN37 577 OUT1 445
×	MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users MinMult did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 31:30: Operator '+' cannot be applied to 'int' and 'string'. It supports numbers and string/char concatenation. 32:30: Operator '+' cannot be applied to 'int' and 'string'. It supports numbers and string/char concatenation. 46:27: Operator '+' cannot be applied to 'int' and 'string'. It supports numbers and string/char concatenation. 53:25: Operator '+' cannot be applied to 'string' and 'int'. It supports numbers and string/char concatenation.	Failed	IN37 577 OUT1 445
Algorithms - Maximum subarray sum (Kadane) 1 passed	100.0%	IN18 792 OUT1 374
✓	MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN18 792 OUT1 374
Algorithms - Merge K sorted lists 1 passed	100.0%	IN8 739 OUT1 093
✓	MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 739 OUT1 093
Algorithms - Merge overlapping intervals 1 passed	100.0%	IN8 771 OUT638
✓	MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 771 OUT638
Algorithms - Min-max normalization 1 passed	100.0%	IN8 748 OUT1 087
✓	MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 748 OUT1 087
Algorithms - Minimum coin change (DP) 1 passed	100.0%	IN27 208 OUT1 396
✓	CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN27 208 OUT1 396
Algorithms - Minimum knight moves (BFS) 1 broken	0.0%	IN8 777 OUT9 819
!	KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users AI output was interrupted/broken	Broken	IN8 777 OUT9 819
Algorithms - Minimum path sum (grid DP) 1 passed	100.0%	IN27 914 OUT1 332
✓	MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN27 914 OUT1 332
Algorithms - Minimum spanning tree (Kruskal) 1 failed	0.0%	IN39 049 OUT2 497
×	KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users MstWeight did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 8:17: Expected a parameter name, found '['.	Failed	IN39 049 OUT2 497
Algorithms - Minimum window substring 1 passed	100.0%	IN8 760 OUT509
✓	MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users	Passed	IN8 760 OUT509
Algorithms - Modular exponentiation (fast pow) 1 passed	100.0%	IN8 771 OUT139
✓	ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 771 OUT139
Algorithms - Modular inverse (extended Euclid) 1 passed	100.0%	IN8 759 OUT198
✓	ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN8 759 OUT198
Algorithms - Moving average 1 passed	100.0%	IN8 750 OUT1 214
✓	MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 750 OUT1 214
Algorithms - Next greater element 1 failed	0.0%	IN40 803 OUT2 463
×	NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users NextGreater did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 9:11: Expected a variable name, found '['.	Failed	IN40 803 OUT2 463
Algorithms - Next lexicographic permutation 1 failed	0.0%	IN37 622 OUT1 604
×	NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users NextPermutation did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 4:22: Type 'List<string>' has no fields. Only struct values support '.Length'.	Failed	IN37 622 OUT1 604
Algorithms - Nth Fibonacci mod m 1 failed	0.0%	IN18 918 OUT1 662
×	FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users Assertion expression evaluated to false: r.ResultJson == RefFibMod(n, mod).ToString() Context: FibMod(20,100) got 8 Values: r.ResultJson = "8"	Failed	IN18 918 OUT1 662
Algorithms - Overlapping substring count 1 passed	100.0%	IN8 725 OUT168
✓	SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 725 OUT168
Algorithms - Palindrome & anagram checks 1 passed	100.0%	IN24 187 OUT6 995
✓	PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN24 187 OUT6 995
Algorithms - Pascal triangle row 1 failed	0.0%	IN18 729 OUT1 402
×	PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp Context: Row(4) got "1 1" expected "1 4 6 4 1" Values: r.ResultJson = "\"1 1\"" exp = "\"1 4 6 4 1\""	Failed	IN18 729 OUT1 402
Algorithms - Point-to-segment distance 1 passed	100.0%	IN17 866 OUT524
✓	PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN17 866 OUT524
Algorithms - Prime factorization 1 passed	100.0%	IN8 744 OUT315
✓	PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 744 OUT315
Algorithms - Product of array except self 1 passed	100.0%	IN8 750 OUT1 349
✓	ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users	Passed	IN8 750 OUT1 349
Algorithms - Quadratic Bezier point 1 passed	100.0%	IN17 908 OUT389
✓	BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users	Passed	IN17 908 OUT389
Algorithms - Range sum via prefix sums 1 failed	0.0%	IN8 732 OUT73
×	PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users RangeSum threw: List index 1 is out of range. The list has 1 element(s).	Failed	IN8 732 OUT73
Algorithms - Reverse words & Hamming distance 1 passed	100.0%	IN8 736 OUT1 128

Agentic - Blue-Green Deploy (health-gated cutover) 1 passed

100.0%

IN14 575 OUT228

✓

AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query 4 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN14 575 OUT228

Agentic - Bugfix (root-cause vs regression) 1 passed

100.0%

IN20 113 OUT256

✓

AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query 1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN20 113 OUT256

Agentic - Cache Invalidation (exact set) 1 passed

100.0%

IN24 056 OUT737

✓

AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query 7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN24 056 OUT737

Agentic - Canary (metric-gated release) 1 passed

100.0%

IN45 087 OUT1 106

✓

AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN45 087 OUT1 106

Agentic - Canary variants (5 metric cases) 5 passed

100.0%

IN187 771 OUT5 593

✓

CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN31 734 OUT1 105

✓

CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN36 466 OUT1 073

✓

CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN31 463 OUT854

✓

CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query 14 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN56 469 OUT1 620

✓

CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN31 639 OUT941

Agentic - Code Review (find injected defect) 1 passed

100.0%

IN13 557 OUT939

✓

AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN13 557 OUT939

Agentic - Config Drift (remediate exact) 1 passed

100.0%

IN12 198 OUT425

✓

AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query 6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN12 198 OUT425

Agentic - Constraints (multi-section satisfaction) 1 passed

100.0%

IN11 676 OUT665

✓

AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN11 676 OUT665

Agentic - Constraints variants (5 cases) 5 passed

100.0%

IN53 476 OUT2 394

✓

ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN10 610 OUT673

✓

ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query 4 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN12 698 OUT380

✓

ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN9 969 OUT397

✓

ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN10 257 OUT581

✓

ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query 3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN9 942 OUT363

Agentic - Data Reconciliation (per-field precedence) 1 passed

100.0%

IN14 564 OUT1 030

✓

AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN14 564 OUT1 030

Agentic - Diagnosis (root cause under budget) 1 passed

100.0%

IN12 607 OUT676

✓

AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN12 607 OUT676

Agentic - Diagnosis variants (5 cases) 5 passed

100.0%

IN106 168 OUT5 832

✓

DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN15 512 OUT916

✓

DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN32 868 OUT1 417

✓

DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query 10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN15 340 OUT830

✓

DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN16 265 OUT1 129

✓

DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query 16 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN26 183 OUT1 540

Agentic - ETL Repair (root-cause in stage 2) 1 passed

100.0%

IN33 759 OUT1 117

✓

AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query 12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN33 759 OUT1 117

Agentic - Feature Flag (staged rollout + kill-switch) 1 passed

100.0%

IN24 381 OUT771

✓

AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN24 381 OUT771

Agentic - Game Backend: 1D bouncing ball 1 passed

100.0%

IN8 783 OUT2 977

✓

GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 783 OUT2 977

Agentic - Game Backend: 1D swept anti-tunneling 1 passed

100.0%

IN8 794 OUT235

✓

GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 794 OUT235

Agentic - Game Backend: 2048 left-move (slide + merge) 1 passed

100.0%

IN8 803 OUT1 616

✓

GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 803 OUT1 616

Agentic - Game Backend: 8-direction from vector 1 passed

100.0%

IN8 810 OUT266

✓

GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 810 OUT266

Agentic - Game Backend: AABB collision resolution (MTV) 1 failed

0.0%

IN28 008 OUT1 053

×

GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: ResolveAabb got "2,4" expected "4,2" Values: r.ResultJson = "\"2,4\"" exp = "\"4,2\""

Failed

IN28 008 OUT1 053

Agentic - Game Backend: AABB containment 1 passed

100.0%

IN8 739 OUT1 246

✓

GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 739 OUT1 246

Agentic - Game Backend: Acceleration to max speed 1 passed

100.0%

IN8 741 OUT118

✓

GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 741 OUT118

Agentic - Game Backend: Achievement thresholds 1 passed

100.0%

IN8 733 OUT789

✓

GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 733 OUT789

Agentic - Game Backend: Active status effects 1 passed

100.0%

IN8 755 OUT1 194

✓

GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 755 OUT1 194

Agentic - Game Backend: Animation frame at time 1 passed

100.0%

IN8 773 OUT90

✓

GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 773 OUT90

Agentic - Game Backend: Army upkeep income 1 passed

100.0%

IN8 721 OUT790

✓

GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 721 OUT790

Agentic - Game Backend: Axis-aligned path follow 1 passed

100.0%

IN20 343 OUT3 116

✓

GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN20 343 OUT3 116

Agentic - Game Backend: Battleship shot resolution 1 passed

100.0%

IN33 633 OUT4 108

✓

GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN33 633 OUT4 108

Agentic - Game Backend: Bounce restitution 1 passed

100.0%

IN8 760 OUT77

✓

GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 760 OUT77

Agentic - Game Backend: Bresenham line rasterization 1 failed

0.0%

IN43 539 OUT3 290

×

GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Line (0,0)->(2,0) got "0,01,02,0" expected "0,0;1,0;2,0" Values: r.ResultJson = "\"0,01,02,0\"" exp = "\"0,0;1,0;2,0\""

Failed

IN43 539 OUT3 290

Agentic - Game Backend: Buff/debuff stacking 1 failed

0.0%

IN37 154 OUT1 162

×

GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

ActivePower did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 22:43: Expected '{' after the struct name. Structs are created with 'new Name { Field = value }', found '('.

Failed

IN37 154 OUT1 162

Agentic - Game Backend: Build queue completions 1 passed

100.0%

IN8 754 OUT1 048

✓

GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 754 OUT1 048

Agentic - Game Backend: Camera-relative movement 1 passed

100.0%

IN18 009 OUT352

✓

GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 009 OUT352

Agentic - Game Backend: Card hand sort 1 failed

0.0%

IN38 420 OUT2 052

×

GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

SortHand did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 26:5: Expected an expression, found 'struct'.

Failed

IN38 420 OUT2 052

Agentic - Game Backend: Card-game rules (blackjack/deck/compare) 1 passed

100.0%

IN18 577 OUT1 242

✓

GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 577 OUT1 242

Agentic - Game Backend: Chart scaling & histogram bins 1 passed

100.0%

IN8 826 OUT1 139

✓

GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 826 OUT1 139

Agentic - Game Backend: Checkers captures 1 passed

100.0%

IN8 796 OUT3 318

✓

GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 796 OUT3 318

Agentic - Game Backend: Collision primitives (circle/aabb/segment) 1 passed

100.0%

IN18 782 OUT1 336

✓

GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 782 OUT1 336

Agentic - Game Backend: Combat turn order (initiative) 1 failed

0.0%

IN37 526 OUT1 414

×

GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

TurnOrder did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 46:20: Unknown variable 'initiatives'. 47:21: Unknown variable 'initiatives'. 52:20: Unknown variable 'names'. 53:21: Unknown variable 'names'.

Failed

IN37 526 OUT1 414

Agentic - Game Backend: Combo scoring (rising multiplier) 1 passed

100.0%

IN8 758 OUT1 551

✓

GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 758 OUT1 551

Agentic - Game Backend: Control-point capture 1 passed

100.0%

IN8 771 OUT216

✓

GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 771 OUT216

Agentic - Game Backend: Controls & Inventory (spec-driven function battery) 1 passed

100.0%

IN18 176 OUT511

✓

GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 176 OUT511

Agentic - Game Backend: Cooldowns & resource regen 1 failed

0.0%

IN8 829 OUT2 843

×

GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

CooldownRemaining did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: Entry point 'CooldownRemaining' was not found. Available functions: AskAi, RunScript, SkipTest, __CreateInMemoryDatabase, __DbExecute, __DbExecuteScript, __DbInsert, __DbQueryFirstJson, __DbQueryJson, __DbScalarJson, AbilityReady.

Failed

IN8 829 OUT2 843

Agentic - Game Backend: Coyote-time jump 1 passed

100.0%

IN17 629 OUT141

✓

GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN17 629 OUT141

Agentic - Game Backend: Crafting capacity 1 failed

0.0%

IN38 486 OUT2 131

×

GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

MaxCrafts did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 68:19: Expected ')' to close the parenthesized expression, found '='.

Failed

IN38 486 OUT2 131

Agentic - Game Backend: Cribbage fifteens 1 passed

100.0%

IN8 751 OUT1 790

✓

GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 751 OUT1 790

Agentic - Game Backend: Critical-hit damage 1 passed

100.0%

IN8 741 OUT465

✓

GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 741 OUT465

Agentic - Game Backend: Damage over time 1 passed

100.0%

IN8 767 OUT164

✓

GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 767 OUT164

Agentic - Game Backend: Day/night cycle phase 1 passed

100.0%

IN8 753 OUT1 392

✓

GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 753 OUT1 392

Agentic - Game Backend: Dice notation max roll 1 passed

100.0%

IN8 768 OUT227

✓

GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 768 OUT227

Agentic - Game Backend: Enemy AI state machine 1 passed

100.0%

IN18 211 OUT844

✓

GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN18 211 OUT844

Agentic - Game Backend: Equipment slot validation 1 passed

100.0%

IN8 768 OUT178

✓

GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 768 OUT178

Agentic - Game Backend: Flood-fill region size 1 passed

100.0%

IN38 317 OUT1 908

✓

GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN38 317 OUT1 908

Agentic - Game Backend: Fog of war visibility 1 passed

100.0%

IN8 796 OUT1 173

✓

GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 796 OUT1 173

Agentic - Game Backend: Fog reveal (Chebyshev radius) 1 failed

0.0%

IN37 500 OUT1 468

×

GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Revealed did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 10:11: Expected a variable name, found '['.

Failed

IN37 500 OUT1 468

Agentic - Game Backend: Friction stop frame 1 passed

100.0%

IN8 749 OUT1 430

✓

GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 749 OUT1 430

Agentic - Game Backend: Game of Life (code-gen vs oracle) 1 failed

0.0%

IN8 808 OUT1 536

×

GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

candidate did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 1:24: Char literals must contain exactly one character and close with '''. 10:5: Char literals must contain exactly one character and close with '''. 15:8: Char literals must contain exactly one character and close with '''. 17:1: Unexpected character '`'. 17:2: Unexpected character '`'. 17:3: Unexpected character '`'. 69:1: Unexpected character '`'. 69:2: Unexpected character '`'. 69:3: Unexpected character '`'. 71:26: Unexpected character '#'. 74:28: Unexpected character '`'. 74:36: Unexpected character '`'. 75:29: Unexpected character '`'. 75:32: Unexpected character '`'. 79:84: Unexpected character '`'. 79:92: Unexpected character '`'. 79:104: Unexpected character '`'. 79:109: Unexpected character '`'. 79:115: Unexpected character '`'. 79:136: Unexpected character '`'. 79:150: Unexpected character '`'. 79:155: Unexpected character '`'. 79:176: Unexpected character '`'. 79:180: Unexpected character '`'. 82:44: Unexpected character '→'. 82:67: Unexpected character '✓'. 83:52: Unexpected character '→'. 83:75: Unexpected character '✓'. 85:49: Unexpected character '#'. 87:50: Unexpected character '\'. 89:8: Char literals must contain exactly one character and close with '''. 93:44: Unexpected character '`'. 93:58: Unexpected character '`'. 93:128: Unexpected character '`'. 93:142: Unexpected character '`'. 93:270: Unexpected character '`'. 93:284: Unexpected character '`'. 97:1: Unexpected character '`'. 97:2: Unexpected character '`'. 97:3: Unexpected character '`'. 149:1: Unexpected character '`'. 149:2: Unexpected character '`'. 149:3: Unexpected character '`'. 153:7: Char literals must contain exactly one character and close with '''. 155:1: Unexpected character '`'. 155:2: Unexpected character '`'. 155:3: Unexpected character '`'. 207:1: Unexpected character '`'. 207:2: Unexpected character '`'. 207:3: Unexpected character '`'.

Failed

IN8 808 OUT1 536

Agentic - Game Backend: Greedy coin change 1 passed

100.0%

IN18 145 OUT701

✓

GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 145 OUT701

Agentic - Game Backend: Grid inventory & stacking (Diablo-style) 1 passed

100.0%

IN8 883 OUT1 063

✓

GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 883 OUT1 063

Agentic - Game Backend: Grid pathfinding (shortest path, BFS) 1 passed

100.0%

IN32 082 OUT3 543

✓

GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.3 AI queries 0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN32 082 OUT3 543

Agentic - Game Backend: Grid raycast 1 passed

100.0%

IN17 981 OUT613

✓

GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN17 981 OUT613

Agentic - Game Backend: Health bar segments 1 failed

0.0%

IN8 807 OUT2 497

×

GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

RenderBar did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: Entry point 'RenderBar' was not found. Available functions: AskAi, RunScript, SkipTest, __CreateInMemoryDatabase, __DbExecute, __DbExecuteScript, __DbInsert, __DbQueryFirstJson, __DbQueryJson, __DbScalarJson, FilledSegments.

Failed

IN8 807 OUT2 497

Agentic - Game Backend: High-score table insert 1 passed

100.0%

IN8 761 OUT3 705

✓

GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 761 OUT3 705

Agentic - Game Backend: Hit chance 1 passed

100.0%

IN8 745 OUT78

✓

GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 745 OUT78

Agentic - Game Backend: Inventory encumbrance 1 passed

100.0%

IN8 742 OUT767

✓

GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 742 OUT767

Agentic - Game Backend: Inventory first-fit placement 1 passed

100.0%

IN31 563 OUT2 916

✓

GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN31 563 OUT2 916

Agentic - Game Backend: Inventory stack split 1 passed

100.0%

IN8 757 OUT65

✓

GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 757 OUT65

Agentic - Game Backend: Isometric projection 1 passed

100.0%

IN17 854 OUT260

✓

GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN17 854 OUT260

Agentic - Game Backend: Jump buffer 1 failed

0.0%

IN8 763 OUT155

×

GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Jumped([PL],0) got true expected false Values: r.ResultJson = "true" exp = "false"

Failed

IN8 763 OUT155

Agentic - Game Backend: Knockback with arena clamp 1 passed

100.0%

IN8 802 OUT865

✓

GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 802 OUT865

Agentic - Game Backend: Leaderboard rank 1 failed

0.0%

IN38 337 OUT1 323

×

GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Rank did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 6:15: Expected a variable name, found '['.

Failed

IN38 337 OUT1 323

Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over) 1 passed

100.0%

IN8 886 OUT210

✓

GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 886 OUT210

Agentic - Game Backend: Lifesteal healing 1 passed

100.0%

IN8 746 OUT1 026

✓

GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 746 OUT1 026

Agentic - Game Backend: Lights Out toggle 1 passed

100.0%

IN20 321 OUT3 019

✓

GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN20 321 OUT3 019

Agentic - Game Backend: Match-3 cascade 1 passed

100.0%

IN8 820 OUT4 249

✓

GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN8 820 OUT4 249

Agentic - Game Backend: Match-3 detection 1 failed

0.0%

IN38 079 OUT1 736

×

GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

CountMatched did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 4:10: Expected ';' after the variable declaration, found ','.

Failed

IN38 079 OUT1 736

Agentic - Game Backend: Match-3 swap validity 1 passed

100.0%

IN18 379 OUT1 197

✓

GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN18 379 OUT1 197

Agentic - Game Backend: Memory match pairs 1 passed

100.0%

IN8 747 OUT1 039

✓

GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 747 OUT1 039

Agentic - Game Backend: Mercy invincibility (i-frames) 1 passed

100.0%

IN17 908 OUT413

✓

GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN17 908 OUT413

Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal) 1 passed

100.0%

IN18 671 OUT1 644

✓

GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.2 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN18 671 OUT1 644

Agentic - Game Backend: Minimap downsample 1 passed

100.0%

IN8 778 OUT1 514

✓

GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 778 OUT1 514

Agentic - Game Backend: Minimap marker position 1 passed

100.0%

IN18 677 OUT1 199

✓

GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN18 677 OUT1 199

Agentic - Game Backend: N-in-a-row win detection (Connect-4) 1 failed

0.0%

IN37 518 OUT1 423

×

GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.4 AI queries 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Winner mismatch 4x4 board=[....XXXX........] got "none" expected "X" Values: r.ResultJson = "\"none\"" exp = "\"X\""

Failed

IN37 518 OUT1 423

Agentic - Game Backend: Nonogram row validation 1 passed

100.0%

IN8 758 OUT3 695

✓

GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 758 OUT3 695

Agentic - Game Backend: OBB collision (rotating rectangles, SAT) 1 passed

100.0%

IN8 905 OUT2 155

✓

GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN8 905 OUT2 155

Agentic - Game Backend: Particle lifetime & expiry 1 passed

100.0%

IN8 840 OUT190

✓

GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 840 OUT190

Agentic - Game Backend: Party XP share 1 passed

100.0%

IN26 630 OUT279

✓

GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN26 630 OUT279

Agentic - Game Backend: Patrol waypoints (ping-pong) 1 passed

100.0%

IN8 781 OUT1 616

✓

GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 781 OUT1 616

Agentic - Game Backend: Platformer jump arc 1 passed

100.0%

IN8 772 OUT109

✓

GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 772 OUT109

Agentic - Game Backend: Point on circle (trig) 1 passed

100.0%

IN8 805 OUT1 149

✓

GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 805 OUT1 149

Agentic - Game Backend: Poker hand comparison 1 failed

0.0%

IN56 531 OUT10 005

×

GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users

Compare did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 94:27: Expected '{' after the struct name. Structs are created with 'new Name { Field = value }', found '('.

Failed

IN56 531 OUT10 005

Agentic - Game Backend: Poker hand ranking 1 passed

100.0%

IN8 824 OUT3 992

✓

GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN8 824 OUT3 992

Agentic - Game Backend: Pop-cap training 1 passed

100.0%

IN8 759 OUT91

✓

GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 759 OUT91

Agentic - Game Backend: Progress bar fill 1 passed

100.0%

IN8 741 OUT109

✓

GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 741 OUT109

Agentic - Game Backend: Projectile range (physics/trig) 1 passed

100.0%

IN17 641 OUT126

✓

GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN17 641 OUT126

Agentic - Game Backend: Projectile with ground bounce 1 passed

100.0%

IN17 788 OUT291

✓

GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN17 788 OUT291

Agentic - Game Backend: Resource gathering 1 passed

100.0%

IN8 758 OUT49

✓

GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 758 OUT49

Agentic - Game Backend: Reversi flip count 1 passed

100.0%

IN8 829 OUT410

✓

GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN8 829 OUT410

Agentic - Game Backend: Rotate 2D vector (trig) 1 passed

100.0%

IN17 727 OUT266

✓

GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN17 727 OUT266

Agentic - Game Backend: Round-robin deal 1 passed

100.0%

IN18 929 OUT1 481

✓

GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 929 OUT1 481

Agentic - Game Backend: RTS resource economy 1 passed

100.0%

IN8 754 OUT1 071

✓

GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 754 OUT1 071

Agentic - Game Backend: RTS unit counters & damage 1 passed

100.0%

IN8 806 OUT177

✓

GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 806 OUT177

Agentic - Game Backend: Safest neighbour (danger map) 1 passed

100.0%

IN36 778 OUT1 592

✓

GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN36 778 OUT1 592

Agentic - Game Backend: Scroll viewport 1 passed

100.0%

IN8 776 OUT111

✓

GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 776 OUT111

Agentic - Game Backend: Set card validity 1 passed

100.0%

IN8 755 OUT918

✓

GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 755 OUT918

Agentic - Game Backend: Shop affordability 1 passed

100.0%

IN8 735 OUT252

✓

GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 735 OUT252

Agentic - Game Backend: Sliding puzzle move 1 passed

100.0%

IN8 766 OUT409

✓

GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 766 OUT409

Agentic - Game Backend: Snake head movement 1 passed

100.0%

IN17 943 OUT509

✓

GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN17 943 OUT509

Agentic - Game Backend: Snake tick 1 failed

0.0%

IN18 219 OUT753

×

GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Tick [1,0;0,0] R food(9,9) got "2,0" expected "2,0;1,0" Values: r.ResultJson = "\"2,0\"" exp = "\"2,0;1,0\""

Failed

IN18 219 OUT753

Agentic - Game Backend: Snap to grid 1 passed

100.0%

IN18 473 OUT662

✓

GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN18 473 OUT662

Agentic - Game Backend: Sokoban move/push legality 1 passed

100.0%

IN8 854 OUT7 390

✓

GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query 0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 854 OUT7 390

Agentic - Game Backend: Solitaire tableau move 1 passed

100.0%

IN8 793 OUT1 941

✓

GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 793 OUT1 941

Agentic - Game Backend: Spawn scheduler count 1 passed

100.0%

IN8 750 OUT2 528

✓

GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 750 OUT2 528

Agentic - Game Backend: Spawn-wave alive count 1 passed

100.0%

IN8 762 OUT139

✓

GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 762 OUT139

Agentic - Game Backend: Stat level scaling 1 passed

100.0%

IN8 754 OUT65

✓

GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 754 OUT65

Agentic - Game Backend: Steering seek (greedy pursuit) 1 passed

100.0%

IN17 795 OUT383

✓

GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN17 795 OUT383

Agentic - Game Backend: Tactical movement range 1 failed

0.0%

IN38 642 OUT1 962

×

GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Reachable did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 10:26: Expected '{' after the struct name. Structs are created with 'new Name { Field = value }', found '<'.

Failed

IN38 642 OUT1 962

Agentic - Game Backend: Tech tree eligibility 1 passed

100.0%

IN8 774 OUT2 627

✓

GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 774 OUT2 627

Agentic - Game Backend: Tetris line clearing 1 passed

100.0%

IN8 765 OUT429

✓

GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 765 OUT429

Agentic - Game Backend: Threat table top target 1 passed

100.0%

IN8 759 OUT3 029

✓

GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 759 OUT3 029

Agentic - Game Backend: Tic-tac-toe result 1 failed

0.0%

IN8 750 OUT224

×

GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Winner([XX.OOOXXX]) got "O" expected "X" Values: r.ResultJson = "\"O\"" exp = "\"X\""

Failed

IN8 750 OUT224

Agentic - Game Backend: Toroidal wrap (Pac-Man) 1 passed

100.0%

IN8 786 OUT1 368

✓

GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 786 OUT1 368

Agentic - Game Backend: Tower targeting (nearest in range) 1 passed

100.0%

IN29 752 OUT2 003

✓

GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN29 752 OUT2 003

Agentic - Game Backend: Transaction ledger 1 passed

100.0%

IN8 777 OUT1 185

✓

GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 777 OUT1 185

Agentic - Game Backend: Trick-taking winner 1 passed

100.0%

IN18 162 OUT1 074

✓

GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN18 162 OUT1 074

Agentic - Game Backend: Turn-based duel resolution 1 passed

100.0%

IN8 837 OUT596

✓

GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 837 OUT596

Agentic - Game Backend: Typed armor mitigation 1 passed

100.0%

IN8 761 OUT2 227

✓

GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 761 OUT2 227

Agentic - Game Backend: UI menu (wrap/clamp/paginate) 1 passed

100.0%

IN8 853 OUT224

✓

GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 853 OUT224

Agentic - Game Backend: Uno playable cards 1 passed

100.0%

IN8 789 OUT1 530

✓

GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 789 OUT1 530

Agentic - Game Backend: Velocity reflection 1 passed

100.0%

IN8 733 OUT746

✓

GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 733 OUT746

Agentic - Game Backend: Wall-slide fall 1 passed

100.0%

IN8 741 OUT83

✓

GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 741 OUT83

Agentic - Game Backend: War card compare 1 passed

100.0%

IN8 761 OUT202

✓

GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN8 761 OUT202

Agentic - Game Backend: Weighted loot table 1 passed

100.0%

IN19 296 OUT1 782

✓

GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users

Passed

IN19 296 OUT1 782

Agentic - Game Backend: Weighted terrain pathfinding 1 failed

0.0%

IN55 884 OUT8 148

×

GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

MinCost did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 25:8: Expected a variable name, found '['.

Failed

IN55 884 OUT8 148

Agentic - Game Backend: Word-wrap line count 1 failed

0.0%

IN8 746 OUT265

×

GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp.ToString() Context: LineCount([toolongword],5) got 2 expected 1 Values: r.ResultJson = "2"

Failed

IN8 746 OUT265

Agentic - Game Backend: XP to character level 1 passed

100.0%

IN8 754 OUT2 257

✓

GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN8 754 OUT2 257

Agentic - Idempotency (poison-retry / no double charge) 1 passed

100.0%

IN18 788 OUT657

✓

AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query 9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN18 788 OUT657

Agentic - Irreversible Commit (measure twice) 1 passed

100.0%

IN20 121 OUT582

✓

AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query 9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN20 121 OUT582

Agentic - Least Privilege (exact grants) 1 passed

100.0%

IN14 871 OUT422

✓

AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query 6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN14 871 OUT422

Agentic - Ledger (simulated DB read-modify-write) 1 passed

100.0%

IN17 585 OUT1 632

✓

AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN17 585 OUT1 632

Agentic - Ledger variants (5 RMW cases) 5 passed

100.0%

IN101 620 OUT5 929

✓

LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query 11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN26 018 OUT1 466

✓

LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query 8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN14 036 OUT1 140

✓

LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN14 593 OUT1 311

✓

LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query 7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN13 212 OUT861

✓

LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN33 761 OUT1 151

Agentic - Lock Ordering (deadlock avoidance) 1 passed

100.0%

IN69 128 OUT2 051

✓

AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query 25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN69 128 OUT2 051

Agentic - Lock Ordering variants (5 cases) 5 passed

100.0%

IN365 598 OUT8 862

✓

LockVariantFiveResFourOps1 AI query 23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN86 313 OUT1 904

✓

LockVariantFourResThreeOps1 AI query 18 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN62 485 OUT1 545

✓

LockVariantHeavyReuse1 AI query 23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN86 645 OUT1 934

✓

LockVariantReverseListed1 AI query 24 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN40 215 OUT1 818

✓

LockVariantThreeLockOp1 AI query 20 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN89 940 OUT1 661

Agentic - Migration (cascading rollback + reapply) 1 passed

100.0%

IN34 078 OUT1 035

✓

AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query 11 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN34 078 OUT1 035

Agentic - Requirements (detect contradiction) 1 passed

100.0%

IN7 569 OUT483

✓

AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query 2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN7 569 OUT483

Agentic - Retry/Circuit-breaker (transient vs hard-down) 1 passed

100.0%

IN35 542 OUT864

✓

AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query 10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN35 542 OUT864

Agentic - Saga (compensation / distributed rollback) 1 passed

100.0%

IN21 437 OUT429

✓

AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN21 437 OUT429

Agentic - Saga variants (5 compensation cases) 5 passed

100.0%

IN94 880 OUT2 259

✓

SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN13 601 OUT316

✓

SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query 4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN13 647 OUT269

✓

SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN20 208 OUT530

✓

SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN20 218 OUT500

✓

SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query 8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN27 206 OUT644

Agentic - Schema Compat (expand-contract) 1 passed

100.0%

IN21 844 OUT570

✓

AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query 6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN21 844 OUT570

Agentic - Scratchpad (self-repair, Dyck, tool-decision) 4 passed3 failed

57.1%

IN250 422 OUT14 899

×

AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query 1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users

Assertion expression evaluated to false: passed Context: Agent failed to produce a passing implementation within 10 attempts. Values: passed = false

Failed

IN21 882 OUT1 193

×

AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query 2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users

Assertion expression evaluated to false: got != "" Context: Model did not provide an ANSWER marker. Values: got = ""

Failed

IN32 173 OUT712

✓

ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN21 004 OUT122

✓

ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN21 392 OUT610

×

ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query 1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users

Assertion expression evaluated to false: answer != "" Context: ToolDecisionFibonacciModulo: missing ANSWER marker. Values: answer = ""

Failed

IN21 675 OUT1 080

✓

ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query 2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN47 567 OUT8 267

✓

ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query 6 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN84 729 OUT2 915

Agentic - Secret Rotation (zero-downtime ordering) 1 passed

100.0%

IN18 016 OUT719

✓

AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query 5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN18 016 OUT719

Agentic - StatefulOps (incident recovery, concurrency replan) 2 passed

100.0%

IN65 514 OUT1 847

✓

AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query 9 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users

Passed

IN33 366 OUT977

✓

AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query 8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users

Passed

IN32 148 OUT870

Agentic - Test Authoring (write a test that catches the bug) 1 passed

100.0%

IN16 933 OUT523

✓

AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query 5 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users

Passed

IN16 933 OUT523

Agentic - Topological Planning (deps + prereq) 1 passed

100.0%

IN33 787 OUT986

✓

AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query 9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users

Passed

IN33 787 OUT986

Algorithms - 0/1 Knapsack (dynamic programming) 1 passed

100.0%

IN37 824 OUT1 576

✓

KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN37 824 OUT1 576

Algorithms - 1D elastic collision 1 passed

100.0%

IN8 808 OUT145

✓

ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 808 OUT145

Algorithms - All-pairs shortest path pair 1 passed

100.0%

IN35 849 OUT5 941

✓

FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN35 849 OUT5 941

Algorithms - Balanced brackets (stack) 1 passed

100.0%

IN8 739 OUT199

✓

ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 739 OUT199

Algorithms - Bellman-Ford shortest path (neg weights/cycles) 1 failed

0.0%

IN51 790 OUT6 982

×

BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

ShortestPath did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 3:12: Expected a field name, found 'int'.

Failed

IN51 790 OUT6 982

Algorithms - Binary Gray code 1 passed

100.0%

IN8 712 OUT22

✓

GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 712 OUT22

Algorithms - Bipartite graph check 1 passed

100.0%

IN18 047 OUT964

✓

BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 047 OUT964

Algorithms - Bit tricks (popcount/pow2/lowbit) 1 passed

100.0%

IN8 786 OUT1 177

✓

BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 786 OUT1 177

Algorithms - Caesar cipher 1 passed

100.0%

IN8 735 OUT1 502

✓

CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 735 OUT1 502

Algorithms - Catalan number 1 passed

100.0%

IN8 757 OUT89

✓

CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 757 OUT89

Algorithms - Chinese remainder theorem 1 passed

100.0%

IN20 993 OUT3 731

✓

CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN20 993 OUT3 731

Algorithms - Circle-circle MTV 1 passed

100.0%

IN17 987 OUT398

✓

CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN17 987 OUT398

Algorithms - Coin change combinations 1 passed

100.0%

IN8 742 OUT200

✓

CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 742 OUT200

Algorithms - Collatz step count 1 passed

100.0%

IN8 738 OUT99

✓

CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 738 OUT99

Algorithms - Connected components (union-find) 1 passed

100.0%

IN47 531 OUT5 166

✓

UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN47 531 OUT5 166

Algorithms - Container with most water 1 passed

100.0%

IN8 746 OUT2 505

✓

ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 746 OUT2 505

Algorithms - Count anagram groups 1 passed

100.0%

IN8 729 OUT1 368

✓

AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 729 OUT1 368

Algorithms - Count grid color regions 1 passed

100.0%

IN8 749 OUT1 898

✓

CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 749 OUT1 898

Algorithms - Count islands 1 passed

100.0%

IN20 018 OUT2 918

✓

CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN20 018 OUT2 918

Algorithms - Count overlapping occurrences 1 passed

100.0%

IN8 735 OUT99

✓

RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 735 OUT99

Algorithms - Count primes (Sieve of Eratosthenes) 1 passed

100.0%

IN18 530 OUT1 257

✓

CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN18 530 OUT1 257

Algorithms - Cumulative distribution 1 passed

100.0%

IN8 736 OUT500

✓

CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 736 OUT500

Algorithms - Dijkstra shortest path (weighted graph) 1 passed

100.0%

IN36 868 OUT5 976

✓

DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN36 868 OUT5 976

Algorithms - Directed cycle detection 1 passed

100.0%

IN27 841 OUT1 426

✓

DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN27 841 OUT1 426

Algorithms - Distinct subsequences 1 passed

100.0%

IN8 731 OUT196

✓

DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN8 731 OUT196

Algorithms - Divisor count 1 passed

100.0%

IN8 719 OUT117

✓

DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 719 OUT117

Algorithms - Downsample by block average 1 passed

100.0%

IN8 760 OUT178

✓

ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 760 OUT178

Algorithms - Dutch national flag sort 1 passed

100.0%

IN8 727 OUT298

✓

DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 727 OUT298

Algorithms - Egg drop trials 1 passed

100.0%

IN8 788 OUT2 572

✓

EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 788 OUT2 572

Algorithms - Equal subset-sum partition 1 passed

100.0%

IN8 730 OUT2 830

✓

PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 730 OUT2 830

Algorithms - Euler totient 1 passed

100.0%

IN8 719 OUT1 935

✓

EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 719 OUT1 935

Algorithms - Eulerian path existence 1 passed

100.0%

IN21 109 OUT4 129

✓

EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN21 109 OUT4 129

Algorithms - Extended Euclidean 1 passed

100.0%

IN17 904 OUT399

✓

ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN17 904 OUT399

Algorithms - Format duration H:MM:SS 1 passed

100.0%

IN17 675 OUT255

✓

FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN17 675 OUT255

Algorithms - GCD and LCM 1 passed

100.0%

IN8 723 OUT127

✓

GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 723 OUT127

Algorithms - Grid transpose 1 failed

0.0%

IN8 754 OUT2 937

×

TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Transpose threw: String index 6 is out of range. The string has 6 character(s).

Failed

IN8 754 OUT2 937

Algorithms - House robber (max non-adjacent sum) 1 passed

100.0%

IN8 718 OUT221

✓

HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 718 OUT221

Algorithms - Integer linear interpolation 1 passed

100.0%

IN8 752 OUT35

✓

LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 752 OUT35

Algorithms - Integer square root 1 passed

100.0%

IN8 734 OUT3 323

✓

IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 734 OUT3 323

Algorithms - Integer to base-N string 1 passed

100.0%

IN26 978 OUT607

✓

BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN26 978 OUT607

Algorithms - Integer to Roman numeral 1 passed

100.0%

IN8 769 OUT209

✓

RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 769 OUT209

Algorithms - Josephus survivor 1 passed

100.0%

IN8 745 OUT4 957

✓

JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 745 OUT4 957

Algorithms - Jump game (min jumps) 1 passed

100.0%

IN8 753 OUT282

✓

JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 753 OUT282

Algorithms - Kth largest element 1 passed

100.0%

IN8 743 OUT2 134

✓

KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 743 OUT2 134

Algorithms - Least-squares line fit 1 passed

100.0%

IN8 785 OUT217

✓

LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 785 OUT217

Algorithms - Levenshtein edit distance (DP) 1 passed

100.0%

IN8 713 OUT417

✓

EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 713 OUT417

Algorithms - Log-scale bucket index 1 passed

100.0%

IN23 414 OUT5 885

✓

LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN23 414 OUT5 885

Algorithms - Logarithmic bin index 1 passed

100.0%

IN8 767 OUT111

✓

LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 767 OUT111

Algorithms - Longest common prefix 1 passed

100.0%

IN8 726 OUT231

✓

CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 726 OUT231

Algorithms - Longest common subsequence 1 passed

100.0%

IN8 711 OUT299

✓

LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 711 OUT299

Algorithms - Longest common substring 1 passed

100.0%

IN8 728 OUT502

✓

CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 728 OUT502

Algorithms - Longest increasing subsequence (DP) 1 failed

0.0%

IN37 397 OUT1 353

×

LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

LIS did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 3:14: Operator '==' cannot compare 'string' and 'null'. Only numbers, bools, chars, strings, date/time values and recursively comparable containers/structs can be compared. 3:36: Type 'string' has no supported method named 'Equals'. 9:21: Type 'List<string>' has no fields. Only struct values support '.Length'. 12:20: Type 'string' has no supported method named 'Equals'.

Failed

IN37 397 OUT1 353

Algorithms - Longest palindromic substring 1 passed

100.0%

IN30 785 OUT2 596

✓

LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN30 785 OUT2 596

Algorithms - Longest repeated substring 1 passed

100.0%

IN8 728 OUT1 586

✓

LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 728 OUT1 586

Algorithms - Lower bound (binary search) 1 passed

100.0%

IN8 756 OUT168

✓

LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 756 OUT168

Algorithms - Luhn checksum validation 1 passed

100.0%

IN8 757 OUT1 479

✓

LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 757 OUT1 479

Algorithms - Majority element 1 passed

100.0%

IN18 649 OUT1 412

✓

MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 649 OUT1 412

Algorithms - Matrix chain multiplication 1 failed

0.0%

IN37 577 OUT1 445

×

MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

MinMult did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 31:30: Operator '+' cannot be applied to 'int' and 'string'. It supports numbers and string/char concatenation. 32:30: Operator '+' cannot be applied to 'int' and 'string'. It supports numbers and string/char concatenation. 46:27: Operator '+' cannot be applied to 'int' and 'string'. It supports numbers and string/char concatenation. 53:25: Operator '+' cannot be applied to 'string' and 'int'. It supports numbers and string/char concatenation.

Failed

IN37 577 OUT1 445

Algorithms - Maximum subarray sum (Kadane) 1 passed

100.0%

IN18 792 OUT1 374

✓

MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN18 792 OUT1 374

Algorithms - Merge K sorted lists 1 passed

100.0%

IN8 739 OUT1 093

✓

MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 739 OUT1 093

Algorithms - Merge overlapping intervals 1 passed

100.0%

IN8 771 OUT638

✓

MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 771 OUT638

Algorithms - Min-max normalization 1 passed

100.0%

IN8 748 OUT1 087

✓

MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 748 OUT1 087

Algorithms - Minimum coin change (DP) 1 passed

100.0%

IN27 208 OUT1 396

✓

CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN27 208 OUT1 396

Algorithms - Minimum knight moves (BFS) 1 broken

0.0%

IN8 777 OUT9 819

!

KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

AI output was interrupted/broken

Broken

IN8 777 OUT9 819

Algorithms - Minimum path sum (grid DP) 1 passed

100.0%

IN27 914 OUT1 332

✓

MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.3 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN27 914 OUT1 332

Algorithms - Minimum spanning tree (Kruskal) 1 failed

0.0%

IN39 049 OUT2 497

×

KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

MstWeight did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 8:17: Expected a parameter name, found '['.

Failed

IN39 049 OUT2 497

Algorithms - Minimum window substring 1 passed

100.0%

IN8 760 OUT509

✓

MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Passed

IN8 760 OUT509

Algorithms - Modular exponentiation (fast pow) 1 passed

100.0%

IN8 771 OUT139

✓

ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 771 OUT139

Algorithms - Modular inverse (extended Euclid) 1 passed

100.0%

IN8 759 OUT198

✓

ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 759 OUT198

Algorithms - Moving average 1 passed

100.0%

IN8 750 OUT1 214

✓

MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 750 OUT1 214

Algorithms - Next greater element 1 failed

0.0%

IN40 803 OUT2 463

×

NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

NextGreater did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 9:11: Expected a variable name, found '['.

Failed

IN40 803 OUT2 463

Algorithms - Next lexicographic permutation 1 failed

0.0%

IN37 622 OUT1 604

×

NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

NextPermutation did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 4:22: Type 'List<string>' has no fields. Only struct values support '.Length'.

Failed

IN37 622 OUT1 604

Algorithms - Nth Fibonacci mod m 1 failed

0.0%

IN18 918 OUT1 662

×

FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == RefFibMod(n, mod).ToString() Context: FibMod(20,100) got 8 Values: r.ResultJson = "8"

Failed

IN18 918 OUT1 662

Algorithms - Overlapping substring count 1 passed

100.0%

IN8 725 OUT168

✓

SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 725 OUT168

Algorithms - Palindrome & anagram checks 1 passed

100.0%

IN24 187 OUT6 995

✓

PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN24 187 OUT6 995

Algorithms - Pascal triangle row 1 failed

0.0%

IN18 729 OUT1 402

×

PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp Context: Row(4) got "1 1" expected "1 4 6 4 1" Values: r.ResultJson = "\"1 1\"" exp = "\"1 4 6 4 1\""

Failed

IN18 729 OUT1 402

Algorithms - Point-to-segment distance 1 passed

100.0%

IN17 866 OUT524

✓

PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN17 866 OUT524

Algorithms - Prime factorization 1 passed

100.0%

IN8 744 OUT315

✓

PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 744 OUT315

Algorithms - Product of array except self 1 passed

100.0%

IN8 750 OUT1 349

✓

ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 750 OUT1 349

Algorithms - Quadratic Bezier point 1 passed

100.0%

IN17 908 OUT389

✓

BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN17 908 OUT389

Algorithms - Range sum via prefix sums 1 failed

0.0%

IN8 732 OUT73

×

PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

RangeSum threw: List index 1 is out of range. The list has 1 element(s).

Failed

IN8 732 OUT73

Algorithms - Reverse words & Hamming distance 1 passed

100.0%

IN8 736 OUT1 128

✓

ReverseWordsCodeGenCode-gen: model implements reverse-words + Hamming; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 736 OUT1 128

Algorithms - Rod cutting 1 passed

100.0%

IN8 766 OUT2 036

✓

RodCuttingCodeGenCode-gen: model implements rod cutting; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 766 OUT2 036

Algorithms - Roman numeral to integer 1 passed

100.0%

IN8 755 OUT1 106

✓

RomanToIntCodeGenCode-gen: model implements Roman-to-int; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 755 OUT1 106

Algorithms - Rotate array right by k 1 passed

100.0%

IN19 840 OUT2 400

✓

RotateArrayCodeGenCode-gen: model implements right rotation; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN19 840 OUT2 400

Algorithms - Rotate grid 90 CW (discriminator) 1 passed

100.0%

IN21 503 OUT4 055

✓

RotateMatrixCodeGenCode-gen: model implements 90-degree CW rotation; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users

Passed

IN21 503 OUT4 055

Algorithms - RPN evaluator 1 passed

100.0%

IN8 764 OUT943

✓

RpnCodeGenCode-gen: model implements an RPN evaluator; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 764 OUT943

Algorithms - Run-length compression (if shorter) 1 failed

0.0%

IN36 682 OUT873

×

StringCompressCodeGenCode-gen: model implements string compression; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Compress did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 3:11: Operator '==' cannot compare 'string' and 'null'. Only numbers, bools, chars, strings, date/time values and recursively comparable containers/structs can be compared.

Failed

IN36 682 OUT873

Algorithms - Run-length encoding (encode/decode) 1 passed

100.0%

IN8 782 OUT344

✓

RunLengthCodeGenCode-gen: model implements RLE encode+decode; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 782 OUT344

Algorithms - Semantic versioning (compare + bump) 1 passed

100.0%

IN8 812 OUT271

✓

SemVerCodeGenCode-gen: model implements semver compare + bump; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:InstructionFollowinglevel:Easysource:Ai4Users

Passed

IN8 812 OUT271

Algorithms - Single-pile Nim (optimal play) 1 passed

100.0%

IN8 744 OUT33

✓

NimCodeGenCode-gen: model implements optimal Nim; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 744 OUT33

Algorithms - Sliding window maximum 1 passed

100.0%

IN8 736 OUT1 369

✓

SlidingWindowMaxCodeGenCode-gen: model implements sliding window maximum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 736 OUT1 369

Algorithms - Spiral matrix traversal 1 passed

100.0%

IN17 900 OUT770

✓

SpiralMatrixCodeGenCode-gen: model implements spiral traversal; graded vs exact oracle.2 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN17 900 OUT770

Algorithms - Square matrix multiply 1 passed

100.0%

IN8 737 OUT1 576

✓

MatMulCodeGenCode-gen: model implements square matrix multiply; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 737 OUT1 576

Algorithms - Strongly connected components 1 failed

0.0%

IN53 472 OUT7 659

×

SccCountCodeGenCode-gen: model implements SCC count; graded vs exact oracle.4 AI queries 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users

Count threw: The script exceeded the timeout of 30000 ms.

Failed

IN53 472 OUT7 659

Algorithms - Subset sum reachability 1 passed

100.0%

IN8 721 OUT183

✓

SubsetSumCodeGenCode-gen: model implements subset-sum reachability; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users

Passed

IN8 721 OUT183

Algorithms - Substring search (KMP) 1 passed

100.0%

IN8 743 OUT499

✓

KmpCodeGenCode-gen: model implements KMP search; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 743 OUT499

Algorithms - Sum of digits & digital root 1 passed

100.0%

IN8 722 OUT182

✓

DigitalRootCodeGenCode-gen: model implements sum-of-digits + digital root; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 722 OUT182

Algorithms - Temperature conversion 1 passed

100.0%

IN8 733 OUT2 166

✓

TemperatureCodeGenCode-gen: model implements temperature conversion; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 733 OUT2 166

Algorithms - Topological sort (lexicographically smallest) 1 passed

100.0%

IN8 774 OUT2 578

✓

TopoCodeGenCode-gen: model implements lexicographically-smallest topological sort.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 774 OUT2 578

Algorithms - Trailing zeros of factorial 1 passed

100.0%

IN8 734 OUT82

✓

TrailingZerosCodeGenCode-gen: model implements trailing zeros; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 734 OUT82

Algorithms - Trapping rain water 1 passed

100.0%

IN8 733 OUT2 380

✓

TrappingRainCodeGenCode-gen: model implements trapping rain water; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 733 OUT2 380

Algorithms - Two-sum indices 1 passed

100.0%

IN8 740 OUT869

✓

TwoSumCodeGenCode-gen: model implements two-sum; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 740 OUT869

Algorithms - Unique grid paths 1 passed

100.0%

IN8 715 OUT163

✓

UniquePathsCodeGenCode-gen: model implements unique paths; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 715 OUT163

Algorithms - Valid Sudoku unit 1 passed

100.0%

IN8 720 OUT846

✓

SudokuUnitCodeGenCode-gen: model implements Sudoku unit validation; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users

Passed

IN8 720 OUT846

Algorithms - Wildcard string matching 1 passed

100.0%

IN8 738 OUT2 059

✓

WildcardMatchCodeGenCode-gen: model implements wildcard match; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 738 OUT2 059

Algorithms - Z-function 1 passed

100.0%

IN8 739 OUT279

✓

ZFunctionCodeGenCode-gen: model implements the Z-function; graded vs exact oracle.1 AI query 0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users

Passed

IN8 739 OUT279

Database - Conditional aggregation pivot 1 passed

100.0%

IN947 OUT63

✓

PivotCountsCodeGenCode-gen: model writes the pivot query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN947 OUT63

Database - Count distinct opponents 1 passed

100.0%

IN927 OUT19

✓

DistinctCountCodeGenCode-gen: model writes the COUNT(DISTINCT) query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Easysource:Ai4Users

Passed

IN927 OUT19

Database - Date-range filter (BETWEEN) 1 passed

100.0%

IN957 OUT43

✓

DateRangeCodeGenCode-gen: model writes the date-range query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN957 OUT43

Database - EXISTS subquery (teams with matches) 1 passed

100.0%

IN934 OUT28

✓

ExistsSubqueryCodeGenCode-gen: model writes the EXISTS query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN934 OUT28

Database - JOIN player totals (two tables) 1 passed

100.0%

IN956 OUT39

✓

JoinScoresCodeGenCode-gen: model writes a JOIN + aggregate query; graded via QueryJson vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN956 OUT39

Database - LEFT JOIN anti-join (no items) 1 passed

100.0%

IN930 OUT27

✓

LeftJoinNullsCodeGenCode-gen: model writes the anti-join query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN930 OUT27

Database - Player stats SQL (multi-aggregate, QueryJson) 1 passed

100.0%

IN960 OUT36

✓

PlayerStatsCodeGenCode-gen: model writes a multi-aggregate stats query; graded by comparing its QueryJson output to the oracle query's (arbitrary shape, no struct).1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN960 OUT36

Database - Players above average score (subquery) 1 passed

100.0%

IN925 OUT26

✓

AboveAverageCodeGenCode-gen: model writes an above-average query; graded via QueryJson vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN925 OUT26

Database - Players who never played (anti-join) 1 passed

100.0%

IN922 OUT29

✓

NeverPlayedCodeGenCode-gen: model writes an anti-join query; graded via QueryJson vs oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN922 OUT29

Database - RANK window function 1 passed

100.0%

IN952 OUT27

✓

WindowRankCodeGenCode-gen: model writes the RANK() query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN952 OUT27

Database - Running total (window function) 1 passed

100.0%

IN949 OUT37

✓

RunningTotalCodeGenCode-gen: model writes the window running-total query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN949 OUT37

Database - Scoreboard SQL (players with >1 win, HAVING) 1 passed

100.0%

IN956 OUT31

✓

WinFilterCodeGenCode-gen: model writes a HAVING query; runs under try/catch with self-repair; result compared to the oracle query.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN956 OUT31

Database - Scoreboard SQL (top scorers query) 1 passed

100.0%

IN957 OUT25

✓

ScoreboardTopScorersCodeGenCode-gen: model writes a scoreboard SQL query; runs under try/catch with a self-repair loop on SQL errors; result compared to the oracle query.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN957 OUT25

Database - Self-join same-team pairs 1 passed

100.0%

IN940 OUT44

✓

SelfJoinCodeGenCode-gen: model writes the self-join query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users

Passed

IN940 OUT44

Database - Top scorer per region 1 passed

100.0%

IN942 OUT34

✓

TopPerGroupCodeGenCode-gen: model writes the per-group top query; compared to the oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN942 OUT34

Database - UPDATE restock (SQL write) 1 passed

100.0%

IN934 OUT17

✓

UpdateInventoryCodeGenCode-gen: model writes an UPDATE; applied under try/catch, resulting state compared (via QueryJson) to the oracle update's state.1 AI query 0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users

Passed

IN934 OUT17

Database - Upsert with max merge 1 passed

100.0%

IN1 983 OUT63

✓

UpsertCodeGenCode-gen: model writes the upsert; executed on a fresh DB and the table compared.2 AI queries 0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users

Passed

IN1 983 OUT63

SDLC - Acceptance criteria check 1 passed

100.0%

IN8 763 OUT5 651

✓

SdlcAcceptanceCodeGenCode-gen: model implements acceptance-criteria check; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 763 OUT5 651

SDLC - Breaking API change detection 1 passed

100.0%

IN8 760 OUT1 435

✓

SdlcApiBreakingCodeGenCode-gen: model implements API-breaking detection; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 760 OUT1 435

SDLC - Condense without losing facts 1 passed

100.0%

IN1 048 OUT55

✓

NoInfoLossCodeGenRedaction test: model condenses keeping facts; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 048 OUT55

SDLC - Config merge with override 1 passed

100.0%

IN8 754 OUT1 092

✓

ConfigMergeCodeGenCode-gen: model implements config merge; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 754 OUT1 092

SDLC - Extract fields into fixed template 1 passed

100.0%

IN1 004 OUT22

✓

FieldExtractCodeGenRedaction test: model extracts fields into the template; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 004 OUT22

SDLC - Locate conflicting requirement pair 1 failed

0.0%

IN50 654 OUT6 343

×

SdlcConstraintConflictCodeGenCode-gen: model locates the conflicting requirement pair; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users

FindConflict did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 8:19: Expected '{' to start the struct body, found 'NewConstraint'.

Failed

IN50 654 OUT6 343

SDLC - Locate redundant requirement 1 failed

0.0%

IN65 625 OUT11 705

×

SdlcRedundantRequirementCodeGenCode-gen: model locates a redundant requirement; graded vs exact oracle.4 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users

FindRedundant did not compile: SC# compilation failed inside the script runner. Compiler diagnostics: 4:23: Type 'List<string>' has no fields. Only struct values support '.Length'.

Failed

IN65 625 OUT11 705

SDLC - Mask PII to placeholder 1 passed

100.0%

IN972 OUT42

✓

MaskPiiCodeGenRedaction test: model masks PII to a placeholder; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN972 OUT42

SDLC - Modules on a dependency cycle 1 failed

0.0%

IN30 311 OUT5 599

×

DependencyCycleModulesCodeGenCode-gen: model implements dependency-cycle detection; graded vs exact oracle.3 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users

Assertion expression evaluated to false: res.ResultJson == exp Context: Cycle(A>B;B>C;C>A) got "A" expected "A,B,C" Values: res.ResultJson = "\"A\"" exp = "\"A,B,C\""

Failed

IN30 311 OUT5 599

SDLC - Remove duplicate requirements 1 passed

100.0%

IN945 OUT10

✓

DeduplicateCodeGenRedaction test: model removes duplicate requirements; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN945 OUT10

SDLC - Renumber requirements sequentially 1 passed

100.0%

IN1 012 OUT42

✓

RenumberCodeGenRedaction test: model renumbers sequentially; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 012 OUT42

SDLC - Reorganize notes into section template 1 passed

100.0%

IN1 126 OUT94

✓

RewriteSectionsCodeGenRedaction test: model reorganizes the notes; deterministic checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 126 OUT94

SDLC - Required semver bump from changelog 1 passed

100.0%

IN8 759 OUT1 243

✓

SemverBumpCodeGenCode-gen: model implements the semver-bump rule; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Easysource:Ai4Users

Passed

IN8 759 OUT1 243

SDLC - Requirement-to-test traceability 1 passed

100.0%

IN20 205 OUT2 800

✓

SdlcTraceabilityCodeGenCode-gen: model implements traceability gap; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN20 205 OUT2 800

SDLC - Requirements completeness (spec coverage gap) 1 failed

0.0%

IN18 569 OUT1 644

×

SdlcSpecCoverageCodeGenCode-gen: model implements spec-gap detection; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Assertion expression evaluated to false: r.ResultJson == exp.ToString() Context: FindGap([0-4;6-9],10) got 0 expected 5 Values: r.ResultJson = "0"

Failed

IN18 569 OUT1 644

SDLC - Requirements consistency (contradiction detection) 1 passed

100.0%

IN21 194 OUT4 333

✓

ConsistencyCodeGenCode-gen: model implements requirements-consistency checking; graded vs oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN21 194 OUT4 333

SDLC - Reshape prose into JSON template 1 passed

100.0%

IN1 014 OUT37

✓

JsonShapeCodeGenRedaction test: model reshapes prose into JSON; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN1 014 OUT37

SDLC - Reshape rows into Markdown table 1 passed

100.0%

IN968 OUT31

✓

TableFormatCodeGenRedaction test: model builds a Markdown table; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN968 OUT31

SDLC - Rewrite in imperative requirement style 1 passed

100.0%

IN991 OUT70

✓

StyleRewriteCodeGenRedaction test: model rewrites notes in imperative style; checker + self-review loop.1 AI query 0 tool callscategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN991 OUT70

SDLC - Roll up estimates over task tree 1 passed

100.0%

IN18 001 OUT704

✓

EstimateRollupCodeGenCode-gen: model implements estimate roll-up over a task tree; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN18 001 OUT704

SDLC - State-machine reachability 1 passed

100.0%

IN8 758 OUT321

✓

SdlcStateReachableCodeGenCode-gen: model implements state reachability; graded vs exact oracle.1 AI query 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN8 758 OUT321

SDLC - Ticket priority sort 1 passed

100.0%

IN19 606 OUT2 379

✓

SdlcPrioritySortCodeGenCode-gen: model implements priority sort; graded vs exact oracle.2 AI queries 0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users

Passed

IN19 606 OUT2 379

SDLC - Unify glossary term consistency 1 passed

100.0%

IN972 OUT38

✓

GlossaryCodeGenRedaction test: model unifies synonyms to the canonical term; checker + self-review loop.1 AI query 0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users

Passed

IN972 OUT38

Qwen 3.6 35B NVFP4 Instruct | Benchmark #21

SC# Coding & Agentic Benchmark by Ai4Users

Results by difficulty

Results by category