Benchmark summary
A custom Ai4Users benchmark suite for SC#, covering code generation, reasoning, and stateful agentic workflows where models use tools, observe results, adapt their actions, recover from failures, and verify the final state.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Agentic - Blue-Green Deploy (health-gated cutover)
1 passed |
100.0% |
IN22 101 OUT453 | |
| ✓ | AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query
6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN22 101 OUT453 |
Agentic - Bugfix (root-cause vs regression)
1 passed |
100.0% |
IN40 925 OUT242 | |
| ✓ | AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query
3 tool callscategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN40 925 OUT242 |
Agentic - Cache Invalidation (exact set)
1 passed |
100.0% |
IN17 000 OUT319 | |
| ✓ | AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query
7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN17 000 OUT319 |
Agentic - Canary (metric-gated release)
1 failed |
0.0% |
IN2 714 OUT474 | |
| × | AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query
0 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: prod == "build_2201"
Context: Healthy candidate build_2201 was not promoted (production is on build_2190).
Values:
prod = "build_2190" |
Failed | IN2 714 OUT474 |
Agentic - Canary variants (5 metric cases)
5 passed |
100.0% |
IN155 986 OUT2 269 | |
| ✓ | CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN29 813 OUT428 |
| ✓ | CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query
11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN27 303 OUT374 |
| ✓ | CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN25 973 OUT325 |
| ✓ | CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN43 213 OUT761 |
| ✓ | CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN29 684 OUT381 |
Agentic - Code Review (find injected defect)
1 passed |
100.0% |
IN11 455 OUT169 | |
| ✓ | AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query
4 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 455 OUT169 |
Agentic - Config Drift (remediate exact)
1 failed |
0.0% |
IN2 485 OUT23 | |
| × | AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query
0 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users Assertion expression evaluated to false: live == DesiredValue(key)
Context: Key max_connections not reconciled: live=50, desired=200.
Values:
live = "50" |
Failed | IN2 485 OUT23 |
Agentic - Constraints (multi-section satisfaction)
1 passed |
100.0% |
IN14 283 OUT410 | |
| ✓ | AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN14 283 OUT410 |
Agentic - Constraints variants (5 cases)
5 passed |
100.0% |
IN43 360 OUT893 | |
| ✓ | ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 566 OUT126 |
| ✓ | ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN7 030 OUT126 |
| ✓ | ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query
5 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN7 324 OUT213 |
| ✓ | ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 563 OUT126 |
| ✓ | ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 877 OUT302 |
Agentic - Data Reconciliation (per-field precedence)
1 passed |
100.0% |
IN21 908 OUT777 | |
| ✓ | AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN21 908 OUT777 |
Agentic - Diagnosis (root cause under budget)
1 passed |
100.0% |
IN14 100 OUT376 | |
| ✓ | AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN14 100 OUT376 |
Agentic - Diagnosis variants (5 cases)
4 passed1 failed |
80.0% |
IN329 285 OUT32 691 | |
| ✓ | DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query
1202 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN275 692 OUT30 526 |
| ✓ | DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query
23 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN14 554 OUT602 |
| ✓ | DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query
26 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 102 OUT649 |
| ✓ | DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN14 918 OUT494 |
| × | DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query
16 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users Assertion expression evaluated to false: reported == root
Context: Reported [n7] is not the root cause (expected [n4] — the unhealthy component whose dependencies are all healthy).
Values:
reported = "n7"
root = "n4" |
Failed | IN9 019 OUT420 |
Agentic - ETL Repair (root-cause in stage 2)
1 passed |
100.0% |
IN25 130 OUT405 | |
| ✓ | AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN25 130 OUT405 |
Agentic - Feature Flag (staged rollout + kill-switch)
1 passed |
100.0% |
IN27 366 OUT650 | |
| ✓ | AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN27 366 OUT650 |
Agentic - Game Backend: 1D bouncing ball
1 passed |
100.0% |
IN8 783 OUT124 | |
| ✓ | GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 783 OUT124 |
Agentic - Game Backend: 1D swept anti-tunneling
1 passed |
100.0% |
IN8 794 OUT203 | |
| ✓ | GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 794 OUT203 |
Agentic - Game Backend: 2048 left-move (slide + merge)
1 passed |
100.0% |
IN8 803 OUT369 | |
| ✓ | GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 803 OUT369 |
Agentic - Game Backend: 8-direction from vector
1 passed |
100.0% |
IN8 810 OUT236 | |
| ✓ | GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 810 OUT236 |
Agentic - Game Backend: AABB collision resolution (MTV)
1 failed |
0.0% |
IN39 752 OUT2 116 | |
| × | GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.4 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users ResolveAabb did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
44:4: Expected a name, found '-'. |
Failed | IN39 752 OUT2 116 |
Agentic - Game Backend: AABB containment
1 passed |
100.0% |
IN8 739 OUT58 | |
| ✓ | GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 739 OUT58 |
Agentic - Game Backend: Acceleration to max speed
1 passed |
100.0% |
IN8 741 OUT90 | |
| ✓ | GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 741 OUT90 |
Agentic - Game Backend: Achievement thresholds
1 passed |
100.0% |
IN8 733 OUT144 | |
| ✓ | GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 733 OUT144 |
Agentic - Game Backend: Active status effects
1 passed |
100.0% |
IN8 755 OUT181 | |
| ✓ | GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT181 |
Agentic - Game Backend: Animation frame at time
1 passed |
100.0% |
IN8 773 OUT85 | |
| ✓ | GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 773 OUT85 |
Agentic - Game Backend: Army upkeep income
1 passed |
100.0% |
IN8 721 OUT99 | |
| ✓ | GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 721 OUT99 |
Agentic - Game Backend: Axis-aligned path follow
1 passed |
100.0% |
IN17 949 OUT619 | |
| ✓ | GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN17 949 OUT619 |
Agentic - Game Backend: Battleship shot resolution
1 passed |
100.0% |
IN8 823 OUT313 | |
| ✓ | GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 823 OUT313 |
Agentic - Game Backend: Bounce restitution
1 passed |
100.0% |
IN8 760 OUT74 | |
| ✓ | GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 760 OUT74 |
Agentic - Game Backend: Bresenham line rasterization
1 passed |
100.0% |
IN8 850 OUT251 | |
| ✓ | GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 850 OUT251 |
Agentic - Game Backend: Buff/debuff stacking
1 passed |
100.0% |
IN8 786 OUT234 | |
| ✓ | GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 786 OUT234 |
Agentic - Game Backend: Build queue completions
1 passed |
100.0% |
IN8 754 OUT95 | |
| ✓ | GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 754 OUT95 |
Agentic - Game Backend: Camera-relative movement
1 passed |
100.0% |
IN26 901 OUT322 | |
| ✓ | GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.3 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN26 901 OUT322 |
Agentic - Game Backend: Card hand sort
1 passed |
100.0% |
IN8 762 OUT393 | |
| ✓ | GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 762 OUT393 |
Agentic - Game Backend: Card-game rules (blackjack/deck/compare)
1 passed |
100.0% |
IN8 937 OUT628 | |
| ✓ | GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 937 OUT628 |
Agentic - Game Backend: Chart scaling & histogram bins
1 passed |
100.0% |
IN8 826 OUT213 | |
| ✓ | GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 826 OUT213 |
Agentic - Game Backend: Checkers captures
1 passed |
100.0% |
IN17 907 OUT512 | |
| ✓ | GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 907 OUT512 |
Agentic - Game Backend: Collision primitives (circle/aabb/segment)
1 passed |
100.0% |
IN20 523 OUT1 166 | |
| ✓ | GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.1 AI query
1 tool callcategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN20 523 OUT1 166 |
Agentic - Game Backend: Combat turn order (initiative)
1 passed |
100.0% |
IN17 772 OUT568 | |
| ✓ | GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 772 OUT568 |
Agentic - Game Backend: Combo scoring (rising multiplier)
1 passed |
100.0% |
IN8 758 OUT136 | |
| ✓ | GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 758 OUT136 |
Agentic - Game Backend: Control-point capture
1 passed |
100.0% |
IN8 771 OUT192 | |
| ✓ | GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 771 OUT192 |
Agentic - Game Backend: Controls & Inventory (spec-driven function battery)
1 passed |
100.0% |
IN18 112 OUT523 | |
| ✓ | GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 112 OUT523 |
Agentic - Game Backend: Cooldowns & resource regen
1 passed |
100.0% |
IN8 829 OUT165 | |
| ✓ | GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 829 OUT165 |
Agentic - Game Backend: Coyote-time jump
1 passed |
100.0% |
IN8 750 OUT71 | |
| ✓ | GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 750 OUT71 |
Agentic - Game Backend: Crafting capacity
1 passed |
100.0% |
IN27 770 OUT1 153 | |
| ✓ | GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.3 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN27 770 OUT1 153 |
Agentic - Game Backend: Cribbage fifteens
1 passed |
100.0% |
IN8 751 OUT142 | |
| ✓ | GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 751 OUT142 |
Agentic - Game Backend: Critical-hit damage
1 passed |
100.0% |
IN8 741 OUT52 | |
| ✓ | GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 741 OUT52 |
Agentic - Game Backend: Damage over time
1 failed |
0.0% |
IN37 055 OUT1 386 | |
| × | GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.4 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users FinalHp did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
27:4: Expected a name, found '-'. |
Failed | IN37 055 OUT1 386 |
Agentic - Game Backend: Day/night cycle phase
1 passed |
100.0% |
IN8 753 OUT92 | |
| ✓ | GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT92 |
Agentic - Game Backend: Dice notation max roll
1 passed |
100.0% |
IN8 768 OUT281 | |
| ✓ | GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 768 OUT281 |
Agentic - Game Backend: Enemy AI state machine
1 passed |
100.0% |
IN8 854 OUT349 | |
| ✓ | GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 854 OUT349 |
Agentic - Game Backend: Equipment slot validation
1 passed |
100.0% |
IN8 768 OUT167 | |
| ✓ | GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 768 OUT167 |
Agentic - Game Backend: Flood-fill region size
1 passed |
100.0% |
IN8 756 OUT516 | |
| ✓ | GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT516 |
Agentic - Game Backend: Fog of war visibility
1 passed |
100.0% |
IN8 796 OUT228 | |
| ✓ | GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 796 OUT228 |
Agentic - Game Backend: Fog reveal (Chebyshev radius)
1 passed |
100.0% |
IN19 304 OUT1 936 | |
| ✓ | GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN19 304 OUT1 936 |
Agentic - Game Backend: Friction stop frame
1 passed |
100.0% |
IN8 749 OUT49 | |
| ✓ | GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 749 OUT49 |
Agentic - Game Backend: Game of Life (code-gen vs oracle)
1 passed |
100.0% |
IN8 808 OUT307 | |
| ✓ | GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 808 OUT307 |
Agentic - Game Backend: Greedy coin change
1 passed |
100.0% |
IN8 750 OUT130 | |
| ✓ | GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 750 OUT130 |
Agentic - Game Backend: Grid inventory & stacking (Diablo-style)
1 passed |
100.0% |
IN18 151 OUT669 | |
| ✓ | GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 151 OUT669 |
Agentic - Game Backend: Grid pathfinding (shortest path, BFS)
1 passed |
100.0% |
IN8 838 OUT661 | |
| ✓ | GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.1 AI query
0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 838 OUT661 |
Agentic - Game Backend: Grid raycast
1 passed |
100.0% |
IN8 806 OUT295 | |
| ✓ | GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 806 OUT295 |
Agentic - Game Backend: Health bar segments
1 passed |
100.0% |
IN8 807 OUT190 | |
| ✓ | GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 807 OUT190 |
Agentic - Game Backend: High-score table insert
1 failed |
0.0% |
IN36 883 OUT1 048 | |
| × | GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.4 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Insert did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Char literals must contain exactly one character and close with '''. |
Failed | IN36 883 OUT1 048 |
Agentic - Game Backend: Hit chance
1 passed |
100.0% |
IN8 745 OUT71 | |
| ✓ | GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 745 OUT71 |
Agentic - Game Backend: Inventory encumbrance
1 passed |
100.0% |
IN8 742 OUT131 | |
| ✓ | GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 742 OUT131 |
Agentic - Game Backend: Inventory first-fit placement
1 passed |
100.0% |
IN17 895 OUT447 | |
| ✓ | GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 895 OUT447 |
Agentic - Game Backend: Inventory stack split
1 passed |
100.0% |
IN17 614 OUT89 | |
| ✓ | GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN17 614 OUT89 |
Agentic - Game Backend: Isometric projection
1 passed |
100.0% |
IN17 699 OUT186 | |
| ✓ | GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN17 699 OUT186 |
Agentic - Game Backend: Jump buffer
1 passed |
100.0% |
IN8 763 OUT128 | |
| ✓ | GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 763 OUT128 |
Agentic - Game Backend: Knockback with arena clamp
1 failed |
0.0% |
IN8 802 OUT207 | |
| × | GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Knockback got "$nx,$ny" expected "5,2"
Values:
r.ResultJson = "\"$nx,$ny\""
exp = "\"5,2\"" |
Failed | IN8 802 OUT207 |
Agentic - Game Backend: Leaderboard rank
1 passed |
100.0% |
IN8 735 OUT120 | |
| ✓ | GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 735 OUT120 |
Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over)
1 passed |
100.0% |
IN18 053 OUT472 | |
| ✓ | GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 053 OUT472 |
Agentic - Game Backend: Lifesteal healing
1 passed |
100.0% |
IN8 746 OUT101 | |
| ✓ | GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 746 OUT101 |
Agentic - Game Backend: Lights Out toggle
1 passed |
100.0% |
IN19 871 OUT2 450 | |
| ✓ | GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN19 871 OUT2 450 |
Agentic - Game Backend: Match-3 cascade
1 passed |
100.0% |
IN8 820 OUT726 | |
| ✓ | GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 820 OUT726 |
Agentic - Game Backend: Match-3 detection
1 passed |
100.0% |
IN18 075 OUT955 | |
| ✓ | GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 075 OUT955 |
Agentic - Game Backend: Match-3 swap validity
1 failed |
0.0% |
IN8 793 OUT493 | |
| × | GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: ValidSwap 4x4 [1111222122111222] (1,0) L got false expected true
Values:
r.ResultJson = "false"
exp = "true" |
Failed | IN8 793 OUT493 |
Agentic - Game Backend: Memory match pairs
1 passed |
100.0% |
IN8 747 OUT80 | |
| ✓ | GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 747 OUT80 |
Agentic - Game Backend: Mercy invincibility (i-frames)
1 passed |
100.0% |
IN8 802 OUT207 | |
| ✓ | GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 802 OUT207 |
Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal)
1 passed |
100.0% |
IN8 884 OUT563 | |
| ✓ | GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 884 OUT563 |
Agentic - Game Backend: Minimap downsample
1 passed |
100.0% |
IN8 778 OUT229 | |
| ✓ | GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 778 OUT229 |
Agentic - Game Backend: Minimap marker position
1 failed |
0.0% |
IN8 781 OUT142 | |
| × | GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Marker got "$mx,$my" expected "5,5"
Values:
r.ResultJson = "\"$mx,$my\""
exp = "\"5,5\"" |
Failed | IN8 781 OUT142 |
Agentic - Game Backend: N-in-a-row win detection (Connect-4)
1 passed |
100.0% |
IN8 796 OUT770 | |
| ✓ | GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 796 OUT770 |
Agentic - Game Backend: Nonogram row validation
1 passed |
100.0% |
IN8 758 OUT275 | |
| ✓ | GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 758 OUT275 |
Agentic - Game Backend: OBB collision (rotating rectangles, SAT)
1 failed |
0.0% |
IN31 326 OUT1 095 | |
| × | GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.1 AI query
2 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == BoolJson(exp)
Context: ObbOverlap mismatch inputs [1,-2,-3,-3,3,2 | 6,-12,-1,4,2,3] got false expected true
Values:
r.ResultJson = "false" |
Failed | IN31 326 OUT1 095 |
Agentic - Game Backend: Particle lifetime & expiry
1 passed |
100.0% |
IN8 840 OUT208 | |
| ✓ | GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 840 OUT208 |
Agentic - Game Backend: Party XP share
1 passed |
100.0% |
IN8 758 OUT105 | |
| ✓ | GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 758 OUT105 |
Agentic - Game Backend: Patrol waypoints (ping-pong)
1 passed |
100.0% |
IN8 781 OUT104 | |
| ✓ | GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 781 OUT104 |
Agentic - Game Backend: Platformer jump arc
1 passed |
100.0% |
IN8 772 OUT103 | |
| ✓ | GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT103 |
Agentic - Game Backend: Point on circle (trig)
1 passed |
100.0% |
IN27 061 OUT495 | |
| ✓ | GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.3 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN27 061 OUT495 |
Agentic - Game Backend: Poker hand comparison
1 passed |
100.0% |
IN8 881 OUT1 546 | |
| ✓ | GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users |
Passed | IN8 881 OUT1 546 |
Agentic - Game Backend: Poker hand ranking
1 passed |
100.0% |
IN8 824 OUT558 | |
| ✓ | GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 824 OUT558 |
Agentic - Game Backend: Pop-cap training
1 passed |
100.0% |
IN8 759 OUT109 | |
| ✓ | GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 759 OUT109 |
Agentic - Game Backend: Progress bar fill
1 passed |
100.0% |
IN8 741 OUT58 | |
| ✓ | GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 741 OUT58 |
Agentic - Game Backend: Projectile range (physics/trig)
1 passed |
100.0% |
IN8 763 OUT76 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 763 OUT76 |
Agentic - Game Backend: Projectile with ground bounce
1 passed |
100.0% |
IN17 780 OUT260 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 780 OUT260 |
Agentic - Game Backend: Resource gathering
1 passed |
100.0% |
IN8 758 OUT41 | |
| ✓ | GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 758 OUT41 |
Agentic - Game Backend: Reversi flip count
1 passed |
100.0% |
IN18 238 OUT790 | |
| ✓ | GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 238 OUT790 |
Agentic - Game Backend: Rotate 2D vector (trig)
1 passed |
100.0% |
IN8 774 OUT108 | |
| ✓ | GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 774 OUT108 |
Agentic - Game Backend: Round-robin deal
1 passed |
100.0% |
IN8 763 OUT95 | |
| ✓ | GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 763 OUT95 |
Agentic - Game Backend: RTS resource economy
1 passed |
100.0% |
IN8 754 OUT100 | |
| ✓ | GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 754 OUT100 |
Agentic - Game Backend: RTS unit counters & damage
1 passed |
100.0% |
IN8 806 OUT159 | |
| ✓ | GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 806 OUT159 |
Agentic - Game Backend: Safest neighbour (danger map)
1 passed |
100.0% |
IN18 127 OUT910 | |
| ✓ | GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 127 OUT910 |
Agentic - Game Backend: Scroll viewport
1 passed |
100.0% |
IN17 679 OUT145 | |
| ✓ | GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 679 OUT145 |
Agentic - Game Backend: Set card validity
1 passed |
100.0% |
IN8 755 OUT133 | |
| ✓ | GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT133 |
Agentic - Game Backend: Shop affordability
1 passed |
100.0% |
IN8 735 OUT60 | |
| ✓ | GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 735 OUT60 |
Agentic - Game Backend: Sliding puzzle move
1 passed |
100.0% |
IN17 824 OUT512 | |
| ✓ | GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 824 OUT512 |
Agentic - Game Backend: Snake head movement
1 passed |
100.0% |
IN8 823 OUT259 | |
| ✓ | GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 823 OUT259 |
Agentic - Game Backend: Snake tick
1 passed |
100.0% |
IN18 196 OUT719 | |
| ✓ | GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 196 OUT719 |
Agentic - Game Backend: Snap to grid
1 passed |
100.0% |
IN8 751 OUT64 | |
| ✓ | GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 751 OUT64 |
Agentic - Game Backend: Sokoban move/push legality
1 failed |
0.0% |
IN37 096 OUT1 738 | |
| × | GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.4 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users CanMove did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:2: Char literals must contain exactly one character and close with '''.
1:97: Char literals must contain exactly one character and close with '''.
3:9: Unexpected character '`'.
3:27: Unexpected character '`'.
4:9: Unexpected character '`'.
4:11: Unexpected character '`'.
5:9: Unexpected character '`'.
5:31: Unexpected character '`'.
6:9: Unexpected character '`'.
6:31: Unexpected character '`'.
7:9: Unexpected character '`'.
7:40: Unexpected character '`'.
8:9: Unexpected character '`'.
8:41: Unexpected character '`'.
9:9: Unexpected character '`'.
9:55: Unexpected character '`'.
10:9: Unexpected character '`'.
10:37: Unexpected character '`'.
11:9: Unexpected character '`'.
11:15: Unexpected character '`'.
12:10: Unexpected character '`'.
12:36: Unexpected character '`'.
13:10: Unexpected character '`'.
13:36: Unexpected character '`'.
14:10: Unexpected character '`'.
14:45: Unexpected character '`'.
15:10: Unexpected character '`'.
15:46: Unexpected character '`'.
16:10: Unexpected character '`'.
16:60: Unexpected character '`'.
17:10: Unexpected character '`'.
17:60: Unexpected character '`'.
18:10: Unexpected character '`'.
18:16: Unexpected character '`'.
19:10: Unexpected character '`'.
19:27: Unexpected character '`'.
20:10: Unexpected character '`'.
20:12: Unexpected character '`'.
22:10: Char literals must contain exactly one character and close with '''.
22:238: Unexpected character '`'.
22:265: Unexpected character '`'.
24:5: Unexpected character '—'.
24:113: Unexpected character '`'.
24:115: Unexpected character '`'.
24:146: Char literals must contain exactly one character and close with '''.
24:186: Unexpected character '`'.
24:188: Unexpected character '`'.
26:257: Unexpected character '→'.
26:310: Unexpected character '`'.
26:312: Unexpected character '`'.
26:327: Char literals must contain exactly one character and close with '''.
28:53: Unexpected character '#'.
28:71: Unexpected character '`'.
28:74: Unexpected character '`'.
28:105: Unexpected character '`'.
28:133: Unexpected character '`'.
32:105: Unexpected character '`'.
32:112: Unexpected character '`'.
32:145: Unexpected character '`'.
32:147: Unexpected character '`'.
32:286: Unexpected character '—'.
34:27: Unexpected character '#'.
34:38: Unexpected character '`'.
34:41: Unexpected character '`'.
34:43: Unexpected character '`'.
34:46: Unexpected character '`'.
34:70: Unexpected character '`'.
34:73: Unexpected character '`'. |
Failed | IN37 096 OUT1 738 |
Agentic - Game Backend: Solitaire tableau move
1 passed |
100.0% |
IN8 793 OUT184 | |
| ✓ | GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 793 OUT184 |
Agentic - Game Backend: Spawn scheduler count
1 passed |
100.0% |
IN8 750 OUT69 | |
| ✓ | GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 750 OUT69 |
Agentic - Game Backend: Spawn-wave alive count
1 passed |
100.0% |
IN8 762 OUT134 | |
| ✓ | GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 762 OUT134 |
Agentic - Game Backend: Stat level scaling
1 passed |
100.0% |
IN8 754 OUT66 | |
| ✓ | GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT66 |
Agentic - Game Backend: Steering seek (greedy pursuit)
1 passed |
100.0% |
IN8 780 OUT127 | |
| ✓ | GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 780 OUT127 |
Agentic - Game Backend: Tactical movement range
1 passed |
100.0% |
IN19 134 OUT2 012 | |
| ✓ | GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN19 134 OUT2 012 |
Agentic - Game Backend: Tech tree eligibility
1 passed |
100.0% |
IN17 835 OUT593 | |
| ✓ | GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 835 OUT593 |
Agentic - Game Backend: Tetris line clearing
1 passed |
100.0% |
IN8 765 OUT335 | |
| ✓ | GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 765 OUT335 |
Agentic - Game Backend: Threat table top target
1 passed |
100.0% |
IN8 759 OUT336 | |
| ✓ | GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 759 OUT336 |
Agentic - Game Backend: Tic-tac-toe result
1 passed |
100.0% |
IN8 750 OUT215 | |
| ✓ | GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 750 OUT215 |
Agentic - Game Backend: Toroidal wrap (Pac-Man)
1 passed |
100.0% |
IN17 689 OUT122 | |
| ✓ | GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 689 OUT122 |
Agentic - Game Backend: Tower targeting (nearest in range)
1 passed |
100.0% |
IN8 774 OUT218 | |
| ✓ | GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 774 OUT218 |
Agentic - Game Backend: Transaction ledger
1 passed |
100.0% |
IN8 777 OUT211 | |
| ✓ | GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 777 OUT211 |
Agentic - Game Backend: Trick-taking winner
1 passed |
100.0% |
IN8 805 OUT358 | |
| ✓ | GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 805 OUT358 |
Agentic - Game Backend: Turn-based duel resolution
1 passed |
100.0% |
IN8 837 OUT352 | |
| ✓ | GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 837 OUT352 |
Agentic - Game Backend: Typed armor mitigation
1 passed |
100.0% |
IN8 761 OUT290 | |
| ✓ | GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT290 |
Agentic - Game Backend: UI menu (wrap/clamp/paginate)
1 passed |
100.0% |
IN8 853 OUT226 | |
| ✓ | GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 853 OUT226 |
Agentic - Game Backend: Uno playable cards
1 passed |
100.0% |
IN8 789 OUT173 | |
| ✓ | GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 789 OUT173 |
Agentic - Game Backend: Velocity reflection
1 passed |
100.0% |
IN8 733 OUT85 | |
| ✓ | GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 733 OUT85 |
Agentic - Game Backend: Wall-slide fall
1 passed |
100.0% |
IN8 741 OUT73 | |
| ✓ | GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 741 OUT73 |
Agentic - Game Backend: War card compare
1 passed |
100.0% |
IN8 761 OUT237 | |
| ✓ | GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 761 OUT237 |
Agentic - Game Backend: Weighted loot table
1 passed |
100.0% |
IN8 764 OUT94 | |
| ✓ | GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 764 OUT94 |
Agentic - Game Backend: Weighted terrain pathfinding
1 failed |
0.0% |
IN47 063 OUT5 840 | |
| × | GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.4 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users MinCost did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
98:4: Expected a name, found '-'. |
Failed | IN47 063 OUT5 840 |
Agentic - Game Backend: Word-wrap line count
1 failed |
0.0% |
IN8 746 OUT173 | |
| × | GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: LineCount([toolongword],5) got 2 expected 1
Values:
r.ResultJson = "2" |
Failed | IN8 746 OUT173 |
Agentic - Game Backend: XP to character level
1 passed |
100.0% |
IN8 754 OUT65 | |
| ✓ | GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT65 |
Agentic - Idempotency (poison-retry / no double charge)
1 passed |
100.0% |
IN22 582 OUT287 | |
| ✓ | AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query
7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN22 582 OUT287 |
Agentic - Irreversible Commit (measure twice)
1 passed |
100.0% |
IN26 968 OUT715 | |
| ✓ | AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query
10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN26 968 OUT715 |
Agentic - Least Privilege (exact grants)
1 passed |
100.0% |
IN20 504 OUT264 | |
| ✓ | AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query
6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN20 504 OUT264 |
Agentic - Ledger (simulated DB read-modify-write)
1 passed |
100.0% |
IN20 456 OUT941 | |
| ✓ | AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query
11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN20 456 OUT941 |
Agentic - Ledger variants (5 RMW cases)
4 passed1 failed |
80.0% |
IN124 037 OUT4 323 | |
| ✓ | LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN35 966 OUT1 627 |
| ✓ | LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN15 867 OUT729 |
| ✓ | LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN16 390 OUT846 |
| × | LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query
14 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: actual == expected
Context: Account w ended at 500 but expected 0 (in-order apply with overdraft-skip).
Values:
actual = 500
expected = 0 |
Failed | IN35 499 OUT630 |
| ✓ | LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN20 315 OUT491 |
Agentic - Lock Ordering (deadlock avoidance)
1 broken |
0.0% |
IN16 381 OUT3 436 | |
| ! | AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query
12 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: response.Completed
Context: Agent response did not complete.
Values:
response.Completed = false |
Broken | IN16 381 OUT3 436 |
Agentic - Lock Ordering variants (5 cases)
3 passed2 broken |
60.0% |
IN169 161 OUT7 299 | |
| ✓ | LockVariantFiveResFourOps1 AI query
53 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN92 903 OUT2 362 |
| ! | LockVariantFourResThreeOps1 AI query
1 tool callcategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: response.Completed
Context: Agent response did not complete.
Values:
response.Completed = false |
Broken | IN5 527 OUT1 626 |
| ✓ | LockVariantHeavyReuse1 AI query
23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN46 869 OUT1 537 |
| ✓ | LockVariantReverseListed1 AI query
26 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN18 335 OUT1 141 |
| ! | LockVariantThreeLockOp1 AI query
1 tool callcategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: response.Completed
Context: Agent response did not complete.
Values:
response.Completed = false |
Broken | IN5 527 OUT633 |
Agentic - Migration (cascading rollback + reapply)
1 failed |
0.0% |
IN2 681 OUT17 | |
| × | AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query
0 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: passed
Context: Agent did not reach a passing migration health check (consistent, all migrations applied).
Values:
passed = false |
Failed | IN2 681 OUT17 |
Agentic - Requirements (detect contradiction)
1 passed |
100.0% |
IN7 087 OUT70 | |
| ✓ | AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query
2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN7 087 OUT70 |
Agentic - Retry/Circuit-breaker (transient vs hard-down)
1 broken |
0.0% |
IN11 292 OUT798 | |
| ! | AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: response.Completed
Context: Agent response did not complete.
Values:
response.Completed = false |
Broken | IN11 292 OUT798 |
Agentic - Saga (compensation / distributed rollback)
1 passed |
100.0% |
IN15 431 OUT425 | |
| ✓ | AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query
12 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN15 431 OUT425 |
Agentic - Saga variants (5 compensation cases)
5 passed |
100.0% |
IN86 586 OUT1 389 | |
| ✓ | SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query
4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN13 407 OUT166 |
| ✓ | SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query
4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN13 457 OUT169 |
| ✓ | SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query
7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 717 OUT290 |
| ✓ | SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query
7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN22 963 OUT386 |
| ✓ | SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query
9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN20 042 OUT378 |
Agentic - Schema Compat (expand-contract)
1 passed |
100.0% |
IN17 681 OUT219 | |
| ✓ | AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN17 681 OUT219 |
Agentic - Scratchpad (self-repair, Dyck, tool-decision)
6 passed1 broken |
85.7% |
IN311 396 OUT23 353 | |
| ✓ | AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query
7 tool callscategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN101 487 OUT3 339 |
| ! | AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query
0 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: response.Completed
Context: AI response did not complete.
Values:
response.Completed = false |
Broken | IN10 602 OUT9 993 |
| ✓ | ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query
2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN31 810 OUT162 |
| ✓ | ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query
2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN32 266 OUT466 |
| ✓ | ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query
3 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN43 283 OUT408 |
| ✓ | ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query
2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN46 601 OUT7 506 |
| ✓ | ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query
3 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN45 347 OUT1 479 |
Agentic - Secret Rotation (zero-downtime ordering)
1 broken |
0.0% |
IN7 621 OUT560 | |
| ! | AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query
2 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: response.Completed
Context: Agent response did not complete.
Values:
response.Completed = false |
Broken | IN7 621 OUT560 |
Agentic - StatefulOps (incident recovery, concurrency replan)
2 passed |
100.0% |
IN62 300 OUT1 277 | |
| ✓ | AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query
7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN26 166 OUT635 |
| ✓ | AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query
9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN36 134 OUT642 |
Agentic - Test Authoring (write a test that catches the bug)
1 passed |
100.0% |
IN15 104 OUT492 | |
| ✓ | AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 104 OUT492 |
Agentic - Topological Planning (deps + prereq)
1 passed |
100.0% |
IN31 124 OUT397 | |
| ✓ | AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN31 124 OUT397 |
Algorithms - 0/1 Knapsack (dynamic programming)
1 passed |
100.0% |
IN8 768 OUT295 | |
| ✓ | KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 768 OUT295 |
Algorithms - 1D elastic collision
1 passed |
100.0% |
IN8 808 OUT140 | |
| ✓ | ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 808 OUT140 |
Algorithms - All-pairs shortest path pair
1 passed |
100.0% |
IN8 776 OUT613 | |
| ✓ | FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 776 OUT613 |
Algorithms - Balanced brackets (stack)
1 passed |
100.0% |
IN8 739 OUT175 | |
| ✓ | ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 739 OUT175 |
Algorithms - Bellman-Ford shortest path (neg weights/cycles)
1 passed |
100.0% |
IN41 573 OUT8 413 | |
| ✓ | BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.3 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN41 573 OUT8 413 |
Algorithms - Binary Gray code
1 passed |
100.0% |
IN8 712 OUT23 | |
| ✓ | GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 712 OUT23 |
Algorithms - Bipartite graph check
1 passed |
100.0% |
IN8 758 OUT403 | |
| ✓ | BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 758 OUT403 |
Algorithms - Bit tricks (popcount/pow2/lowbit)
1 passed |
100.0% |
IN8 786 OUT124 | |
| ✓ | BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 786 OUT124 |
Algorithms - Caesar cipher
1 passed |
100.0% |
IN8 735 OUT216 | |
| ✓ | CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 735 OUT216 |
Algorithms - Catalan number
1 passed |
100.0% |
IN8 757 OUT69 | |
| ✓ | CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 757 OUT69 |
Algorithms - Chinese remainder theorem
1 failed |
0.0% |
IN8 775 OUT387 | |
| × | CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: Solve(1,2,2,3) got 3 expected 5
Values:
r.ResultJson = "3" |
Failed | IN8 775 OUT387 |
Algorithms - Circle-circle MTV
1 passed |
100.0% |
IN8 852 OUT149 | |
| ✓ | CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 852 OUT149 |
Algorithms - Coin change combinations
1 passed |
100.0% |
IN8 742 OUT208 | |
| ✓ | CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 742 OUT208 |
Algorithms - Collatz step count
1 passed |
100.0% |
IN8 738 OUT78 | |
| ✓ | CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 738 OUT78 |
Algorithms - Connected components (union-find)
1 passed |
100.0% |
IN8 754 OUT367 | |
| ✓ | UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT367 |
Algorithms - Container with most water
1 passed |
100.0% |
IN8 746 OUT181 | |
| ✓ | ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 746 OUT181 |
Algorithms - Count anagram groups
1 passed |
100.0% |
IN8 729 OUT130 | |
| ✓ | AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 729 OUT130 |
Algorithms - Count grid color regions
1 passed |
100.0% |
IN18 004 OUT942 | |
| ✓ | CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN18 004 OUT942 |
Algorithms - Count islands
1 passed |
100.0% |
IN18 889 OUT1 904 | |
| ✓ | CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN18 889 OUT1 904 |
Algorithms - Count overlapping occurrences
1 passed |
100.0% |
IN8 735 OUT99 | |
| ✓ | RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 735 OUT99 |
Algorithms - Count primes (Sieve of Eratosthenes)
1 passed |
100.0% |
IN8 714 OUT170 | |
| ✓ | CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 714 OUT170 |
Algorithms - Cumulative distribution
1 passed |
100.0% |
IN8 736 OUT178 | |
| ✓ | CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 736 OUT178 |
Algorithms - Dijkstra shortest path (weighted graph)
1 passed |
100.0% |
IN8 788 OUT554 | |
| ✓ | DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 788 OUT554 |
Algorithms - Directed cycle detection
1 passed |
100.0% |
IN18 859 OUT1 600 | |
| ✓ | DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN18 859 OUT1 600 |
Algorithms - Distinct subsequences
1 passed |
100.0% |
IN8 731 OUT180 | |
| ✓ | DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 731 OUT180 |
Algorithms - Divisor count
1 passed |
100.0% |
IN8 719 OUT63 | |
| ✓ | DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 719 OUT63 |
Algorithms - Downsample by block average
1 passed |
100.0% |
IN8 760 OUT139 | |
| ✓ | ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 760 OUT139 |
Algorithms - Dutch national flag sort
1 passed |
100.0% |
IN8 727 OUT273 | |
| ✓ | DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 727 OUT273 |
Algorithms - Egg drop trials
1 failed |
0.0% |
IN8 788 OUT165 | |
| × | EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: MinTrials(1,10) got 4 expected 10
Values:
r.ResultJson = "4" |
Failed | IN8 788 OUT165 |
Algorithms - Equal subset-sum partition
1 passed |
100.0% |
IN8 730 OUT246 | |
| ✓ | PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 730 OUT246 |
Algorithms - Euler totient
1 passed |
100.0% |
IN8 719 OUT207 | |
| ✓ | EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 719 OUT207 |
Algorithms - Eulerian path existence
1 passed |
100.0% |
IN8 778 OUT557 | |
| ✓ | EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 778 OUT557 |
Algorithms - Extended Euclidean
1 passed |
100.0% |
IN8 831 OUT189 | |
| ✓ | ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 831 OUT189 |
Algorithms - Format duration H:MM:SS
1 passed |
100.0% |
IN8 746 OUT130 | |
| ✓ | FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 746 OUT130 |
Algorithms - GCD and LCM
1 passed |
100.0% |
IN30 416 OUT295 | |
| ✓ | GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.1 AI query
2 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN30 416 OUT295 |
Algorithms - Grid transpose
1 failed |
0.0% |
IN8 754 OUT88 | |
| × | TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Transpose 3x2 [abcdef] got "acbdce" expected "adbecf"
Values:
r.ResultJson = "\"acbdce\""
exp = "\"adbecf\"" |
Failed | IN8 754 OUT88 |
Algorithms - House robber (max non-adjacent sum)
1 passed |
100.0% |
IN8 718 OUT112 | |
| ✓ | HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 718 OUT112 |
Algorithms - Integer linear interpolation
1 passed |
100.0% |
IN8 752 OUT36 | |
| ✓ | LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 752 OUT36 |
Algorithms - Integer square root
1 passed |
100.0% |
IN8 734 OUT132 | |
| ✓ | IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 734 OUT132 |
Algorithms - Integer to base-N string
1 passed |
100.0% |
IN19 763 OUT1 182 | |
| ✓ | BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.1 AI query
1 tool callcategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN19 763 OUT1 182 |
Algorithms - Integer to Roman numeral
1 passed |
100.0% |
IN8 769 OUT207 | |
| ✓ | RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT207 |
Algorithms - Josephus survivor
1 passed |
100.0% |
IN8 745 OUT78 | |
| ✓ | JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 745 OUT78 |
Algorithms - Jump game (min jumps)
1 passed |
100.0% |
IN8 753 OUT262 | |
| ✓ | JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT262 |
Algorithms - Kth largest element
1 passed |
100.0% |
IN8 743 OUT90 | |
| ✓ | KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 743 OUT90 |
Algorithms - Least-squares line fit
1 passed |
100.0% |
IN8 785 OUT207 | |
| ✓ | LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 785 OUT207 |
Algorithms - Levenshtein edit distance (DP)
1 passed |
100.0% |
IN8 713 OUT340 | |
| ✓ | EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 713 OUT340 |
Algorithms - Log-scale bucket index
1 passed |
100.0% |
IN8 778 OUT60 | |
| ✓ | LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 778 OUT60 |
Algorithms - Logarithmic bin index
1 passed |
100.0% |
IN8 767 OUT89 | |
| ✓ | LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 767 OUT89 |
Algorithms - Longest common prefix
1 passed |
100.0% |
IN8 726 OUT197 | |
| ✓ | CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 726 OUT197 |
Algorithms - Longest common subsequence
1 passed |
100.0% |
IN8 711 OUT246 | |
| ✓ | LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 711 OUT246 |
Algorithms - Longest common substring
1 passed |
100.0% |
IN8 728 OUT162 | |
| ✓ | CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 728 OUT162 |
Algorithms - Longest increasing subsequence (DP)
1 passed |
100.0% |
IN8 745 OUT205 | |
| ✓ | LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 745 OUT205 |
Algorithms - Longest palindromic substring
1 passed |
100.0% |
IN8 723 OUT227 | |
| ✓ | LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 723 OUT227 |
Algorithms - Longest repeated substring
1 passed |
100.0% |
IN8 728 OUT189 | |
| ✓ | LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 728 OUT189 |
Algorithms - Lower bound (binary search)
1 passed |
100.0% |
IN8 756 OUT115 | |
| ✓ | LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT115 |
Algorithms - Luhn checksum validation
1 passed |
100.0% |
IN8 757 OUT135 | |
| ✓ | LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 757 OUT135 |
Algorithms - Majority element
1 passed |
100.0% |
IN8 721 OUT199 | |
| ✓ | MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 721 OUT199 |
Algorithms - Matrix chain multiplication
1 passed |
100.0% |
IN8 756 OUT314 | |
| ✓ | MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 756 OUT314 |
Algorithms - Maximum subarray sum (Kadane)
1 passed |
100.0% |
IN8 746 OUT99 | |
| ✓ | MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 746 OUT99 |
Algorithms - Merge K sorted lists
1 passed |
100.0% |
IN8 739 OUT197 | |
| ✓ | MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 739 OUT197 |
Algorithms - Merge overlapping intervals
1 passed |
100.0% |
IN27 731 OUT1 242 | |
| ✓ | MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.3 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN27 731 OUT1 242 |
Algorithms - Min-max normalization
1 passed |
100.0% |
IN8 748 OUT289 | |
| ✓ | MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 748 OUT289 |
Algorithms - Minimum coin change (DP)
1 passed |
100.0% |
IN8 739 OUT312 | |
| ✓ | CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 739 OUT312 |
Algorithms - Minimum knight moves (BFS)
1 passed |
100.0% |
IN18 875 OUT2 621 | |
| ✓ | KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN18 875 OUT2 621 |
Algorithms - Minimum path sum (grid DP)
1 passed |
100.0% |
IN8 752 OUT229 | |
| ✓ | MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT229 |
Algorithms - Minimum spanning tree (Kruskal)
1 passed |
100.0% |
IN8 752 OUT482 | |
| ✓ | KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 752 OUT482 |
Algorithms - Minimum window substring
1 failed |
0.0% |
IN8 760 OUT406 | |
| × | MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: res.ResultJson == exp
Context: Window(a,aa) got "a" expected ""
Values:
res.ResultJson = "\"a\""
exp = "\"\"" |
Failed | IN8 760 OUT406 |
Algorithms - Modular exponentiation (fast pow)
1 passed |
100.0% |
IN8 771 OUT146 | |
| ✓ | ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 771 OUT146 |
Algorithms - Modular inverse (extended Euclid)
1 passed |
100.0% |
IN8 759 OUT231 | |
| ✓ | ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 759 OUT231 |
Algorithms - Moving average
1 passed |
100.0% |
IN8 750 OUT188 | |
| ✓ | MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 750 OUT188 |
Algorithms - Next greater element
1 passed |
100.0% |
IN8 733 OUT277 | |
| ✓ | NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT277 |
Algorithms - Next lexicographic permutation
1 passed |
100.0% |
IN17 907 OUT1 459 | |
| ✓ | NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN17 907 OUT1 459 |
Algorithms - Nth Fibonacci mod m
1 passed |
100.0% |
IN8 740 OUT117 | |
| ✓ | FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 740 OUT117 |
Algorithms - Overlapping substring count
1 passed |
100.0% |
IN8 725 OUT102 | |
| ✓ | SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 725 OUT102 |
Algorithms - Palindrome & anagram checks
1 passed |
100.0% |
IN8 755 OUT632 | |
| ✓ | PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT632 |
Algorithms - Pascal triangle row
1 passed |
100.0% |
IN8 733 OUT153 | |
| ✓ | PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT153 |
Algorithms - Point-to-segment distance
1 passed |
100.0% |
IN17 825 OUT463 | |
| ✓ | PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN17 825 OUT463 |
Algorithms - Prime factorization
1 passed |
100.0% |
IN8 744 OUT160 | |
| ✓ | PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT160 |
Algorithms - Product of array except self
1 passed |
100.0% |
IN8 750 OUT269 | |
| ✓ | ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 750 OUT269 |
Algorithms - Quadratic Bezier point
1 passed |
100.0% |
IN8 810 OUT196 | |
| ✓ | BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 810 OUT196 |
Algorithms - Range sum via prefix sums
1 passed |
100.0% |
IN8 732 OUT69 | |
| ✓ | PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 732 OUT69 |
Algorithms - Reverse words & Hamming distance
1 passed |
100.0% |
IN8 736 OUT110 | |
| ✓ | ReverseWordsCodeGenCode-gen: model implements reverse-words + Hamming; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 736 OUT110 |
Algorithms - Rod cutting
1 passed |
100.0% |
IN8 766 OUT190 | |
| ✓ | RodCuttingCodeGenCode-gen: model implements rod cutting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 766 OUT190 |
Algorithms - Roman numeral to integer
1 passed |
100.0% |
IN8 755 OUT240 | |
| ✓ | RomanToIntCodeGenCode-gen: model implements Roman-to-int; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT240 |
Algorithms - Rotate array right by k
1 passed |
100.0% |
IN8 759 OUT140 | |
| ✓ | RotateArrayCodeGenCode-gen: model implements right rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 759 OUT140 |
Algorithms - Rotate grid 90 CW (discriminator)
1 failed |
0.0% |
IN8 762 OUT102 | |
| × | RotateMatrixCodeGenCode-gen: model implements 90-degree CW rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Rotate90 2x3 [abcdef] got "efcdab" expected "ecafdb"
Values:
r.ResultJson = "\"efcdab\""
exp = "\"ecafdb\"" |
Failed | IN8 762 OUT102 |
Algorithms - RPN evaluator
1 passed |
100.0% |
IN8 764 OUT354 | |
| ✓ | RpnCodeGenCode-gen: model implements an RPN evaluator; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 764 OUT354 |
Algorithms - Run-length compression (if shorter)
1 passed |
100.0% |
IN8 736 OUT150 | |
| ✓ | StringCompressCodeGenCode-gen: model implements string compression; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 736 OUT150 |
Algorithms - Run-length encoding (encode/decode)
1 passed |
100.0% |
IN8 782 OUT277 | |
| ✓ | RunLengthCodeGenCode-gen: model implements RLE encode+decode; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 782 OUT277 |
Algorithms - Semantic versioning (compare + bump)
1 passed |
100.0% |
IN17 949 OUT533 | |
| ✓ | SemVerCodeGenCode-gen: model implements semver compare + bump; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:InstructionFollowinglevel:Easysource:Ai4Users |
Passed | IN17 949 OUT533 |
Algorithms - Single-pile Nim (optimal play)
1 passed |
100.0% |
IN8 744 OUT48 | |
| ✓ | NimCodeGenCode-gen: model implements optimal Nim; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT48 |
Algorithms - Sliding window maximum
1 passed |
100.0% |
IN8 736 OUT266 | |
| ✓ | SlidingWindowMaxCodeGenCode-gen: model implements sliding window maximum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 736 OUT266 |
Algorithms - Spiral matrix traversal
1 passed |
100.0% |
IN28 641 OUT2 068 | |
| ✓ | SpiralMatrixCodeGenCode-gen: model implements spiral traversal; graded vs exact oracle.3 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN28 641 OUT2 068 |
Algorithms - Square matrix multiply
1 passed |
100.0% |
IN8 737 OUT319 | |
| ✓ | MatMulCodeGenCode-gen: model implements square matrix multiply; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 737 OUT319 |
Algorithms - Strongly connected components
1 passed |
100.0% |
IN18 654 OUT1 777 | |
| ✓ | SccCountCodeGenCode-gen: model implements SCC count; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN18 654 OUT1 777 |
Algorithms - Subset sum reachability
1 passed |
100.0% |
IN17 713 OUT388 | |
| ✓ | SubsetSumCodeGenCode-gen: model implements subset-sum reachability; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN17 713 OUT388 |
Algorithms - Substring search (KMP)
1 passed |
100.0% |
IN8 743 OUT359 | |
| ✓ | KmpCodeGenCode-gen: model implements KMP search; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 743 OUT359 |
Algorithms - Sum of digits & digital root
1 passed |
100.0% |
IN8 722 OUT127 | |
| ✓ | DigitalRootCodeGenCode-gen: model implements sum-of-digits + digital root; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 722 OUT127 |
Algorithms - Temperature conversion
1 passed |
100.0% |
IN8 733 OUT55 | |
| ✓ | TemperatureCodeGenCode-gen: model implements temperature conversion; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 733 OUT55 |
Algorithms - Topological sort (lexicographically smallest)
1 passed |
100.0% |
IN19 102 OUT2 003 | |
| ✓ | TopoCodeGenCode-gen: model implements lexicographically-smallest topological sort.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN19 102 OUT2 003 |
Algorithms - Trailing zeros of factorial
1 passed |
100.0% |
IN8 734 OUT65 | |
| ✓ | TrailingZerosCodeGenCode-gen: model implements trailing zeros; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 734 OUT65 |
Algorithms - Trapping rain water
1 failed |
0.0% |
IN8 733 OUT265 | |
| × | TrappingRainCodeGenCode-gen: model implements trapping rain water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: Trapped([0 1 0 2 1 0 1 3 2 1 2 1]) got 1 expected 6
Values:
r.ResultJson = "1" |
Failed | IN8 733 OUT265 |
Algorithms - Two-sum indices
1 passed |
100.0% |
IN8 740 OUT122 | |
| ✓ | TwoSumCodeGenCode-gen: model implements two-sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 740 OUT122 |
Algorithms - Unique grid paths
1 passed |
100.0% |
IN8 715 OUT143 | |
| ✓ | UniquePathsCodeGenCode-gen: model implements unique paths; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 715 OUT143 |
Algorithms - Valid Sudoku unit
1 passed |
100.0% |
IN17 554 OUT136 | |
| ✓ | SudokuUnitCodeGenCode-gen: model implements Sudoku unit validation; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN17 554 OUT136 |
Algorithms - Wildcard string matching
1 passed |
100.0% |
IN8 738 OUT333 | |
| ✓ | WildcardMatchCodeGenCode-gen: model implements wildcard match; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 738 OUT333 |
Algorithms - Z-function
1 failed |
0.0% |
IN8 739 OUT308 | |
| × | ZFunctionCodeGenCode-gen: model implements the Z-function; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: ZArray([aaaaa]) got "0 4 4 4 4" expected "0 4 3 2 1"
Values:
r.ResultJson = "\"0 4 4 4 4\""
exp = "\"0 4 3 2 1\"" |
Failed | IN8 739 OUT308 |
Database - Conditional aggregation pivot
1 passed |
100.0% |
IN947 OUT69 | |
| ✓ | PivotCountsCodeGenCode-gen: model writes the pivot query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN947 OUT69 |
Database - Count distinct opponents
1 passed |
100.0% |
IN927 OUT19 | |
| ✓ | DistinctCountCodeGenCode-gen: model writes the COUNT(DISTINCT) query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Easysource:Ai4Users |
Passed | IN927 OUT19 |
Database - Date-range filter (BETWEEN)
1 passed |
100.0% |
IN957 OUT41 | |
| ✓ | DateRangeCodeGenCode-gen: model writes the date-range query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN957 OUT41 |
Database - EXISTS subquery (teams with matches)
1 passed |
100.0% |
IN934 OUT31 | |
| ✓ | ExistsSubqueryCodeGenCode-gen: model writes the EXISTS query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN934 OUT31 |
Database - JOIN player totals (two tables)
1 passed |
100.0% |
IN956 OUT39 | |
| ✓ | JoinScoresCodeGenCode-gen: model writes a JOIN + aggregate query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN956 OUT39 |
Database - LEFT JOIN anti-join (no items)
1 passed |
100.0% |
IN930 OUT34 | |
| ✓ | LeftJoinNullsCodeGenCode-gen: model writes the anti-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN930 OUT34 |
Database - Player stats SQL (multi-aggregate, QueryJson)
1 passed |
100.0% |
IN960 OUT37 | |
| ✓ | PlayerStatsCodeGenCode-gen: model writes a multi-aggregate stats query; graded by comparing its QueryJson output to the oracle query's (arbitrary shape, no struct).1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN960 OUT37 |
Database - Players above average score (subquery)
1 passed |
100.0% |
IN925 OUT35 | |
| ✓ | AboveAverageCodeGenCode-gen: model writes an above-average query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN925 OUT35 |
Database - Players who never played (anti-join)
1 passed |
100.0% |
IN922 OUT33 | |
| ✓ | NeverPlayedCodeGenCode-gen: model writes an anti-join query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN922 OUT33 |
Database - RANK window function
1 passed |
100.0% |
IN952 OUT30 | |
| ✓ | WindowRankCodeGenCode-gen: model writes the RANK() query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN952 OUT30 |
Database - Running total (window function)
1 passed |
100.0% |
IN949 OUT50 | |
| ✓ | RunningTotalCodeGenCode-gen: model writes the window running-total query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN949 OUT50 |
Database - Scoreboard SQL (players with >1 win, HAVING)
1 passed |
100.0% |
IN956 OUT33 | |
| ✓ | WinFilterCodeGenCode-gen: model writes a HAVING query; runs under try/catch with self-repair; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN956 OUT33 |
Database - Scoreboard SQL (top scorers query)
1 passed |
100.0% |
IN957 OUT28 | |
| ✓ | ScoreboardTopScorersCodeGenCode-gen: model writes a scoreboard SQL query; runs under try/catch with a self-repair loop on SQL errors; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN957 OUT28 |
Database - Self-join same-team pairs
1 passed |
100.0% |
IN940 OUT53 | |
| ✓ | SelfJoinCodeGenCode-gen: model writes the self-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN940 OUT53 |
Database - Top scorer per region
1 passed |
100.0% |
IN942 OUT67 | |
| ✓ | TopPerGroupCodeGenCode-gen: model writes the per-group top query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN942 OUT67 |
Database - UPDATE restock (SQL write)
1 passed |
100.0% |
IN934 OUT17 | |
| ✓ | UpdateInventoryCodeGenCode-gen: model writes an UPDATE; applied under try/catch, resulting state compared (via QueryJson) to the oracle update's state.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN934 OUT17 |
Database - Upsert with max merge
1 passed |
100.0% |
IN1 987 OUT70 | |
| ✓ | UpsertCodeGenCode-gen: model writes the upsert; executed on a fresh DB and the table compared.2 AI queries
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN1 987 OUT70 |
SDLC - Acceptance criteria check
1 passed |
100.0% |
IN8 763 OUT423 | |
| ✓ | SdlcAcceptanceCodeGenCode-gen: model implements acceptance-criteria check; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 763 OUT423 |
SDLC - Breaking API change detection
1 passed |
100.0% |
IN18 421 OUT2 561 | |
| ✓ | SdlcApiBreakingCodeGenCode-gen: model implements API-breaking detection; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN18 421 OUT2 561 |
SDLC - Condense without losing facts
1 passed |
100.0% |
IN1 048 OUT54 | |
| ✓ | NoInfoLossCodeGenRedaction test: model condenses keeping facts; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 048 OUT54 |
SDLC - Config merge with override
1 passed |
100.0% |
IN8 754 OUT294 | |
| ✓ | ConfigMergeCodeGenCode-gen: model implements config merge; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT294 |
SDLC - Extract fields into fixed template
1 passed |
100.0% |
IN1 004 OUT22 | |
| ✓ | FieldExtractCodeGenRedaction test: model extracts fields into the template; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 004 OUT22 |
SDLC - Locate conflicting requirement pair
1 passed |
100.0% |
IN33 679 OUT4 949 | |
| ✓ | SdlcConstraintConflictCodeGenCode-gen: model locates the conflicting requirement pair; graded vs exact oracle.3 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN33 679 OUT4 949 |
SDLC - Locate redundant requirement
1 passed |
100.0% |
IN147 594 OUT7 486 | |
| ✓ | SdlcRedundantRequirementCodeGenCode-gen: model locates a redundant requirement; graded vs exact oracle.1 AI query
9 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN147 594 OUT7 486 |
SDLC - Mask PII to placeholder
1 passed |
100.0% |
IN972 OUT42 | |
| ✓ | MaskPiiCodeGenRedaction test: model masks PII to a placeholder; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN972 OUT42 |
SDLC - Modules on a dependency cycle
1 failed |
0.0% |
IN43 612 OUT2 291 | |
| × | DependencyCycleModulesCodeGenCode-gen: model implements dependency-cycle detection; graded vs exact oracle.2 AI queries
2 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users Assertion expression evaluated to false: res.ResultJson == exp
Context: Cycle(A>B;B>C;C>A) got "C" expected "A,B,C"
Values:
res.ResultJson = "\"C\""
exp = "\"A,B,C\"" |
Failed | IN43 612 OUT2 291 |
SDLC - Remove duplicate requirements
1 passed |
100.0% |
IN945 OUT10 | |
| ✓ | DeduplicateCodeGenRedaction test: model removes duplicate requirements; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN945 OUT10 |
SDLC - Renumber requirements sequentially
1 passed |
100.0% |
IN2 199 OUT109 | |
| ✓ | RenumberCodeGenRedaction test: model renumbers sequentially; checker + self-review loop.2 AI queries
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN2 199 OUT109 |
SDLC - Reorganize notes into section template
1 passed |
100.0% |
IN1 126 OUT94 | |
| ✓ | RewriteSectionsCodeGenRedaction test: model reorganizes the notes; deterministic checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 126 OUT94 |
SDLC - Required semver bump from changelog
1 passed |
100.0% |
IN8 759 OUT177 | |
| ✓ | SemverBumpCodeGenCode-gen: model implements the semver-bump rule; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Easysource:Ai4Users |
Passed | IN8 759 OUT177 |
SDLC - Requirement-to-test traceability
1 passed |
100.0% |
IN8 781 OUT281 | |
| ✓ | SdlcTraceabilityCodeGenCode-gen: model implements traceability gap; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 781 OUT281 |
SDLC - Requirements completeness (spec coverage gap)
1 passed |
100.0% |
IN8 776 OUT349 | |
| ✓ | SdlcSpecCoverageCodeGenCode-gen: model implements spec-gap detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 776 OUT349 |
SDLC - Requirements consistency (contradiction detection)
1 passed |
100.0% |
IN8 761 OUT460 | |
| ✓ | ConsistencyCodeGenCode-gen: model implements requirements-consistency checking; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT460 |
SDLC - Reshape prose into JSON template
1 passed |
100.0% |
IN1 014 OUT28 | |
| ✓ | JsonShapeCodeGenRedaction test: model reshapes prose into JSON; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 014 OUT28 |
SDLC - Reshape rows into Markdown table
1 passed |
100.0% |
IN968 OUT31 | |
| ✓ | TableFormatCodeGenRedaction test: model builds a Markdown table; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN968 OUT31 |
SDLC - Rewrite in imperative requirement style
1 passed |
100.0% |
IN991 OUT72 | |
| ✓ | StyleRewriteCodeGenRedaction test: model rewrites notes in imperative style; checker + self-review loop.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN991 OUT72 |
SDLC - Roll up estimates over task tree
1 passed |
100.0% |
IN8 772 OUT186 | |
| ✓ | EstimateRollupCodeGenCode-gen: model implements estimate roll-up over a task tree; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT186 |
SDLC - State-machine reachability
1 passed |
100.0% |
IN8 758 OUT311 | |
| ✓ | SdlcStateReachableCodeGenCode-gen: model implements state reachability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 758 OUT311 |
SDLC - Ticket priority sort
1 passed |
100.0% |
IN8 751 OUT298 | |
| ✓ | SdlcPrioritySortCodeGenCode-gen: model implements priority sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 751 OUT298 |
SDLC - Unify glossary term consistency
1 passed |
100.0% |
IN972 OUT38 | |
| ✓ | GlossaryCodeGenRedaction test: model unifies synonyms to the canonical term; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN972 OUT38 |