Benchmark summary
A custom Ai4Users benchmark suite for SC#, covering code generation, reasoning, and stateful agentic workflows where models use tools, observe results, adapt their actions, recover from failures, and verify the final state.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Agentic - Blue-Green Deploy (health-gated cutover)
1 passed |
100.0% |
IN19 101 OUT417 | |
| ✓ | AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query
6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN19 101 OUT417 |
Agentic - Bugfix (root-cause vs regression)
1 passed |
100.0% |
IN20 366 OUT218 | |
| ✓ | AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query
1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN20 366 OUT218 |
Agentic - Cache Invalidation (exact set)
1 passed |
100.0% |
IN14 851 OUT606 | |
| ✓ | AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query
7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN14 851 OUT606 |
Agentic - Canary (metric-gated release)
1 passed |
100.0% |
IN19 144 OUT571 | |
| ✓ | AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query
6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN19 144 OUT571 |
Agentic - Canary variants (5 metric cases)
5 passed |
100.0% |
IN165 972 OUT5 181 | |
| ✓ | CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN23 826 OUT909 |
| ✓ | CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN32 006 OUT986 |
| ✓ | CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN27 278 OUT776 |
| ✓ | CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query
14 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN54 828 OUT1 491 |
| ✓ | CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN28 034 OUT1 019 |
Agentic - Code Review (find injected defect)
1 passed |
100.0% |
IN13 446 OUT759 | |
| ✓ | AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN13 446 OUT759 |
Agentic - Config Drift (remediate exact)
1 passed |
100.0% |
IN12 162 OUT434 | |
| ✓ | AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query
6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN12 162 OUT434 |
Agentic - Constraints (multi-section satisfaction)
1 passed |
100.0% |
IN11 486 OUT533 | |
| ✓ | AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 486 OUT533 |
Agentic - Constraints variants (5 cases)
5 passed |
100.0% |
IN50 723 OUT2 343 | |
| ✓ | ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN10 572 OUT626 |
| ✓ | ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 922 OUT382 |
| ✓ | ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 980 OUT452 |
| ✓ | ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN10 277 OUT480 |
| ✓ | ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 972 OUT403 |
Agentic - Data Reconciliation (per-field precedence)
1 passed |
100.0% |
IN14 087 OUT785 | |
| ✓ | AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN14 087 OUT785 |
Agentic - Diagnosis (root cause under budget)
1 passed |
100.0% |
IN12 628 OUT778 | |
| ✓ | AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN12 628 OUT778 |
Agentic - Diagnosis variants (5 cases)
5 passed |
100.0% |
IN78 006 OUT4 324 | |
| ✓ | DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 409 OUT930 |
| ✓ | DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 433 OUT870 |
| ✓ | DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 172 OUT633 |
| ✓ | DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 593 OUT876 |
| ✓ | DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN16 399 OUT1 015 |
Agentic - ETL Repair (root-cause in stage 2)
1 passed |
100.0% |
IN39 612 OUT1 066 | |
| ✓ | AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query
15 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN39 612 OUT1 066 |
Agentic - Feature Flag (staged rollout + kill-switch)
1 passed |
100.0% |
IN28 058 OUT832 | |
| ✓ | AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN28 058 OUT832 |
Agentic - Game Backend: 1D bouncing ball
1 passed |
100.0% |
IN8 913 OUT136 | |
| ✓ | GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 913 OUT136 |
Agentic - Game Backend: 1D swept anti-tunneling
1 passed |
100.0% |
IN8 924 OUT221 | |
| ✓ | GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 924 OUT221 |
Agentic - Game Backend: 2048 left-move (slide + merge)
1 passed |
100.0% |
IN8 933 OUT318 | |
| ✓ | GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 933 OUT318 |
Agentic - Game Backend: 8-direction from vector
1 passed |
100.0% |
IN18 137 OUT496 | |
| ✓ | GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN18 137 OUT496 |
Agentic - Game Backend: AABB collision resolution (MTV)
1 passed |
100.0% |
IN9 029 OUT306 | |
| ✓ | GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN9 029 OUT306 |
Agentic - Game Backend: AABB containment
1 passed |
100.0% |
IN8 869 OUT58 | |
| ✓ | GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 869 OUT58 |
Agentic - Game Backend: Acceleration to max speed
1 passed |
100.0% |
IN8 871 OUT80 | |
| ✓ | GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 871 OUT80 |
Agentic - Game Backend: Achievement thresholds
1 passed |
100.0% |
IN8 863 OUT132 | |
| ✓ | GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 863 OUT132 |
Agentic - Game Backend: Active status effects
1 passed |
100.0% |
IN8 885 OUT171 | |
| ✓ | GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 885 OUT171 |
Agentic - Game Backend: Animation frame at time
1 passed |
100.0% |
IN8 903 OUT94 | |
| ✓ | GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 903 OUT94 |
Agentic - Game Backend: Army upkeep income
1 passed |
100.0% |
IN8 851 OUT109 | |
| ✓ | GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 851 OUT109 |
Agentic - Game Backend: Axis-aligned path follow
1 passed |
100.0% |
IN8 894 OUT370 | |
| ✓ | GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 894 OUT370 |
Agentic - Game Backend: Battleship shot resolution
1 passed |
100.0% |
IN8 953 OUT292 | |
| ✓ | GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 953 OUT292 |
Agentic - Game Backend: Bounce restitution
1 passed |
100.0% |
IN8 890 OUT73 | |
| ✓ | GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 890 OUT73 |
Agentic - Game Backend: Bresenham line rasterization
1 passed |
100.0% |
IN27 833 OUT738 | |
| ✓ | GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.3 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN27 833 OUT738 |
Agentic - Game Backend: Buff/debuff stacking
1 passed |
100.0% |
IN8 916 OUT259 | |
| ✓ | GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 916 OUT259 |
Agentic - Game Backend: Build queue completions
1 passed |
100.0% |
IN8 884 OUT106 | |
| ✓ | GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 884 OUT106 |
Agentic - Game Backend: Camera-relative movement
1 passed |
100.0% |
IN8 971 OUT289 | |
| ✓ | GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 971 OUT289 |
Agentic - Game Backend: Card hand sort
1 passed |
100.0% |
IN8 892 OUT439 | |
| ✓ | GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 892 OUT439 |
Agentic - Game Backend: Card-game rules (blackjack/deck/compare)
1 passed |
100.0% |
IN9 067 OUT604 | |
| ✓ | GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN9 067 OUT604 |
Agentic - Game Backend: Chart scaling & histogram bins
1 passed |
100.0% |
IN8 956 OUT240 | |
| ✓ | GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 956 OUT240 |
Agentic - Game Backend: Checkers captures
1 passed |
100.0% |
IN37 735 OUT1 566 | |
| ✓ | GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.4 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN37 735 OUT1 566 |
Agentic - Game Backend: Collision primitives (circle/aabb/segment)
1 failed |
0.0% |
IN9 129 OUT624 | |
| × | GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == BoolJson(RefSegmentsIntersect(ax, ay, bx, by, cx, cy, dx, dy))
Context: SegmentsIntersect mismatch got true
Values:
r.ResultJson = "true" |
Failed | IN9 129 OUT624 |
Agentic - Game Backend: Combat turn order (initiative)
1 passed |
100.0% |
IN8 865 OUT357 | |
| ✓ | GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 865 OUT357 |
Agentic - Game Backend: Combo scoring (rising multiplier)
1 passed |
100.0% |
IN8 888 OUT136 | |
| ✓ | GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 888 OUT136 |
Agentic - Game Backend: Control-point capture
1 passed |
100.0% |
IN8 901 OUT177 | |
| ✓ | GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 901 OUT177 |
Agentic - Game Backend: Controls & Inventory (spec-driven function battery)
1 passed |
100.0% |
IN9 042 OUT257 | |
| ✓ | GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN9 042 OUT257 |
Agentic - Game Backend: Cooldowns & resource regen
1 passed |
100.0% |
IN8 959 OUT176 | |
| ✓ | GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 959 OUT176 |
Agentic - Game Backend: Coyote-time jump
1 passed |
100.0% |
IN8 880 OUT70 | |
| ✓ | GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 880 OUT70 |
Agentic - Game Backend: Crafting capacity
1 passed |
100.0% |
IN8 904 OUT340 | |
| ✓ | GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 904 OUT340 |
Agentic - Game Backend: Cribbage fifteens
1 passed |
100.0% |
IN8 881 OUT218 | |
| ✓ | GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 881 OUT218 |
Agentic - Game Backend: Critical-hit damage
1 passed |
100.0% |
IN8 871 OUT57 | |
| ✓ | GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 871 OUT57 |
Agentic - Game Backend: Damage over time
1 passed |
100.0% |
IN8 897 OUT166 | |
| ✓ | GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 897 OUT166 |
Agentic - Game Backend: Day/night cycle phase
1 passed |
100.0% |
IN8 883 OUT91 | |
| ✓ | GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 883 OUT91 |
Agentic - Game Backend: Dice notation max roll
1 passed |
100.0% |
IN8 898 OUT259 | |
| ✓ | GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 898 OUT259 |
Agentic - Game Backend: Enemy AI state machine
1 passed |
100.0% |
IN8 984 OUT475 | |
| ✓ | GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 984 OUT475 |
Agentic - Game Backend: Equipment slot validation
1 passed |
100.0% |
IN8 898 OUT163 | |
| ✓ | GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 898 OUT163 |
Agentic - Game Backend: Flood-fill region size
1 passed |
100.0% |
IN18 236 OUT766 | |
| ✓ | GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 236 OUT766 |
Agentic - Game Backend: Fog of war visibility
1 passed |
100.0% |
IN8 926 OUT188 | |
| ✓ | GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 926 OUT188 |
Agentic - Game Backend: Fog reveal (Chebyshev radius)
1 passed |
100.0% |
IN8 899 OUT247 | |
| ✓ | GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 899 OUT247 |
Agentic - Game Backend: Friction stop frame
1 passed |
100.0% |
IN8 879 OUT49 | |
| ✓ | GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 879 OUT49 |
Agentic - Game Backend: Game of Life (code-gen vs oracle)
1 passed |
100.0% |
IN8 938 OUT320 | |
| ✓ | GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 938 OUT320 |
Agentic - Game Backend: Greedy coin change
1 failed |
0.0% |
IN17 929 OUT253 | |
| × | GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: MakeChange(87,[25 10 5 1]) got "3102" expected "3 1 0 2"
Values:
r.ResultJson = "\"3102\""
exp = "\"3 1 0 2\"" |
Failed | IN17 929 OUT253 |
Agentic - Game Backend: Grid inventory & stacking (Diablo-style)
1 passed |
100.0% |
IN9 013 OUT342 | |
| ✓ | GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN9 013 OUT342 |
Agentic - Game Backend: Grid pathfinding (shortest path, BFS)
1 failed |
0.0% |
IN30 476 OUT3 071 | |
| × | GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.3 AI queries
0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: ShortestPath mismatch grid 4x4 (0,0)->(3,3) got -1 expected 6
Values:
r.ResultJson = "-1" |
Failed | IN30 476 OUT3 071 |
Agentic - Game Backend: Grid raycast
1 passed |
100.0% |
IN8 936 OUT262 | |
| ✓ | GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 936 OUT262 |
Agentic - Game Backend: Health bar segments
1 passed |
100.0% |
IN8 937 OUT289 | |
| ✓ | GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 937 OUT289 |
Agentic - Game Backend: High-score table insert
1 failed |
0.0% |
IN37 607 OUT1 359 | |
| × | GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.4 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Insert did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:39: Unexpected character '#'.
1:75: Unexpected character '`'.
1:81: Unexpected character '`'.
1:91: Unexpected character '`'.
1:101: Unexpected character '`'.
1:118: Unexpected character '`'.
1:125: Unexpected character '`'.
1:130: Unexpected character '`'.
1:135: Unexpected character '`'.
1:182: Unexpected character '`'.
1:189: Unexpected character '`'.
1:210: Unexpected character '`'.
1:236: Unexpected character '`'.
1:248: Unexpected character '`'.
1:256: Unexpected character '`'.
1:398: Unexpected character '`'.
1:405: Unexpected character '`'.
3:80: Unexpected character '`'.
3:115: Unexpected character '`'.
3:138: Unexpected character '`'.
3:145: Unexpected character '`'.
5:34: Unexpected character '#'.
5:103: Unexpected character '`'.
5:120: Unexpected character '`'.
5:177: Unexpected character '#'.
5:184: Char literals must contain exactly one character and close with '''. |
Failed | IN37 607 OUT1 359 |
Agentic - Game Backend: Hit chance
1 passed |
100.0% |
IN8 875 OUT59 | |
| ✓ | GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 875 OUT59 |
Agentic - Game Backend: Inventory encumbrance
1 passed |
100.0% |
IN8 872 OUT149 | |
| ✓ | GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 872 OUT149 |
Agentic - Game Backend: Inventory first-fit placement
1 passed |
100.0% |
IN8 938 OUT213 | |
| ✓ | GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 938 OUT213 |
Agentic - Game Backend: Inventory stack split
1 passed |
100.0% |
IN8 887 OUT46 | |
| ✓ | GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 887 OUT46 |
Agentic - Game Backend: Isometric projection
1 passed |
100.0% |
IN8 905 OUT96 | |
| ✓ | GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 905 OUT96 |
Agentic - Game Backend: Jump buffer
1 passed |
100.0% |
IN8 893 OUT137 | |
| ✓ | GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 893 OUT137 |
Agentic - Game Backend: Knockback with arena clamp
1 passed |
100.0% |
IN8 932 OUT212 | |
| ✓ | GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 932 OUT212 |
Agentic - Game Backend: Leaderboard rank
1 passed |
100.0% |
IN8 865 OUT103 | |
| ✓ | GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 865 OUT103 |
Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over)
1 passed |
100.0% |
IN9 016 OUT210 | |
| ✓ | GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN9 016 OUT210 |
Agentic - Game Backend: Lifesteal healing
1 passed |
100.0% |
IN8 876 OUT127 | |
| ✓ | GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 876 OUT127 |
Agentic - Game Backend: Lights Out toggle
1 passed |
100.0% |
IN8 892 OUT238 | |
| ✓ | GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 892 OUT238 |
Agentic - Game Backend: Match-3 cascade
1 failed |
0.0% |
IN18 950 OUT1 826 | |
| × | GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: Resolve 5x4 [33113133323222333331] got 10 expected 13
Values:
r.ResultJson = "10" |
Failed | IN18 950 OUT1 826 |
Agentic - Game Backend: Match-3 detection
1 passed |
100.0% |
IN8 914 OUT511 | |
| ✓ | GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 914 OUT511 |
Agentic - Game Backend: Match-3 swap validity
1 passed |
100.0% |
IN8 923 OUT493 | |
| ✓ | GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 923 OUT493 |
Agentic - Game Backend: Memory match pairs
1 passed |
100.0% |
IN8 877 OUT90 | |
| ✓ | GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 877 OUT90 |
Agentic - Game Backend: Mercy invincibility (i-frames)
1 passed |
100.0% |
IN8 932 OUT210 | |
| ✓ | GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 932 OUT210 |
Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal)
1 passed |
100.0% |
IN18 697 OUT1 197 | |
| ✓ | GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 697 OUT1 197 |
Agentic - Game Backend: Minimap downsample
1 passed |
100.0% |
IN8 908 OUT221 | |
| ✓ | GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 908 OUT221 |
Agentic - Game Backend: Minimap marker position
1 passed |
100.0% |
IN8 911 OUT140 | |
| ✓ | GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 911 OUT140 |
Agentic - Game Backend: N-in-a-row win detection (Connect-4)
1 failed |
0.0% |
IN39 992 OUT2 479 | |
| × | GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.4 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Winner did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
1:21: Unexpected character '#'.
1:28: Char literals must contain exactly one character and close with '''. |
Failed | IN39 992 OUT2 479 |
Agentic - Game Backend: Nonogram row validation
1 passed |
100.0% |
IN8 888 OUT289 | |
| ✓ | GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 888 OUT289 |
Agentic - Game Backend: OBB collision (rotating rectangles, SAT)
1 passed |
100.0% |
IN19 394 OUT2 328 | |
| ✓ | GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN19 394 OUT2 328 |
Agentic - Game Backend: Particle lifetime & expiry
1 passed |
100.0% |
IN8 970 OUT190 | |
| ✓ | GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 970 OUT190 |
Agentic - Game Backend: Party XP share
1 passed |
100.0% |
IN8 888 OUT111 | |
| ✓ | GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 888 OUT111 |
Agentic - Game Backend: Patrol waypoints (ping-pong)
1 passed |
100.0% |
IN8 911 OUT128 | |
| ✓ | GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 911 OUT128 |
Agentic - Game Backend: Platformer jump arc
1 passed |
100.0% |
IN8 902 OUT102 | |
| ✓ | GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 902 OUT102 |
Agentic - Game Backend: Point on circle (trig)
1 passed |
100.0% |
IN8 935 OUT124 | |
| ✓ | GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 935 OUT124 |
Agentic - Game Backend: Poker hand comparison
1 passed |
100.0% |
IN46 847 OUT6 971 | |
| ✓ | GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.4 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users |
Passed | IN46 847 OUT6 971 |
Agentic - Game Backend: Poker hand ranking
1 passed |
100.0% |
IN18 624 OUT1 371 | |
| ✓ | GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 624 OUT1 371 |
Agentic - Game Backend: Pop-cap training
1 passed |
100.0% |
IN8 889 OUT95 | |
| ✓ | GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 889 OUT95 |
Agentic - Game Backend: Progress bar fill
1 passed |
100.0% |
IN8 871 OUT58 | |
| ✓ | GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 871 OUT58 |
Agentic - Game Backend: Projectile range (physics/trig)
1 passed |
100.0% |
IN8 893 OUT92 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 893 OUT92 |
Agentic - Game Backend: Projectile with ground bounce
1 passed |
100.0% |
IN8 927 OUT131 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 927 OUT131 |
Agentic - Game Backend: Resource gathering
1 passed |
100.0% |
IN8 888 OUT40 | |
| ✓ | GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 888 OUT40 |
Agentic - Game Backend: Reversi flip count
1 passed |
100.0% |
IN18 459 OUT818 | |
| ✓ | GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 459 OUT818 |
Agentic - Game Backend: Rotate 2D vector (trig)
1 passed |
100.0% |
IN8 904 OUT115 | |
| ✓ | GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 904 OUT115 |
Agentic - Game Backend: Round-robin deal
1 passed |
100.0% |
IN8 893 OUT85 | |
| ✓ | GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 893 OUT85 |
Agentic - Game Backend: RTS resource economy
1 passed |
100.0% |
IN8 884 OUT100 | |
| ✓ | GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 884 OUT100 |
Agentic - Game Backend: RTS unit counters & damage
1 passed |
100.0% |
IN8 936 OUT171 | |
| ✓ | GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 936 OUT171 |
Agentic - Game Backend: Safest neighbour (danger map)
1 passed |
100.0% |
IN18 117 OUT807 | |
| ✓ | GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 117 OUT807 |
Agentic - Game Backend: Scroll viewport
1 passed |
100.0% |
IN8 906 OUT73 | |
| ✓ | GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 906 OUT73 |
Agentic - Game Backend: Set card validity
1 passed |
100.0% |
IN8 885 OUT164 | |
| ✓ | GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 885 OUT164 |
Agentic - Game Backend: Shop affordability
1 passed |
100.0% |
IN8 865 OUT62 | |
| ✓ | GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 865 OUT62 |
Agentic - Game Backend: Sliding puzzle move
1 passed |
100.0% |
IN8 896 OUT286 | |
| ✓ | GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 896 OUT286 |
Agentic - Game Backend: Snake head movement
1 passed |
100.0% |
IN8 953 OUT277 | |
| ✓ | GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 953 OUT277 |
Agentic - Game Backend: Snake tick
1 failed |
0.0% |
IN18 450 OUT705 | |
| × | GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Tick [1,0;0,0] R food(9,9) got "2,0;1,0;0,0" expected "2,0;1,0"
Values:
r.ResultJson = "\"2,0;1,0;0,0\""
exp = "\"2,0;1,0\"" |
Failed | IN18 450 OUT705 |
Agentic - Game Backend: Snap to grid
1 passed |
100.0% |
IN8 881 OUT63 | |
| ✓ | GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 881 OUT63 |
Agentic - Game Backend: Sokoban move/push legality
1 passed |
100.0% |
IN8 984 OUT244 | |
| ✓ | GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 984 OUT244 |
Agentic - Game Backend: Solitaire tableau move
1 passed |
100.0% |
IN8 923 OUT210 | |
| ✓ | GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 923 OUT210 |
Agentic - Game Backend: Spawn scheduler count
1 passed |
100.0% |
IN8 880 OUT82 | |
| ✓ | GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 880 OUT82 |
Agentic - Game Backend: Spawn-wave alive count
1 passed |
100.0% |
IN8 892 OUT139 | |
| ✓ | GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 892 OUT139 |
Agentic - Game Backend: Stat level scaling
1 passed |
100.0% |
IN8 884 OUT65 | |
| ✓ | GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 884 OUT65 |
Agentic - Game Backend: Steering seek (greedy pursuit)
1 passed |
100.0% |
IN8 910 OUT135 | |
| ✓ | GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 910 OUT135 |
Agentic - Game Backend: Tactical movement range
1 passed |
100.0% |
IN38 848 OUT1 989 | |
| ✓ | GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.4 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN38 848 OUT1 989 |
Agentic - Game Backend: Tech tree eligibility
1 passed |
100.0% |
IN8 904 OUT210 | |
| ✓ | GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 904 OUT210 |
Agentic - Game Backend: Tetris line clearing
1 passed |
100.0% |
IN8 895 OUT289 | |
| ✓ | GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 895 OUT289 |
Agentic - Game Backend: Threat table top target
1 passed |
100.0% |
IN8 889 OUT302 | |
| ✓ | GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 889 OUT302 |
Agentic - Game Backend: Tic-tac-toe result
1 passed |
100.0% |
IN8 880 OUT291 | |
| ✓ | GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 880 OUT291 |
Agentic - Game Backend: Toroidal wrap (Pac-Man)
1 passed |
100.0% |
IN8 916 OUT60 | |
| ✓ | GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 916 OUT60 |
Agentic - Game Backend: Tower targeting (nearest in range)
1 passed |
100.0% |
IN8 904 OUT203 | |
| ✓ | GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 904 OUT203 |
Agentic - Game Backend: Transaction ledger
1 passed |
100.0% |
IN8 907 OUT213 | |
| ✓ | GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 907 OUT213 |
Agentic - Game Backend: Trick-taking winner
1 passed |
100.0% |
IN8 935 OUT503 | |
| ✓ | GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 935 OUT503 |
Agentic - Game Backend: Turn-based duel resolution
1 passed |
100.0% |
IN8 967 OUT287 | |
| ✓ | GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 967 OUT287 |
Agentic - Game Backend: Typed armor mitigation
1 passed |
100.0% |
IN8 891 OUT364 | |
| ✓ | GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 891 OUT364 |
Agentic - Game Backend: UI menu (wrap/clamp/paginate)
1 passed |
100.0% |
IN8 983 OUT266 | |
| ✓ | GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 983 OUT266 |
Agentic - Game Backend: Uno playable cards
1 passed |
100.0% |
IN8 919 OUT195 | |
| ✓ | GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 919 OUT195 |
Agentic - Game Backend: Velocity reflection
1 passed |
100.0% |
IN8 863 OUT114 | |
| ✓ | GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 863 OUT114 |
Agentic - Game Backend: Wall-slide fall
1 passed |
100.0% |
IN8 871 OUT73 | |
| ✓ | GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 871 OUT73 |
Agentic - Game Backend: War card compare
1 passed |
100.0% |
IN8 891 OUT232 | |
| ✓ | GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 891 OUT232 |
Agentic - Game Backend: Weighted loot table
1 passed |
100.0% |
IN8 894 OUT99 | |
| ✓ | GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 894 OUT99 |
Agentic - Game Backend: Weighted terrain pathfinding
1 passed |
100.0% |
IN40 408 OUT3 257 | |
| ✓ | GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.4 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN40 408 OUT3 257 |
Agentic - Game Backend: Word-wrap line count
1 passed |
100.0% |
IN8 876 OUT168 | |
| ✓ | GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 876 OUT168 |
Agentic - Game Backend: XP to character level
1 passed |
100.0% |
IN8 884 OUT88 | |
| ✓ | GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 884 OUT88 |
Agentic - Idempotency (poison-retry / no double charge)
1 passed |
100.0% |
IN14 365 OUT403 | |
| ✓ | AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN14 365 OUT403 |
Agentic - Irreversible Commit (measure twice)
1 passed |
100.0% |
IN15 966 OUT413 | |
| ✓ | AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query
7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN15 966 OUT413 |
Agentic - Least Privilege (exact grants)
1 passed |
100.0% |
IN11 652 OUT304 | |
| ✓ | AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query
5 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN11 652 OUT304 |
Agentic - Ledger (simulated DB read-modify-write)
1 failed |
0.0% |
IN39 877 OUT1 648 | |
| × | AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query
16 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: actual == expected
Context: Account alice ended at 600 but expected 0 (in-order apply with overdraft-skip).
Values:
actual = 600
expected = 0 |
Failed | IN39 877 OUT1 648 |
Agentic - Ledger variants (5 RMW cases)
5 passed |
100.0% |
IN162 972 OUT6 996 | |
| ✓ | LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN16 080 OUT919 |
| ✓ | LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query
17 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN48 961 OUT1 933 |
| ✓ | LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN17 242 OUT1 403 |
| ✓ | LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query
18 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN60 238 OUT1 700 |
| ✓ | LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN20 451 OUT1 041 |
Agentic - Lock Ordering (deadlock avoidance)
1 passed |
100.0% |
IN67 223 OUT1 762 | |
| ✓ | AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query
25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN67 223 OUT1 762 |
Agentic - Lock Ordering variants (5 cases)
5 passed |
100.0% |
IN298 746 OUT8 258 | |
| ✓ | LockVariantFiveResFourOps1 AI query
23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN67 092 OUT1 728 |
| ✓ | LockVariantFourResThreeOps1 AI query
18 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN49 075 OUT1 312 |
| ✓ | LockVariantHeavyReuse1 AI query
23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN67 093 OUT1 722 |
| ✓ | LockVariantReverseListed1 AI query
25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN64 423 OUT1 939 |
| ✓ | LockVariantThreeLockOp1 AI query
20 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN51 063 OUT1 557 |
Agentic - Migration (cascading rollback + reapply)
1 passed |
100.0% |
IN21 240 OUT853 | |
| ✓ | AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query
11 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN21 240 OUT853 |
Agentic - Requirements (detect contradiction)
1 passed |
100.0% |
IN7 457 OUT376 | |
| ✓ | AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query
2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN7 457 OUT376 |
Agentic - Retry/Circuit-breaker (transient vs hard-down)
1 passed |
100.0% |
IN26 529 OUT800 | |
| ✓ | AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query
10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN26 529 OUT800 |
Agentic - Saga (compensation / distributed rollback)
1 passed |
100.0% |
IN21 324 OUT304 | |
| ✓ | AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN21 324 OUT304 |
Agentic - Saga variants (5 compensation cases)
5 passed |
100.0% |
IN102 077 OUT2 101 | |
| ✓ | SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query
4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN13 553 OUT228 |
| ✓ | SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query
4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN13 749 OUT318 |
| ✓ | SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query
7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN23 658 OUT530 |
| ✓ | SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN20 060 OUT377 |
| ✓ | SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query
9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN31 057 OUT648 |
Agentic - Schema Compat (expand-contract)
1 passed |
100.0% |
IN21 563 OUT511 | |
| ✓ | AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN21 563 OUT511 |
Agentic - Scratchpad (self-repair, Dyck, tool-decision)
7 passed |
100.0% |
IN227 086 OUT8 839 | |
| ✓ | AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query
1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN22 187 OUT909 |
| ✓ | AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query
4 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN55 297 OUT779 |
| ✓ | ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query
0 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN10 658 OUT27 |
| ✓ | ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query
1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN21 664 OUT470 |
| ✓ | ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query
1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN21 480 OUT269 |
| ✓ | ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query
4 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN62 437 OUT5 581 |
| ✓ | ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query
2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN33 363 OUT804 |
Agentic - Secret Rotation (zero-downtime ordering)
1 passed |
100.0% |
IN16 479 OUT259 | |
| ✓ | AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 479 OUT259 |
Agentic - StatefulOps (incident recovery, concurrency replan)
2 passed |
100.0% |
IN54 395 OUT1 442 | |
| ✓ | AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query
7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN22 504 OUT662 |
| ✓ | AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query
8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN31 891 OUT780 |
Agentic - Test Authoring (write a test that catches the bug)
1 passed |
100.0% |
IN16 854 OUT471 | |
| ✓ | AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query
5 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN16 854 OUT471 |
Agentic - Topological Planning (deps + prereq)
1 passed |
100.0% |
IN25 457 OUT760 | |
| ✓ | AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN25 457 OUT760 |
Algorithms - 0/1 Knapsack (dynamic programming)
1 passed |
100.0% |
IN18 133 OUT650 | |
| ✓ | KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN18 133 OUT650 |
Algorithms - 1D elastic collision
1 passed |
100.0% |
IN8 938 OUT179 | |
| ✓ | ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 938 OUT179 |
Algorithms - All-pairs shortest path pair
1 passed |
100.0% |
IN8 906 OUT480 | |
| ✓ | FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 906 OUT480 |
Algorithms - Balanced brackets (stack)
1 passed |
100.0% |
IN8 869 OUT217 | |
| ✓ | ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 869 OUT217 |
Algorithms - Bellman-Ford shortest path (neg weights/cycles)
1 passed |
100.0% |
IN8 915 OUT1 094 | |
| ✓ | BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 915 OUT1 094 |
Algorithms - Binary Gray code
1 passed |
100.0% |
IN8 842 OUT22 | |
| ✓ | GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 842 OUT22 |
Algorithms - Bipartite graph check
1 failed |
0.0% |
IN8 888 OUT386 | |
| × | BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: IsBipartite(4,[0,1;1,2;2,3;3,0]) got false expected true
Values:
r.ResultJson = "false"
exp = "true" |
Failed | IN8 888 OUT386 |
Algorithms - Bit tricks (popcount/pow2/lowbit)
1 passed |
100.0% |
IN8 916 OUT124 | |
| ✓ | BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 916 OUT124 |
Algorithms - Caesar cipher
1 passed |
100.0% |
IN8 865 OUT202 | |
| ✓ | CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 865 OUT202 |
Algorithms - Catalan number
1 passed |
100.0% |
IN8 887 OUT84 | |
| ✓ | CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 887 OUT84 |
Algorithms - Chinese remainder theorem
1 passed |
100.0% |
IN8 905 OUT757 | |
| ✓ | CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 905 OUT757 |
Algorithms - Circle-circle MTV
1 passed |
100.0% |
IN8 982 OUT157 | |
| ✓ | CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 982 OUT157 |
Algorithms - Coin change combinations
1 passed |
100.0% |
IN18 074 OUT631 | |
| ✓ | CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN18 074 OUT631 |
Algorithms - Collatz step count
1 passed |
100.0% |
IN8 868 OUT92 | |
| ✓ | CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 868 OUT92 |
Algorithms - Connected components (union-find)
1 failed |
0.0% |
IN37 711 OUT1 338 | |
| × | UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.4 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users CountComponents did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
3:8: Expected a variable name, found '['. |
Failed | IN37 711 OUT1 338 |
Algorithms - Container with most water
1 passed |
100.0% |
IN8 876 OUT225 | |
| ✓ | ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 876 OUT225 |
Algorithms - Count anagram groups
1 passed |
100.0% |
IN8 859 OUT144 | |
| ✓ | AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 859 OUT144 |
Algorithms - Count grid color regions
1 passed |
100.0% |
IN8 879 OUT444 | |
| ✓ | CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 879 OUT444 |
Algorithms - Count islands
1 passed |
100.0% |
IN8 882 OUT501 | |
| ✓ | CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 882 OUT501 |
Algorithms - Count overlapping occurrences
1 passed |
100.0% |
IN8 865 OUT102 | |
| ✓ | RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 865 OUT102 |
Algorithms - Count primes (Sieve of Eratosthenes)
1 passed |
100.0% |
IN8 844 OUT134 | |
| ✓ | CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 844 OUT134 |
Algorithms - Cumulative distribution
1 passed |
100.0% |
IN8 866 OUT246 | |
| ✓ | CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 866 OUT246 |
Algorithms - Dijkstra shortest path (weighted graph)
1 passed |
100.0% |
IN8 918 OUT526 | |
| ✓ | DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 918 OUT526 |
Algorithms - Directed cycle detection
1 failed |
0.0% |
IN38 122 OUT1 576 | |
| × | DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.4 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users HasCycle did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
3:40: Expected '(' after the List type. Lists are created with 'new List<T>()' or 'new List<T>(items)', found '['. |
Failed | IN38 122 OUT1 576 |
Algorithms - Distinct subsequences
1 passed |
100.0% |
IN8 861 OUT314 | |
| ✓ | DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 861 OUT314 |
Algorithms - Divisor count
1 passed |
100.0% |
IN8 849 OUT68 | |
| ✓ | DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 849 OUT68 |
Algorithms - Downsample by block average
1 passed |
100.0% |
IN8 890 OUT209 | |
| ✓ | ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 890 OUT209 |
Algorithms - Dutch national flag sort
1 passed |
100.0% |
IN8 857 OUT272 | |
| ✓ | DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 857 OUT272 |
Algorithms - Egg drop trials
1 failed |
0.0% |
IN8 918 OUT238 | |
| × | EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: MinTrials(2,10) got 10 expected 4
Values:
r.ResultJson = "10" |
Failed | IN8 918 OUT238 |
Algorithms - Equal subset-sum partition
1 passed |
100.0% |
IN8 860 OUT257 | |
| ✓ | PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 860 OUT257 |
Algorithms - Euler totient
1 passed |
100.0% |
IN8 849 OUT205 | |
| ✓ | EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 849 OUT205 |
Algorithms - Eulerian path existence
1 passed |
100.0% |
IN18 341 OUT947 | |
| ✓ | EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN18 341 OUT947 |
Algorithms - Extended Euclidean
1 passed |
100.0% |
IN8 961 OUT189 | |
| ✓ | ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 961 OUT189 |
Algorithms - Format duration H:MM:SS
1 passed |
100.0% |
IN8 876 OUT184 | |
| ✓ | FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 876 OUT184 |
Algorithms - GCD and LCM
1 passed |
100.0% |
IN17 876 OUT229 | |
| ✓ | GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN17 876 OUT229 |
Algorithms - Grid transpose
1 passed |
100.0% |
IN8 884 OUT86 | |
| ✓ | TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 884 OUT86 |
Algorithms - House robber (max non-adjacent sum)
1 passed |
100.0% |
IN8 848 OUT217 | |
| ✓ | HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 848 OUT217 |
Algorithms - Integer linear interpolation
1 passed |
100.0% |
IN8 882 OUT35 | |
| ✓ | LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 882 OUT35 |
Algorithms - Integer square root
1 passed |
100.0% |
IN8 864 OUT153 | |
| ✓ | IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 864 OUT153 |
Algorithms - Integer to base-N string
1 passed |
100.0% |
IN8 870 OUT107 | |
| ✓ | BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 870 OUT107 |
Algorithms - Integer to Roman numeral
1 passed |
100.0% |
IN8 899 OUT515 | |
| ✓ | RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 899 OUT515 |
Algorithms - Josephus survivor
1 passed |
100.0% |
IN8 875 OUT237 | |
| ✓ | JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 875 OUT237 |
Algorithms - Jump game (min jumps)
1 passed |
100.0% |
IN8 883 OUT238 | |
| ✓ | JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 883 OUT238 |
Algorithms - Kth largest element
1 passed |
100.0% |
IN8 873 OUT94 | |
| ✓ | KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 873 OUT94 |
Algorithms - Least-squares line fit
1 passed |
100.0% |
IN8 915 OUT263 | |
| ✓ | LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 915 OUT263 |
Algorithms - Levenshtein edit distance (DP)
1 passed |
100.0% |
IN8 843 OUT307 | |
| ✓ | EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 843 OUT307 |
Algorithms - Log-scale bucket index
1 passed |
100.0% |
IN8 908 OUT116 | |
| ✓ | LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 908 OUT116 |
Algorithms - Logarithmic bin index
1 passed |
100.0% |
IN8 897 OUT86 | |
| ✓ | LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 897 OUT86 |
Algorithms - Longest common prefix
1 passed |
100.0% |
IN8 856 OUT175 | |
| ✓ | CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 856 OUT175 |
Algorithms - Longest common subsequence
1 passed |
100.0% |
IN8 841 OUT246 | |
| ✓ | LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 841 OUT246 |
Algorithms - Longest common substring
1 passed |
100.0% |
IN8 858 OUT179 | |
| ✓ | CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 858 OUT179 |
Algorithms - Longest increasing subsequence (DP)
1 passed |
100.0% |
IN8 875 OUT302 | |
| ✓ | LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 875 OUT302 |
Algorithms - Longest palindromic substring
1 passed |
100.0% |
IN8 853 OUT258 | |
| ✓ | LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 853 OUT258 |
Algorithms - Longest repeated substring
1 failed |
0.0% |
IN8 858 OUT180 | |
| × | LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: Length([aaaa]) got 2 expected 3
Values:
r.ResultJson = "2" |
Failed | IN8 858 OUT180 |
Algorithms - Lower bound (binary search)
1 passed |
100.0% |
IN8 886 OUT150 | |
| ✓ | LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 886 OUT150 |
Algorithms - Luhn checksum validation
1 passed |
100.0% |
IN8 887 OUT188 | |
| ✓ | LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 887 OUT188 |
Algorithms - Majority element
1 passed |
100.0% |
IN8 851 OUT254 | |
| ✓ | MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 851 OUT254 |
Algorithms - Matrix chain multiplication
1 passed |
100.0% |
IN8 886 OUT364 | |
| ✓ | MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 886 OUT364 |
Algorithms - Maximum subarray sum (Kadane)
1 passed |
100.0% |
IN8 876 OUT131 | |
| ✓ | MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 876 OUT131 |
Algorithms - Merge K sorted lists
1 passed |
100.0% |
IN8 869 OUT225 | |
| ✓ | MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 869 OUT225 |
Algorithms - Merge overlapping intervals
1 passed |
100.0% |
IN18 284 OUT905 | |
| ✓ | MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN18 284 OUT905 |
Algorithms - Min-max normalization
1 passed |
100.0% |
IN8 878 OUT281 | |
| ✓ | MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 878 OUT281 |
Algorithms - Minimum coin change (DP)
1 passed |
100.0% |
IN8 869 OUT288 | |
| ✓ | CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 869 OUT288 |
Algorithms - Minimum knight moves (BFS)
1 passed |
100.0% |
IN28 126 OUT1 274 | |
| ✓ | KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.3 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN28 126 OUT1 274 |
Algorithms - Minimum path sum (grid DP)
1 passed |
100.0% |
IN8 882 OUT350 | |
| ✓ | MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 882 OUT350 |
Algorithms - Minimum spanning tree (Kruskal)
1 passed |
100.0% |
IN18 406 OUT1 229 | |
| ✓ | KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN18 406 OUT1 229 |
Algorithms - Minimum window substring
1 passed |
100.0% |
IN8 890 OUT462 | |
| ✓ | MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 890 OUT462 |
Algorithms - Modular exponentiation (fast pow)
1 passed |
100.0% |
IN8 901 OUT142 | |
| ✓ | ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 901 OUT142 |
Algorithms - Modular inverse (extended Euclid)
1 passed |
100.0% |
IN27 770 OUT976 | |
| ✓ | ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.3 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN27 770 OUT976 |
Algorithms - Moving average
1 passed |
100.0% |
IN8 880 OUT230 | |
| ✓ | MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 880 OUT230 |
Algorithms - Next greater element
1 passed |
100.0% |
IN8 863 OUT323 | |
| ✓ | NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 863 OUT323 |
Algorithms - Next lexicographic permutation
1 passed |
100.0% |
IN8 871 OUT324 | |
| ✓ | NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 871 OUT324 |
Algorithms - Nth Fibonacci mod m
1 passed |
100.0% |
IN8 870 OUT119 | |
| ✓ | FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 870 OUT119 |
Algorithms - Overlapping substring count
1 passed |
100.0% |
IN8 855 OUT96 | |
| ✓ | SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 855 OUT96 |
Algorithms - Palindrome & anagram checks
1 passed |
100.0% |
IN8 885 OUT642 | |
| ✓ | PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 885 OUT642 |
Algorithms - Pascal triangle row
1 passed |
100.0% |
IN8 863 OUT148 | |
| ✓ | PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 863 OUT148 |
Algorithms - Point-to-segment distance
1 passed |
100.0% |
IN27 791 OUT953 | |
| ✓ | PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.3 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN27 791 OUT953 |
Algorithms - Prime factorization
1 passed |
100.0% |
IN8 874 OUT131 | |
| ✓ | PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 874 OUT131 |
Algorithms - Product of array except self
1 passed |
100.0% |
IN18 117 OUT652 | |
| ✓ | ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN18 117 OUT652 |
Algorithms - Quadratic Bezier point
1 passed |
100.0% |
IN8 940 OUT149 | |
| ✓ | BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 940 OUT149 |
Algorithms - Range sum via prefix sums
1 passed |
100.0% |
IN8 862 OUT69 | |
| ✓ | PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 862 OUT69 |
Algorithms - Reverse words & Hamming distance
1 passed |
100.0% |
IN8 866 OUT110 | |
| ✓ | ReverseWordsCodeGenCode-gen: model implements reverse-words + Hamming; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 866 OUT110 |
Algorithms - Rod cutting
1 passed |
100.0% |
IN8 896 OUT224 | |
| ✓ | RodCuttingCodeGenCode-gen: model implements rod cutting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 896 OUT224 |
Algorithms - Roman numeral to integer
1 passed |
100.0% |
IN8 885 OUT220 | |
| ✓ | RomanToIntCodeGenCode-gen: model implements Roman-to-int; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 885 OUT220 |
Algorithms - Rotate array right by k
1 passed |
100.0% |
IN8 889 OUT215 | |
| ✓ | RotateArrayCodeGenCode-gen: model implements right rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 889 OUT215 |
Algorithms - Rotate grid 90 CW (discriminator)
1 failed |
0.0% |
IN8 892 OUT112 | |
| × | RotateMatrixCodeGenCode-gen: model implements 90-degree CW rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Rotate90 2x3 [abcdef] got "badcfe" expected "ecafdb"
Values:
r.ResultJson = "\"badcfe\""
exp = "\"ecafdb\"" |
Failed | IN8 892 OUT112 |
Algorithms - RPN evaluator
1 passed |
100.0% |
IN8 894 OUT233 | |
| ✓ | RpnCodeGenCode-gen: model implements an RPN evaluator; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 894 OUT233 |
Algorithms - Run-length compression (if shorter)
1 passed |
100.0% |
IN8 866 OUT168 | |
| ✓ | StringCompressCodeGenCode-gen: model implements string compression; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 866 OUT168 |
Algorithms - Run-length encoding (encode/decode)
1 passed |
100.0% |
IN8 912 OUT299 | |
| ✓ | RunLengthCodeGenCode-gen: model implements RLE encode+decode; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 912 OUT299 |
Algorithms - Semantic versioning (compare + bump)
1 passed |
100.0% |
IN8 942 OUT363 | |
| ✓ | SemVerCodeGenCode-gen: model implements semver compare + bump; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:InstructionFollowinglevel:Easysource:Ai4Users |
Passed | IN8 942 OUT363 |
Algorithms - Single-pile Nim (optimal play)
1 passed |
100.0% |
IN8 874 OUT33 | |
| ✓ | NimCodeGenCode-gen: model implements optimal Nim; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 874 OUT33 |
Algorithms - Sliding window maximum
1 passed |
100.0% |
IN8 866 OUT251 | |
| ✓ | SlidingWindowMaxCodeGenCode-gen: model implements sliding window maximum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 866 OUT251 |
Algorithms - Spiral matrix traversal
1 passed |
100.0% |
IN8 875 OUT276 | |
| ✓ | SpiralMatrixCodeGenCode-gen: model implements spiral traversal; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 875 OUT276 |
Algorithms - Square matrix multiply
1 passed |
100.0% |
IN8 867 OUT294 | |
| ✓ | MatMulCodeGenCode-gen: model implements square matrix multiply; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 867 OUT294 |
Algorithms - Strongly connected components
1 passed |
100.0% |
IN8 878 OUT562 | |
| ✓ | SccCountCodeGenCode-gen: model implements SCC count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 878 OUT562 |
Algorithms - Subset sum reachability
1 passed |
100.0% |
IN18 041 OUT463 | |
| ✓ | SubsetSumCodeGenCode-gen: model implements subset-sum reachability; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN18 041 OUT463 |
Algorithms - Substring search (KMP)
1 passed |
100.0% |
IN27 657 OUT902 | |
| ✓ | KmpCodeGenCode-gen: model implements KMP search; graded vs exact oracle.3 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN27 657 OUT902 |
Algorithms - Sum of digits & digital root
1 passed |
100.0% |
IN8 852 OUT143 | |
| ✓ | DigitalRootCodeGenCode-gen: model implements sum-of-digits + digital root; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 852 OUT143 |
Algorithms - Temperature conversion
1 passed |
100.0% |
IN8 863 OUT55 | |
| ✓ | TemperatureCodeGenCode-gen: model implements temperature conversion; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 863 OUT55 |
Algorithms - Topological sort (lexicographically smallest)
1 passed |
100.0% |
IN8 904 OUT429 | |
| ✓ | TopoCodeGenCode-gen: model implements lexicographically-smallest topological sort.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 904 OUT429 |
Algorithms - Trailing zeros of factorial
1 passed |
100.0% |
IN8 864 OUT79 | |
| ✓ | TrailingZerosCodeGenCode-gen: model implements trailing zeros; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 864 OUT79 |
Algorithms - Trapping rain water
1 passed |
100.0% |
IN8 863 OUT364 | |
| ✓ | TrappingRainCodeGenCode-gen: model implements trapping rain water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 863 OUT364 |
Algorithms - Two-sum indices
1 passed |
100.0% |
IN8 870 OUT132 | |
| ✓ | TwoSumCodeGenCode-gen: model implements two-sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 870 OUT132 |
Algorithms - Unique grid paths
1 passed |
100.0% |
IN8 845 OUT292 | |
| ✓ | UniquePathsCodeGenCode-gen: model implements unique paths; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 845 OUT292 |
Algorithms - Valid Sudoku unit
1 passed |
100.0% |
IN8 850 OUT70 | |
| ✓ | SudokuUnitCodeGenCode-gen: model implements Sudoku unit validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 850 OUT70 |
Algorithms - Wildcard string matching
1 passed |
100.0% |
IN8 868 OUT256 | |
| ✓ | WildcardMatchCodeGenCode-gen: model implements wildcard match; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 868 OUT256 |
Algorithms - Z-function
1 passed |
100.0% |
IN8 869 OUT299 | |
| ✓ | ZFunctionCodeGenCode-gen: model implements the Z-function; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 869 OUT299 |
Database - Conditional aggregation pivot
1 passed |
100.0% |
IN947 OUT64 | |
| ✓ | PivotCountsCodeGenCode-gen: model writes the pivot query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN947 OUT64 |
Database - Count distinct opponents
1 passed |
100.0% |
IN927 OUT19 | |
| ✓ | DistinctCountCodeGenCode-gen: model writes the COUNT(DISTINCT) query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Easysource:Ai4Users |
Passed | IN927 OUT19 |
Database - Date-range filter (BETWEEN)
1 passed |
100.0% |
IN957 OUT43 | |
| ✓ | DateRangeCodeGenCode-gen: model writes the date-range query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN957 OUT43 |
Database - EXISTS subquery (teams with matches)
1 passed |
100.0% |
IN934 OUT32 | |
| ✓ | ExistsSubqueryCodeGenCode-gen: model writes the EXISTS query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN934 OUT32 |
Database - JOIN player totals (two tables)
1 passed |
100.0% |
IN956 OUT38 | |
| ✓ | JoinScoresCodeGenCode-gen: model writes a JOIN + aggregate query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN956 OUT38 |
Database - LEFT JOIN anti-join (no items)
1 passed |
100.0% |
IN930 OUT28 | |
| ✓ | LeftJoinNullsCodeGenCode-gen: model writes the anti-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN930 OUT28 |
Database - Player stats SQL (multi-aggregate, QueryJson)
1 passed |
100.0% |
IN960 OUT36 | |
| ✓ | PlayerStatsCodeGenCode-gen: model writes a multi-aggregate stats query; graded by comparing its QueryJson output to the oracle query's (arbitrary shape, no struct).1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN960 OUT36 |
Database - Players above average score (subquery)
1 passed |
100.0% |
IN925 OUT27 | |
| ✓ | AboveAverageCodeGenCode-gen: model writes an above-average query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN925 OUT27 |
Database - Players who never played (anti-join)
1 passed |
100.0% |
IN922 OUT30 | |
| ✓ | NeverPlayedCodeGenCode-gen: model writes an anti-join query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN922 OUT30 |
Database - RANK window function
1 passed |
100.0% |
IN952 OUT27 | |
| ✓ | WindowRankCodeGenCode-gen: model writes the RANK() query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN952 OUT27 |
Database - Running total (window function)
1 passed |
100.0% |
IN949 OUT24 | |
| ✓ | RunningTotalCodeGenCode-gen: model writes the window running-total query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN949 OUT24 |
Database - Scoreboard SQL (players with >1 win, HAVING)
1 passed |
100.0% |
IN956 OUT34 | |
| ✓ | WinFilterCodeGenCode-gen: model writes a HAVING query; runs under try/catch with self-repair; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN956 OUT34 |
Database - Scoreboard SQL (top scorers query)
1 passed |
100.0% |
IN957 OUT25 | |
| ✓ | ScoreboardTopScorersCodeGenCode-gen: model writes a scoreboard SQL query; runs under try/catch with a self-repair loop on SQL errors; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN957 OUT25 |
Database - Self-join same-team pairs
1 passed |
100.0% |
IN940 OUT44 | |
| ✓ | SelfJoinCodeGenCode-gen: model writes the self-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN940 OUT44 |
Database - Top scorer per region
1 passed |
100.0% |
IN942 OUT34 | |
| ✓ | TopPerGroupCodeGenCode-gen: model writes the per-group top query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN942 OUT34 |
Database - UPDATE restock (SQL write)
1 passed |
100.0% |
IN934 OUT17 | |
| ✓ | UpdateInventoryCodeGenCode-gen: model writes an UPDATE; applied under try/catch, resulting state compared (via QueryJson) to the oracle update's state.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN934 OUT17 |
Database - Upsert with max merge
1 passed |
100.0% |
IN1 988 OUT72 | |
| ✓ | UpsertCodeGenCode-gen: model writes the upsert; executed on a fresh DB and the table compared.2 AI queries
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN1 988 OUT72 |
SDLC - Acceptance criteria check
1 passed |
100.0% |
IN8 893 OUT550 | |
| ✓ | SdlcAcceptanceCodeGenCode-gen: model implements acceptance-criteria check; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 893 OUT550 |
SDLC - Breaking API change detection
1 passed |
100.0% |
IN8 890 OUT346 | |
| ✓ | SdlcApiBreakingCodeGenCode-gen: model implements API-breaking detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 890 OUT346 |
SDLC - Condense without losing facts
1 passed |
100.0% |
IN1 048 OUT60 | |
| ✓ | NoInfoLossCodeGenRedaction test: model condenses keeping facts; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 048 OUT60 |
SDLC - Config merge with override
1 passed |
100.0% |
IN8 884 OUT306 | |
| ✓ | ConfigMergeCodeGenCode-gen: model implements config merge; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 884 OUT306 |
SDLC - Extract fields into fixed template
1 passed |
100.0% |
IN1 004 OUT22 | |
| ✓ | FieldExtractCodeGenRedaction test: model extracts fields into the template; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 004 OUT22 |
SDLC - Locate conflicting requirement pair
1 failed |
0.0% |
IN18 799 OUT1 546 | |
| × | SdlcConstraintConflictCodeGenCode-gen: model locates the conflicting requirement pair; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users FindConflict threw: unsupported operator |
Failed | IN18 799 OUT1 546 |
SDLC - Locate redundant requirement
1 failed |
0.0% |
IN8 931 OUT868 | |
| × | SdlcRedundantRequirementCodeGenCode-gen: model locates a redundant requirement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users FindRedundant threw: Cannot parse '= 7' as int. |
Failed | IN8 931 OUT868 |
SDLC - Mask PII to placeholder
1 passed |
100.0% |
IN972 OUT42 | |
| ✓ | MaskPiiCodeGenRedaction test: model masks PII to a placeholder; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN972 OUT42 |
SDLC - Modules on a dependency cycle
1 passed |
100.0% |
IN8 896 OUT489 | |
| ✓ | DependencyCycleModulesCodeGenCode-gen: model implements dependency-cycle detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN8 896 OUT489 |
SDLC - Remove duplicate requirements
1 passed |
100.0% |
IN945 OUT10 | |
| ✓ | DeduplicateCodeGenRedaction test: model removes duplicate requirements; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN945 OUT10 |
SDLC - Renumber requirements sequentially
1 passed |
100.0% |
IN1 012 OUT42 | |
| ✓ | RenumberCodeGenRedaction test: model renumbers sequentially; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 012 OUT42 |
SDLC - Reorganize notes into section template
1 passed |
100.0% |
IN1 126 OUT94 | |
| ✓ | RewriteSectionsCodeGenRedaction test: model reorganizes the notes; deterministic checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 126 OUT94 |
SDLC - Required semver bump from changelog
1 passed |
100.0% |
IN8 889 OUT205 | |
| ✓ | SemverBumpCodeGenCode-gen: model implements the semver-bump rule; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Easysource:Ai4Users |
Passed | IN8 889 OUT205 |
SDLC - Requirement-to-test traceability
1 passed |
100.0% |
IN8 911 OUT188 | |
| ✓ | SdlcTraceabilityCodeGenCode-gen: model implements traceability gap; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 911 OUT188 |
SDLC - Requirements completeness (spec coverage gap)
1 passed |
100.0% |
IN8 906 OUT293 | |
| ✓ | SdlcSpecCoverageCodeGenCode-gen: model implements spec-gap detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 906 OUT293 |
SDLC - Requirements consistency (contradiction detection)
1 failed |
0.0% |
IN39 412 OUT2 625 | |
| × | ConsistencyCodeGenCode-gen: model implements requirements-consistency checking; graded vs oracle.4 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users Consistent did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
48:14: Expected a variable name, found '['. |
Failed | IN39 412 OUT2 625 |
SDLC - Reshape prose into JSON template
1 passed |
100.0% |
IN1 014 OUT28 | |
| ✓ | JsonShapeCodeGenRedaction test: model reshapes prose into JSON; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 014 OUT28 |
SDLC - Reshape rows into Markdown table
1 passed |
100.0% |
IN968 OUT31 | |
| ✓ | TableFormatCodeGenRedaction test: model builds a Markdown table; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN968 OUT31 |
SDLC - Rewrite in imperative requirement style
1 passed |
100.0% |
IN991 OUT50 | |
| ✓ | StyleRewriteCodeGenRedaction test: model rewrites notes in imperative style; checker + self-review loop.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN991 OUT50 |
SDLC - Roll up estimates over task tree
1 passed |
100.0% |
IN8 902 OUT222 | |
| ✓ | EstimateRollupCodeGenCode-gen: model implements estimate roll-up over a task tree; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 902 OUT222 |
SDLC - State-machine reachability
1 passed |
100.0% |
IN8 888 OUT337 | |
| ✓ | SdlcStateReachableCodeGenCode-gen: model implements state reachability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 888 OUT337 |
SDLC - Ticket priority sort
1 passed |
100.0% |
IN8 881 OUT404 | |
| ✓ | SdlcPrioritySortCodeGenCode-gen: model implements priority sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 881 OUT404 |
SDLC - Unify glossary term consistency
1 passed |
100.0% |
IN972 OUT38 | |
| ✓ | GlossaryCodeGenRedaction test: model unifies synonyms to the canonical term; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN972 OUT38 |