Benchmark summary
A custom Ai4Users benchmark suite for SC#, covering code generation, reasoning, and stateful agentic workflows where models use tools, observe results, adapt their actions, recover from failures, and verify the final state.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Agentic - Blue-Green Deploy (health-gated cutover)
1 passed |
100.0% |
IN14 565 OUT1 357 | |
| ✓ | AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query
4 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN14 565 OUT1 357 |
Agentic - Bugfix (root-cause vs regression)
1 passed |
100.0% |
IN172 517 OUT5 998 | |
| ✓ | AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query
14 tool callscategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN172 517 OUT5 998 |
Agentic - Cache Invalidation (exact set)
1 passed |
100.0% |
IN14 313 OUT1 919 | |
| ✓ | AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query
7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN14 313 OUT1 919 |
Agentic - Canary (metric-gated release)
1 passed |
100.0% |
IN38 777 OUT3 774 | |
| ✓ | AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query
11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN38 777 OUT3 774 |
Agentic - Canary variants (5 metric cases)
5 passed |
100.0% |
IN162 272 OUT15 564 | |
| ✓ | CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN26 006 OUT2 596 |
| ✓ | CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query
11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN38 000 OUT3 451 |
| ✓ | CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN29 771 OUT3 076 |
| ✓ | CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN42 543 OUT4 169 |
| ✓ | CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN25 952 OUT2 272 |
Agentic - Code Review (find injected defect)
1 passed |
100.0% |
IN15 767 OUT1 649 | |
| ✓ | AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 767 OUT1 649 |
Agentic - Config Drift (remediate exact)
1 passed |
100.0% |
IN11 823 OUT781 | |
| ✓ | AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query
6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN11 823 OUT781 |
Agentic - Constraints (multi-section satisfaction)
1 passed |
100.0% |
IN11 149 OUT1 774 | |
| ✓ | AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 149 OUT1 774 |
Agentic - Constraints variants (5 cases)
5 passed |
100.0% |
IN47 555 OUT4 195 | |
| ✓ | ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 558 OUT1 291 |
| ✓ | ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 447 OUT523 |
| ✓ | ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 490 OUT837 |
| ✓ | ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 555 OUT750 |
| ✓ | ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 505 OUT794 |
Agentic - Data Reconciliation (per-field precedence)
1 passed |
100.0% |
IN13 382 OUT1 546 | |
| ✓ | AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN13 382 OUT1 546 |
Agentic - Diagnosis (root cause under budget)
1 passed |
100.0% |
IN12 234 OUT1 165 | |
| ✓ | AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN12 234 OUT1 165 |
Agentic - Diagnosis variants (5 cases)
5 passed |
100.0% |
IN76 131 OUT7 665 | |
| ✓ | DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN14 895 OUT2 067 |
| ✓ | DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 191 OUT1 369 |
| ✓ | DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 115 OUT1 335 |
| ✓ | DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 027 OUT1 436 |
| ✓ | DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 903 OUT1 458 |
Agentic - ETL Repair (root-cause in stage 2)
1 passed |
100.0% |
IN22 518 OUT1 529 | |
| ✓ | AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query
14 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN22 518 OUT1 529 |
Agentic - Feature Flag (staged rollout + kill-switch)
1 passed |
100.0% |
IN23 153 OUT2 086 | |
| ✓ | AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN23 153 OUT2 086 |
Agentic - Game Backend: 1D bouncing ball
1 passed |
100.0% |
IN8 781 OUT1 750 | |
| ✓ | GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 781 OUT1 750 |
Agentic - Game Backend: 1D swept anti-tunneling
1 passed |
100.0% |
IN8 792 OUT3 454 | |
| ✓ | GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 792 OUT3 454 |
Agentic - Game Backend: 2048 left-move (slide + merge)
1 passed |
100.0% |
IN8 801 OUT3 305 | |
| ✓ | GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 801 OUT3 305 |
Agentic - Game Backend: 8-direction from vector
1 passed |
100.0% |
IN8 808 OUT2 692 | |
| ✓ | GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 808 OUT2 692 |
Agentic - Game Backend: AABB collision resolution (MTV)
1 passed |
100.0% |
IN8 897 OUT4 462 | |
| ✓ | GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 897 OUT4 462 |
Agentic - Game Backend: AABB containment
1 passed |
100.0% |
IN8 737 OUT1 286 | |
| ✓ | GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 737 OUT1 286 |
Agentic - Game Backend: Acceleration to max speed
1 passed |
100.0% |
IN8 739 OUT825 | |
| ✓ | GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 739 OUT825 |
Agentic - Game Backend: Achievement thresholds
1 passed |
100.0% |
IN8 731 OUT1 768 | |
| ✓ | GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 731 OUT1 768 |
Agentic - Game Backend: Active status effects
1 passed |
100.0% |
IN8 753 OUT2 424 | |
| ✓ | GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT2 424 |
Agentic - Game Backend: Animation frame at time
1 passed |
100.0% |
IN8 771 OUT1 198 | |
| ✓ | GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 771 OUT1 198 |
Agentic - Game Backend: Army upkeep income
1 passed |
100.0% |
IN8 719 OUT920 | |
| ✓ | GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 719 OUT920 |
Agentic - Game Backend: Axis-aligned path follow
1 passed |
100.0% |
IN8 762 OUT5 523 | |
| ✓ | GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 762 OUT5 523 |
Agentic - Game Backend: Battleship shot resolution
1 passed |
100.0% |
IN8 821 OUT4 182 | |
| ✓ | GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 821 OUT4 182 |
Agentic - Game Backend: Bounce restitution
1 passed |
100.0% |
IN8 758 OUT935 | |
| ✓ | GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 758 OUT935 |
Agentic - Game Backend: Bresenham line rasterization
1 passed |
100.0% |
IN27 469 OUT9 998 | |
| ✓ | GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.3 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN27 469 OUT9 998 |
Agentic - Game Backend: Buff/debuff stacking
1 passed |
100.0% |
IN17 850 OUT4 926 | |
| ✓ | GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 850 OUT4 926 |
Agentic - Game Backend: Build queue completions
1 passed |
100.0% |
IN8 752 OUT2 150 | |
| ✓ | GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 752 OUT2 150 |
Agentic - Game Backend: Camera-relative movement
1 passed |
100.0% |
IN17 866 OUT4 355 | |
| ✓ | GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 866 OUT4 355 |
Agentic - Game Backend: Card hand sort
1 passed |
100.0% |
IN8 760 OUT3 490 | |
| ✓ | GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 760 OUT3 490 |
Agentic - Game Backend: Card-game rules (blackjack/deck/compare)
1 passed |
100.0% |
IN8 935 OUT5 803 | |
| ✓ | GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 935 OUT5 803 |
Agentic - Game Backend: Chart scaling & histogram bins
1 passed |
100.0% |
IN8 824 OUT2 100 | |
| ✓ | GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 824 OUT2 100 |
Agentic - Game Backend: Checkers captures
1 passed |
100.0% |
IN8 794 OUT2 554 | |
| ✓ | GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 794 OUT2 554 |
Agentic - Game Backend: Collision primitives (circle/aabb/segment)
1 passed |
100.0% |
IN19 279 OUT7 308 | |
| ✓ | GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN19 279 OUT7 308 |
Agentic - Game Backend: Combat turn order (initiative)
1 failed |
0.0% |
IN8 733 OUT3 796 | |
| × | GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: TurnOrder([alice,10;bob,20;carol,20]) got "alice bob carol" expected "bob carol alice"
Values:
r.ResultJson = "\"alice bob carol\""
exp = "\"bob carol alice\"" |
Failed | IN8 733 OUT3 796 |
Agentic - Game Backend: Combo scoring (rising multiplier)
1 passed |
100.0% |
IN8 756 OUT1 657 | |
| ✓ | GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 756 OUT1 657 |
Agentic - Game Backend: Control-point capture
1 passed |
100.0% |
IN8 769 OUT2 893 | |
| ✓ | GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT2 893 |
Agentic - Game Backend: Controls & Inventory (spec-driven function battery)
1 passed |
100.0% |
IN8 910 OUT1 670 | |
| ✓ | GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 910 OUT1 670 |
Agentic - Game Backend: Cooldowns & resource regen
1 passed |
100.0% |
IN8 827 OUT1 484 | |
| ✓ | GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 827 OUT1 484 |
Agentic - Game Backend: Coyote-time jump
1 passed |
100.0% |
IN8 748 OUT860 | |
| ✓ | GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 748 OUT860 |
Agentic - Game Backend: Crafting capacity
1 passed |
100.0% |
IN17 999 OUT6 681 | |
| ✓ | GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 999 OUT6 681 |
Agentic - Game Backend: Cribbage fifteens
1 passed |
100.0% |
IN8 749 OUT3 364 | |
| ✓ | GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 749 OUT3 364 |
Agentic - Game Backend: Critical-hit damage
1 passed |
100.0% |
IN8 739 OUT710 | |
| ✓ | GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 739 OUT710 |
Agentic - Game Backend: Damage over time
1 passed |
100.0% |
IN8 765 OUT2 137 | |
| ✓ | GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 765 OUT2 137 |
Agentic - Game Backend: Day/night cycle phase
1 passed |
100.0% |
IN8 751 OUT1 383 | |
| ✓ | GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 751 OUT1 383 |
Agentic - Game Backend: Dice notation max roll
1 passed |
100.0% |
IN8 766 OUT5 386 | |
| ✓ | GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 766 OUT5 386 |
Agentic - Game Backend: Enemy AI state machine
1 passed |
100.0% |
IN8 852 OUT4 108 | |
| ✓ | GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 852 OUT4 108 |
Agentic - Game Backend: Equipment slot validation
1 passed |
100.0% |
IN8 766 OUT2 058 | |
| ✓ | GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 766 OUT2 058 |
Agentic - Game Backend: Flood-fill region size
1 passed |
100.0% |
IN8 754 OUT3 049 | |
| ✓ | GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT3 049 |
Agentic - Game Backend: Fog of war visibility
1 passed |
100.0% |
IN17 874 OUT4 920 | |
| ✓ | GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN17 874 OUT4 920 |
Agentic - Game Backend: Fog reveal (Chebyshev radius)
1 passed |
100.0% |
IN8 767 OUT4 190 | |
| ✓ | GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 767 OUT4 190 |
Agentic - Game Backend: Friction stop frame
1 passed |
100.0% |
IN8 747 OUT2 005 | |
| ✓ | GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 747 OUT2 005 |
Agentic - Game Backend: Game of Life (code-gen vs oracle)
1 passed |
100.0% |
IN8 806 OUT3 413 | |
| ✓ | GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 806 OUT3 413 |
Agentic - Game Backend: Greedy coin change
1 passed |
100.0% |
IN17 679 OUT4 395 | |
| ✓ | GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN17 679 OUT4 395 |
Agentic - Game Backend: Grid inventory & stacking (Diablo-style)
1 passed |
100.0% |
IN18 117 OUT6 420 | |
| ✓ | GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 117 OUT6 420 |
Agentic - Game Backend: Grid pathfinding (shortest path, BFS)
1 failed |
0.0% |
IN8 836 OUT1 489 | |
| × | GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.1 AI query
0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: ShortestPath mismatch grid 4x4 (0,0)->(3,3) got -1 expected 6
Values:
r.ResultJson = "-1" |
Failed | IN8 836 OUT1 489 |
Agentic - Game Backend: Grid raycast
1 passed |
100.0% |
IN8 804 OUT3 502 | |
| ✓ | GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 804 OUT3 502 |
Agentic - Game Backend: Health bar segments
1 passed |
100.0% |
IN8 805 OUT2 679 | |
| ✓ | GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 805 OUT2 679 |
Agentic - Game Backend: High-score table insert
1 passed |
100.0% |
IN17 784 OUT8 034 | |
| ✓ | GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 784 OUT8 034 |
Agentic - Game Backend: Hit chance
1 passed |
100.0% |
IN8 743 OUT670 | |
| ✓ | GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 743 OUT670 |
Agentic - Game Backend: Inventory encumbrance
1 passed |
100.0% |
IN8 740 OUT2 252 | |
| ✓ | GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 740 OUT2 252 |
Agentic - Game Backend: Inventory first-fit placement
1 passed |
100.0% |
IN8 806 OUT3 283 | |
| ✓ | GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 806 OUT3 283 |
Agentic - Game Backend: Inventory stack split
1 passed |
100.0% |
IN8 755 OUT1 495 | |
| ✓ | GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 755 OUT1 495 |
Agentic - Game Backend: Isometric projection
1 passed |
100.0% |
IN8 773 OUT863 | |
| ✓ | GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 773 OUT863 |
Agentic - Game Backend: Jump buffer
1 passed |
100.0% |
IN8 761 OUT2 083 | |
| ✓ | GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT2 083 |
Agentic - Game Backend: Knockback with arena clamp
1 passed |
100.0% |
IN8 800 OUT2 489 | |
| ✓ | GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 800 OUT2 489 |
Agentic - Game Backend: Leaderboard rank
1 passed |
100.0% |
IN8 733 OUT2 112 | |
| ✓ | GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 733 OUT2 112 |
Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over)
1 passed |
100.0% |
IN8 884 OUT1 736 | |
| ✓ | GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 884 OUT1 736 |
Agentic - Game Backend: Lifesteal healing
1 passed |
100.0% |
IN8 744 OUT2 354 | |
| ✓ | GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 744 OUT2 354 |
Agentic - Game Backend: Lights Out toggle
1 failed |
0.0% |
IN8 760 OUT4 716 | |
| × | GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Toggle 4 [1001001100011000] (0,2) got "1001101011010000" expected "1001101111010000"
Values:
r.ResultJson = "\"1001101011010000\""
exp = "\"1001101111010000\"" |
Failed | IN8 760 OUT4 716 |
Agentic - Game Backend: Match-3 cascade
1 passed |
100.0% |
IN8 818 OUT4 017 | |
| ✓ | GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 818 OUT4 017 |
Agentic - Game Backend: Match-3 detection
1 passed |
100.0% |
IN8 782 OUT4 666 | |
| ✓ | GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 782 OUT4 666 |
Agentic - Game Backend: Match-3 swap validity
1 passed |
100.0% |
IN8 791 OUT3 993 | |
| ✓ | GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 791 OUT3 993 |
Agentic - Game Backend: Memory match pairs
1 passed |
100.0% |
IN8 745 OUT2 831 | |
| ✓ | GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 745 OUT2 831 |
Agentic - Game Backend: Mercy invincibility (i-frames)
1 passed |
100.0% |
IN8 800 OUT3 351 | |
| ✓ | GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 800 OUT3 351 |
Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal)
1 passed |
100.0% |
IN18 439 OUT10 101 | |
| ✓ | GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 439 OUT10 101 |
Agentic - Game Backend: Minimap downsample
1 passed |
100.0% |
IN8 776 OUT2 304 | |
| ✓ | GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 776 OUT2 304 |
Agentic - Game Backend: Minimap marker position
1 passed |
100.0% |
IN8 779 OUT600 | |
| ✓ | GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 779 OUT600 |
Agentic - Game Backend: N-in-a-row win detection (Connect-4)
1 passed |
100.0% |
IN18 189 OUT8 347 | |
| ✓ | GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 189 OUT8 347 |
Agentic - Game Backend: Nonogram row validation
1 passed |
100.0% |
IN8 756 OUT3 474 | |
| ✓ | GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT3 474 |
Agentic - Game Backend: OBB collision (rotating rectangles, SAT)
1 passed |
100.0% |
IN8 903 OUT8 886 | |
| ✓ | GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 903 OUT8 886 |
Agentic - Game Backend: Particle lifetime & expiry
1 passed |
100.0% |
IN8 838 OUT1 933 | |
| ✓ | GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 838 OUT1 933 |
Agentic - Game Backend: Party XP share
1 passed |
100.0% |
IN8 756 OUT2 137 | |
| ✓ | GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 756 OUT2 137 |
Agentic - Game Backend: Patrol waypoints (ping-pong)
1 passed |
100.0% |
IN8 779 OUT2 802 | |
| ✓ | GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 779 OUT2 802 |
Agentic - Game Backend: Platformer jump arc
1 passed |
100.0% |
IN8 770 OUT974 | |
| ✓ | GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 770 OUT974 |
Agentic - Game Backend: Point on circle (trig)
1 passed |
100.0% |
IN8 803 OUT709 | |
| ✓ | GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 803 OUT709 |
Agentic - Game Backend: Poker hand comparison
1 broken |
0.0% |
IN19 138 OUT18 103 | |
| ! | GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users repair response incomplete: AI output was marked broken by AiRunner. The response text is partial and must not be treated as a complete answer. |
Broken | IN19 138 OUT18 103 |
Agentic - Game Backend: Poker hand ranking
1 passed |
100.0% |
IN8 822 OUT6 197 | |
| ✓ | GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 822 OUT6 197 |
Agentic - Game Backend: Pop-cap training
1 passed |
100.0% |
IN8 757 OUT907 | |
| ✓ | GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 757 OUT907 |
Agentic - Game Backend: Progress bar fill
1 passed |
100.0% |
IN8 739 OUT1 133 | |
| ✓ | GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 739 OUT1 133 |
Agentic - Game Backend: Projectile range (physics/trig)
1 passed |
100.0% |
IN8 761 OUT2 089 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT2 089 |
Agentic - Game Backend: Projectile with ground bounce
1 passed |
100.0% |
IN8 795 OUT1 541 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 795 OUT1 541 |
Agentic - Game Backend: Resource gathering
1 passed |
100.0% |
IN8 756 OUT707 | |
| ✓ | GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 756 OUT707 |
Agentic - Game Backend: Reversi flip count
1 passed |
100.0% |
IN8 827 OUT3 429 | |
| ✓ | GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 827 OUT3 429 |
Agentic - Game Backend: Rotate 2D vector (trig)
1 passed |
100.0% |
IN8 772 OUT1 847 | |
| ✓ | GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT1 847 |
Agentic - Game Backend: Round-robin deal
1 passed |
100.0% |
IN8 761 OUT1 503 | |
| ✓ | GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 761 OUT1 503 |
Agentic - Game Backend: RTS resource economy
1 passed |
100.0% |
IN8 752 OUT1 003 | |
| ✓ | GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 752 OUT1 003 |
Agentic - Game Backend: RTS unit counters & damage
1 passed |
100.0% |
IN8 804 OUT1 627 | |
| ✓ | GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 804 OUT1 627 |
Agentic - Game Backend: Safest neighbour (danger map)
1 passed |
100.0% |
IN8 792 OUT3 759 | |
| ✓ | GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 792 OUT3 759 |
Agentic - Game Backend: Scroll viewport
1 passed |
100.0% |
IN8 774 OUT2 185 | |
| ✓ | GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 774 OUT2 185 |
Agentic - Game Backend: Set card validity
1 passed |
100.0% |
IN8 753 OUT906 | |
| ✓ | GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT906 |
Agentic - Game Backend: Shop affordability
1 passed |
100.0% |
IN8 733 OUT972 | |
| ✓ | GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 733 OUT972 |
Agentic - Game Backend: Sliding puzzle move
1 passed |
100.0% |
IN8 764 OUT6 200 | |
| ✓ | GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 764 OUT6 200 |
Agentic - Game Backend: Snake head movement
1 passed |
100.0% |
IN8 821 OUT2 052 | |
| ✓ | GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 821 OUT2 052 |
Agentic - Game Backend: Snake tick
1 failed |
0.0% |
IN27 953 OUT9 879 | |
| × | GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.3 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Tick [1,0;0,0] R food(9,9) got "2,0" expected "2,0;1,0"
Values:
r.ResultJson = "\"2,0\""
exp = "\"2,0;1,0\"" |
Failed | IN27 953 OUT9 879 |
Agentic - Game Backend: Snap to grid
1 passed |
100.0% |
IN8 749 OUT1 274 | |
| ✓ | GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 749 OUT1 274 |
Agentic - Game Backend: Sokoban move/push legality
1 passed |
100.0% |
IN8 852 OUT1 519 | |
| ✓ | GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 852 OUT1 519 |
Agentic - Game Backend: Solitaire tableau move
1 passed |
100.0% |
IN8 791 OUT3 002 | |
| ✓ | GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 791 OUT3 002 |
Agentic - Game Backend: Spawn scheduler count
1 passed |
100.0% |
IN8 748 OUT1 482 | |
| ✓ | GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 748 OUT1 482 |
Agentic - Game Backend: Spawn-wave alive count
1 passed |
100.0% |
IN8 760 OUT2 172 | |
| ✓ | GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 760 OUT2 172 |
Agentic - Game Backend: Stat level scaling
1 passed |
100.0% |
IN8 752 OUT1 036 | |
| ✓ | GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT1 036 |
Agentic - Game Backend: Steering seek (greedy pursuit)
1 passed |
100.0% |
IN8 778 OUT2 770 | |
| ✓ | GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 778 OUT2 770 |
Agentic - Game Backend: Tactical movement range
1 passed |
100.0% |
IN8 790 OUT5 661 | |
| ✓ | GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 790 OUT5 661 |
Agentic - Game Backend: Tech tree eligibility
1 passed |
100.0% |
IN8 772 OUT3 021 | |
| ✓ | GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT3 021 |
Agentic - Game Backend: Tetris line clearing
1 passed |
100.0% |
IN8 763 OUT3 685 | |
| ✓ | GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 763 OUT3 685 |
Agentic - Game Backend: Threat table top target
1 passed |
100.0% |
IN8 757 OUT3 075 | |
| ✓ | GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 757 OUT3 075 |
Agentic - Game Backend: Tic-tac-toe result
1 passed |
100.0% |
IN8 748 OUT4 246 | |
| ✓ | GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 748 OUT4 246 |
Agentic - Game Backend: Toroidal wrap (Pac-Man)
1 passed |
100.0% |
IN17 685 OUT2 391 | |
| ✓ | GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 685 OUT2 391 |
Agentic - Game Backend: Tower targeting (nearest in range)
1 passed |
100.0% |
IN8 772 OUT2 353 | |
| ✓ | GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT2 353 |
Agentic - Game Backend: Transaction ledger
1 passed |
100.0% |
IN8 775 OUT2 747 | |
| ✓ | GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 775 OUT2 747 |
Agentic - Game Backend: Trick-taking winner
1 passed |
100.0% |
IN8 803 OUT4 359 | |
| ✓ | GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 803 OUT4 359 |
Agentic - Game Backend: Turn-based duel resolution
1 passed |
100.0% |
IN8 835 OUT3 717 | |
| ✓ | GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 835 OUT3 717 |
Agentic - Game Backend: Typed armor mitigation
1 passed |
100.0% |
IN17 937 OUT6 328 | |
| ✓ | GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 937 OUT6 328 |
Agentic - Game Backend: UI menu (wrap/clamp/paginate)
1 passed |
100.0% |
IN8 851 OUT1 711 | |
| ✓ | GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 851 OUT1 711 |
Agentic - Game Backend: Uno playable cards
1 passed |
100.0% |
IN8 787 OUT2 765 | |
| ✓ | GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 787 OUT2 765 |
Agentic - Game Backend: Velocity reflection
1 passed |
100.0% |
IN17 615 OUT3 769 | |
| ✓ | GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN17 615 OUT3 769 |
Agentic - Game Backend: Wall-slide fall
1 passed |
100.0% |
IN8 739 OUT858 | |
| ✓ | GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 739 OUT858 |
Agentic - Game Backend: War card compare
1 passed |
100.0% |
IN8 759 OUT3 459 | |
| ✓ | GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 759 OUT3 459 |
Agentic - Game Backend: Weighted loot table
1 passed |
100.0% |
IN8 762 OUT1 433 | |
| ✓ | GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 762 OUT1 433 |
Agentic - Game Backend: Weighted terrain pathfinding
1 passed |
100.0% |
IN18 416 OUT12 429 | |
| ✓ | GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 416 OUT12 429 |
Agentic - Game Backend: Word-wrap line count
1 passed |
100.0% |
IN8 744 OUT3 942 | |
| ✓ | GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT3 942 |
Agentic - Game Backend: XP to character level
1 passed |
100.0% |
IN8 752 OUT2 131 | |
| ✓ | GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT2 131 |
Agentic - Idempotency (poison-retry / no double charge)
1 passed |
100.0% |
IN14 683 OUT931 | |
| ✓ | AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query
8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN14 683 OUT931 |
Agentic - Irreversible Commit (measure twice)
1 passed |
100.0% |
IN15 630 OUT936 | |
| ✓ | AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query
7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN15 630 OUT936 |
Agentic - Least Privilege (exact grants)
1 passed |
100.0% |
IN11 501 OUT572 | |
| ✓ | AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query
5 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN11 501 OUT572 |
Agentic - Ledger (simulated DB read-modify-write)
1 passed |
100.0% |
IN12 352 OUT4 653 | |
| ✓ | AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 352 OUT4 653 |
Agentic - Ledger variants (5 RMW cases)
5 passed |
100.0% |
IN65 116 OUT19 393 | |
| ✓ | LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 159 OUT2 718 |
| ✓ | LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 533 OUT4 129 |
| ✓ | LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 906 OUT3 786 |
| ✓ | LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN15 794 OUT6 499 |
| ✓ | LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query
6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN11 724 OUT2 261 |
Agentic - Lock Ordering (deadlock avoidance)
1 passed |
100.0% |
IN87 389 OUT5 232 | |
| ✓ | AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query
25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN87 389 OUT5 232 |
Agentic - Lock Ordering variants (5 cases)
4 passed1 failed |
80.0% |
IN272 732 OUT43 774 | |
| × | LockVariantFiveResFourOps1 AI query
621 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: !Globals.Get<bool>("lock.held." + r, false)
Context: Lock left held at end: cache |
Failed | IN98 541 OUT29 345 |
| ✓ | LockVariantFourResThreeOps1 AI query
19 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN58 280 OUT5 743 |
| ✓ | LockVariantHeavyReuse1 AI query
23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN15 019 OUT2 145 |
| ✓ | LockVariantReverseListed1 AI query
24 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN52 051 OUT3 468 |
| ✓ | LockVariantThreeLockOp1 AI query
20 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN48 841 OUT3 073 |
Agentic - Migration (cascading rollback + reapply)
1 passed |
100.0% |
IN24 639 OUT2 371 | |
| ✓ | AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query
11 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN24 639 OUT2 371 |
Agentic - Requirements (detect contradiction)
1 passed |
100.0% |
IN7 149 OUT737 | |
| ✓ | AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query
2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN7 149 OUT737 |
Agentic - Retry/Circuit-breaker (transient vs hard-down)
1 passed |
100.0% |
IN34 384 OUT2 036 | |
| ✓ | AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query
10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN34 384 OUT2 036 |
Agentic - Saga (compensation / distributed rollback)
1 passed |
100.0% |
IN18 246 OUT934 | |
| ✓ | AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query
10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN18 246 OUT934 |
Agentic - Saga variants (5 compensation cases)
5 passed |
100.0% |
IN80 624 OUT3 109 | |
| ✓ | SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query
4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN13 397 OUT501 |
| ✓ | SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query
4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN13 447 OUT451 |
| ✓ | SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 773 OUT655 |
| ✓ | SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 773 OUT639 |
| ✓ | SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query
8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN20 234 OUT863 |
Agentic - Schema Compat (expand-contract)
1 passed |
100.0% |
IN17 669 OUT1 297 | |
| ✓ | AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN17 669 OUT1 297 |
Agentic - Scratchpad (self-repair, Dyck, tool-decision)
6 passed1 broken |
85.7% |
IN651 073 OUT108 688 | |
| ✓ | AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query
1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN21 390 OUT2 948 |
| ! | AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query
27 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: response.Completed
Context: AI response did not complete.
Values:
response.Completed = false |
Broken | IN397 047 OUT75 810 |
| ✓ | ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query
1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN21 000 OUT249 |
| ✓ | ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query
1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN23 088 OUT1 775 |
| ✓ | ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query
4 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN71 856 OUT2 629 |
| ✓ | ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query
3 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN46 402 OUT16 648 |
| ✓ | ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query
5 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN70 290 OUT8 629 |
Agentic - Secret Rotation (zero-downtime ordering)
1 passed |
100.0% |
IN16 327 OUT1 631 | |
| ✓ | AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 327 OUT1 631 |
Agentic - StatefulOps (incident recovery, concurrency replan)
2 passed |
100.0% |
IN74 044 OUT3 769 | |
| ✓ | AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query
11 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN42 570 OUT2 380 |
| ✓ | AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query
8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN31 474 OUT1 389 |
Agentic - Test Authoring (write a test that catches the bug)
1 passed |
100.0% |
IN11 288 OUT1 290 | |
| ✓ | AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query
6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 288 OUT1 290 |
Agentic - Topological Planning (deps + prereq)
1 passed |
100.0% |
IN24 839 OUT1 821 | |
| ✓ | AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN24 839 OUT1 821 |
Algorithms - 0/1 Knapsack (dynamic programming)
1 passed |
100.0% |
IN8 766 OUT3 113 | |
| ✓ | KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 766 OUT3 113 |
Algorithms - 1D elastic collision
1 passed |
100.0% |
IN8 806 OUT2 293 | |
| ✓ | ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 806 OUT2 293 |
Algorithms - All-pairs shortest path pair
1 passed |
100.0% |
IN39 275 OUT25 331 | |
| ✓ | FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.4 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN39 275 OUT25 331 |
Algorithms - Balanced brackets (stack)
1 passed |
100.0% |
IN8 737 OUT2 575 | |
| ✓ | ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 737 OUT2 575 |
Algorithms - Bellman-Ford shortest path (neg weights/cycles)
1 passed |
100.0% |
IN18 303 OUT11 589 | |
| ✓ | BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN18 303 OUT11 589 |
Algorithms - Binary Gray code
1 passed |
100.0% |
IN8 710 OUT484 | |
| ✓ | GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 710 OUT484 |
Algorithms - Bipartite graph check
1 passed |
100.0% |
IN17 990 OUT6 875 | |
| ✓ | BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN17 990 OUT6 875 |
Algorithms - Bit tricks (popcount/pow2/lowbit)
1 passed |
100.0% |
IN8 784 OUT2 457 | |
| ✓ | BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 784 OUT2 457 |
Algorithms - Caesar cipher
1 passed |
100.0% |
IN8 733 OUT3 215 | |
| ✓ | CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT3 215 |
Algorithms - Catalan number
1 passed |
100.0% |
IN8 755 OUT1 443 | |
| ✓ | CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT1 443 |
Algorithms - Chinese remainder theorem
1 passed |
100.0% |
IN8 773 OUT5 515 | |
| ✓ | CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 773 OUT5 515 |
Algorithms - Circle-circle MTV
1 passed |
100.0% |
IN8 850 OUT2 587 | |
| ✓ | CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 850 OUT2 587 |
Algorithms - Coin change combinations
1 passed |
100.0% |
IN8 740 OUT3 572 | |
| ✓ | CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 740 OUT3 572 |
Algorithms - Collatz step count
1 passed |
100.0% |
IN8 736 OUT1 171 | |
| ✓ | CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 736 OUT1 171 |
Algorithms - Connected components (union-find)
1 passed |
100.0% |
IN17 836 OUT8 542 | |
| ✓ | UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN17 836 OUT8 542 |
Algorithms - Container with most water
1 passed |
100.0% |
IN8 744 OUT2 331 | |
| ✓ | ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT2 331 |
Algorithms - Count anagram groups
1 passed |
100.0% |
IN8 727 OUT1 899 | |
| ✓ | AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 727 OUT1 899 |
Algorithms - Count grid color regions
1 passed |
100.0% |
IN8 747 OUT4 038 | |
| ✓ | CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 747 OUT4 038 |
Algorithms - Count islands
1 passed |
100.0% |
IN8 750 OUT4 533 | |
| ✓ | CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 750 OUT4 533 |
Algorithms - Count overlapping occurrences
1 passed |
100.0% |
IN8 733 OUT1 775 | |
| ✓ | RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT1 775 |
Algorithms - Count primes (Sieve of Eratosthenes)
1 passed |
100.0% |
IN8 712 OUT2 326 | |
| ✓ | CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 712 OUT2 326 |
Algorithms - Cumulative distribution
1 passed |
100.0% |
IN17 764 OUT6 002 | |
| ✓ | CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN17 764 OUT6 002 |
Algorithms - Dijkstra shortest path (weighted graph)
1 passed |
100.0% |
IN8 786 OUT4 796 | |
| ✓ | DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 786 OUT4 796 |
Algorithms - Directed cycle detection
1 passed |
100.0% |
IN8 743 OUT5 466 | |
| ✓ | DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 743 OUT5 466 |
Algorithms - Distinct subsequences
1 passed |
100.0% |
IN8 729 OUT2 129 | |
| ✓ | DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 729 OUT2 129 |
Algorithms - Divisor count
1 passed |
100.0% |
IN8 717 OUT1 556 | |
| ✓ | DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 717 OUT1 556 |
Algorithms - Downsample by block average
1 passed |
100.0% |
IN8 758 OUT4 621 | |
| ✓ | ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 758 OUT4 621 |
Algorithms - Dutch national flag sort
1 passed |
100.0% |
IN8 725 OUT4 223 | |
| ✓ | DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 725 OUT4 223 |
Algorithms - Egg drop trials
1 passed |
100.0% |
IN8 786 OUT5 281 | |
| ✓ | EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 786 OUT5 281 |
Algorithms - Equal subset-sum partition
1 passed |
100.0% |
IN8 728 OUT3 460 | |
| ✓ | PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 728 OUT3 460 |
Algorithms - Euler totient
1 passed |
100.0% |
IN8 717 OUT1 848 | |
| ✓ | EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 717 OUT1 848 |
Algorithms - Eulerian path existence
1 passed |
100.0% |
IN8 776 OUT2 647 | |
| ✓ | EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 776 OUT2 647 |
Algorithms - Extended Euclidean
1 passed |
100.0% |
IN17 909 OUT4 756 | |
| ✓ | ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN17 909 OUT4 756 |
Algorithms - Format duration H:MM:SS
1 passed |
100.0% |
IN8 744 OUT2 712 | |
| ✓ | FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 744 OUT2 712 |
Algorithms - GCD and LCM
1 passed |
100.0% |
IN8 721 OUT1 284 | |
| ✓ | GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 721 OUT1 284 |
Algorithms - Grid transpose
1 passed |
100.0% |
IN8 752 OUT4 117 | |
| ✓ | TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT4 117 |
Algorithms - House robber (max non-adjacent sum)
1 passed |
100.0% |
IN8 716 OUT2 304 | |
| ✓ | HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 716 OUT2 304 |
Algorithms - Integer linear interpolation
1 passed |
100.0% |
IN8 750 OUT1 235 | |
| ✓ | LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 750 OUT1 235 |
Algorithms - Integer square root
1 passed |
100.0% |
IN8 732 OUT3 445 | |
| ✓ | IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 732 OUT3 445 |
Algorithms - Integer to base-N string
1 passed |
100.0% |
IN8 738 OUT2 350 | |
| ✓ | BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 738 OUT2 350 |
Algorithms - Integer to Roman numeral
1 passed |
100.0% |
IN8 767 OUT1 811 | |
| ✓ | RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 767 OUT1 811 |
Algorithms - Josephus survivor
1 passed |
100.0% |
IN8 743 OUT1 536 | |
| ✓ | JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 743 OUT1 536 |
Algorithms - Jump game (min jumps)
1 passed |
100.0% |
IN8 751 OUT3 113 | |
| ✓ | JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 751 OUT3 113 |
Algorithms - Kth largest element
1 passed |
100.0% |
IN8 741 OUT1 267 | |
| ✓ | KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 741 OUT1 267 |
Algorithms - Least-squares line fit
1 passed |
100.0% |
IN8 783 OUT3 629 | |
| ✓ | LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 783 OUT3 629 |
Algorithms - Levenshtein edit distance (DP)
1 passed |
100.0% |
IN8 711 OUT2 605 | |
| ✓ | EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 711 OUT2 605 |
Algorithms - Log-scale bucket index
1 passed |
100.0% |
IN8 776 OUT1 908 | |
| ✓ | LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 776 OUT1 908 |
Algorithms - Logarithmic bin index
1 passed |
100.0% |
IN8 765 OUT1 837 | |
| ✓ | LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 765 OUT1 837 |
Algorithms - Longest common prefix
1 passed |
100.0% |
IN8 724 OUT2 095 | |
| ✓ | CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 724 OUT2 095 |
Algorithms - Longest common subsequence
1 passed |
100.0% |
IN8 709 OUT3 210 | |
| ✓ | LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 709 OUT3 210 |
Algorithms - Longest common substring
1 passed |
100.0% |
IN8 726 OUT3 164 | |
| ✓ | CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 726 OUT3 164 |
Algorithms - Longest increasing subsequence (DP)
1 passed |
100.0% |
IN8 743 OUT3 055 | |
| ✓ | LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 743 OUT3 055 |
Algorithms - Longest palindromic substring
1 passed |
100.0% |
IN8 721 OUT2 566 | |
| ✓ | LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 721 OUT2 566 |
Algorithms - Longest repeated substring
1 passed |
100.0% |
IN8 726 OUT1 833 | |
| ✓ | LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 726 OUT1 833 |
Algorithms - Lower bound (binary search)
1 passed |
100.0% |
IN8 754 OUT1 970 | |
| ✓ | LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT1 970 |
Algorithms - Luhn checksum validation
1 passed |
100.0% |
IN8 755 OUT1 769 | |
| ✓ | LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT1 769 |
Algorithms - Majority element
1 passed |
100.0% |
IN8 719 OUT2 807 | |
| ✓ | MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 719 OUT2 807 |
Algorithms - Matrix chain multiplication
1 failed |
0.0% |
IN8 754 OUT5 654 | |
| × | MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users MinMult threw: List index 5 is out of range. The list has 5 element(s). |
Failed | IN8 754 OUT5 654 |
Algorithms - Maximum subarray sum (Kadane)
1 passed |
100.0% |
IN8 744 OUT1 913 | |
| ✓ | MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT1 913 |
Algorithms - Merge K sorted lists
1 passed |
100.0% |
IN8 737 OUT3 296 | |
| ✓ | MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 737 OUT3 296 |
Algorithms - Merge overlapping intervals
1 passed |
100.0% |
IN18 018 OUT6 120 | |
| ✓ | MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN18 018 OUT6 120 |
Algorithms - Min-max normalization
1 passed |
100.0% |
IN8 746 OUT3 987 | |
| ✓ | MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 746 OUT3 987 |
Algorithms - Minimum coin change (DP)
1 passed |
100.0% |
IN8 737 OUT3 652 | |
| ✓ | CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 737 OUT3 652 |
Algorithms - Minimum knight moves (BFS)
1 failed |
0.0% |
IN17 964 OUT10 214 | |
| × | KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: MinMoves(8,0,0->2,2) got 3 expected 4
Values:
r.ResultJson = "3" |
Failed | IN17 964 OUT10 214 |
Algorithms - Minimum path sum (grid DP)
1 passed |
100.0% |
IN8 750 OUT2 810 | |
| ✓ | MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 750 OUT2 810 |
Algorithms - Minimum spanning tree (Kruskal)
1 passed |
100.0% |
IN18 195 OUT10 387 | |
| ✓ | KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN18 195 OUT10 387 |
Algorithms - Minimum window substring
1 passed |
100.0% |
IN8 758 OUT6 254 | |
| ✓ | MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 758 OUT6 254 |
Algorithms - Modular exponentiation (fast pow)
1 passed |
100.0% |
IN8 769 OUT2 114 | |
| ✓ | ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT2 114 |
Algorithms - Modular inverse (extended Euclid)
1 passed |
100.0% |
IN8 757 OUT2 524 | |
| ✓ | ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 757 OUT2 524 |
Algorithms - Moving average
1 passed |
100.0% |
IN8 748 OUT3 113 | |
| ✓ | MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 748 OUT3 113 |
Algorithms - Next greater element
1 passed |
100.0% |
IN8 731 OUT2 691 | |
| ✓ | NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 731 OUT2 691 |
Algorithms - Next lexicographic permutation
1 passed |
100.0% |
IN8 739 OUT4 502 | |
| ✓ | NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 739 OUT4 502 |
Algorithms - Nth Fibonacci mod m
1 passed |
100.0% |
IN8 738 OUT1 762 | |
| ✓ | FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 738 OUT1 762 |
Algorithms - Overlapping substring count
1 passed |
100.0% |
IN8 723 OUT1 703 | |
| ✓ | SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 723 OUT1 703 |
Algorithms - Palindrome & anagram checks
1 passed |
100.0% |
IN18 014 OUT7 987 | |
| ✓ | PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN18 014 OUT7 987 |
Algorithms - Pascal triangle row
1 passed |
100.0% |
IN8 731 OUT2 149 | |
| ✓ | PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 731 OUT2 149 |
Algorithms - Point-to-segment distance
1 passed |
100.0% |
IN8 771 OUT2 973 | |
| ✓ | PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 771 OUT2 973 |
Algorithms - Prime factorization
1 passed |
100.0% |
IN8 742 OUT3 246 | |
| ✓ | PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 742 OUT3 246 |
Algorithms - Product of array except self
1 passed |
100.0% |
IN17 841 OUT6 666 | |
| ✓ | ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN17 841 OUT6 666 |
Algorithms - Quadratic Bezier point
1 passed |
100.0% |
IN8 808 OUT2 013 | |
| ✓ | BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 808 OUT2 013 |
Algorithms - Range sum via prefix sums
1 passed |
100.0% |
IN8 730 OUT1 310 | |
| ✓ | PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 730 OUT1 310 |
Algorithms - Reverse words & Hamming distance
1 passed |
100.0% |
IN8 734 OUT2 227 | |
| ✓ | ReverseWordsCodeGenCode-gen: model implements reverse-words + Hamming; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 734 OUT2 227 |
Algorithms - Rod cutting
1 passed |
100.0% |
IN8 764 OUT4 274 | |
| ✓ | RodCuttingCodeGenCode-gen: model implements rod cutting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 764 OUT4 274 |
Algorithms - Roman numeral to integer
1 passed |
100.0% |
IN8 753 OUT2 945 | |
| ✓ | RomanToIntCodeGenCode-gen: model implements Roman-to-int; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT2 945 |
Algorithms - Rotate array right by k
1 passed |
100.0% |
IN8 757 OUT3 531 | |
| ✓ | RotateArrayCodeGenCode-gen: model implements right rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 757 OUT3 531 |
Algorithms - Rotate grid 90 CW (discriminator)
1 passed |
100.0% |
IN8 760 OUT3 735 | |
| ✓ | RotateMatrixCodeGenCode-gen: model implements 90-degree CW rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 760 OUT3 735 |
Algorithms - RPN evaluator
1 passed |
100.0% |
IN8 762 OUT3 208 | |
| ✓ | RpnCodeGenCode-gen: model implements an RPN evaluator; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 762 OUT3 208 |
Algorithms - Run-length compression (if shorter)
1 failed |
0.0% |
IN8 734 OUT2 861 | |
| × | StringCompressCodeGenCode-gen: model implements string compression; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Compress([aaabbbccccd]) got "aaabbbccccd" expected "a3b3c4d1"
Values:
r.ResultJson = "\"aaabbbccccd\""
exp = "\"a3b3c4d1\"" |
Failed | IN8 734 OUT2 861 |
Algorithms - Run-length encoding (encode/decode)
1 passed |
100.0% |
IN8 780 OUT2 930 | |
| ✓ | RunLengthCodeGenCode-gen: model implements RLE encode+decode; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 780 OUT2 930 |
Algorithms - Semantic versioning (compare + bump)
1 passed |
100.0% |
IN17 951 OUT4 659 | |
| ✓ | SemVerCodeGenCode-gen: model implements semver compare + bump; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:InstructionFollowinglevel:Easysource:Ai4Users |
Passed | IN17 951 OUT4 659 |
Algorithms - Single-pile Nim (optimal play)
1 passed |
100.0% |
IN8 742 OUT1 216 | |
| ✓ | NimCodeGenCode-gen: model implements optimal Nim; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 742 OUT1 216 |
Algorithms - Sliding window maximum
1 passed |
100.0% |
IN17 794 OUT7 316 | |
| ✓ | SlidingWindowMaxCodeGenCode-gen: model implements sliding window maximum; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN17 794 OUT7 316 |
Algorithms - Spiral matrix traversal
1 passed |
100.0% |
IN8 743 OUT3 123 | |
| ✓ | SpiralMatrixCodeGenCode-gen: model implements spiral traversal; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 743 OUT3 123 |
Algorithms - Square matrix multiply
1 passed |
100.0% |
IN8 735 OUT3 360 | |
| ✓ | MatMulCodeGenCode-gen: model implements square matrix multiply; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 735 OUT3 360 |
Algorithms - Strongly connected components
1 passed |
100.0% |
IN28 718 OUT19 088 | |
| ✓ | SccCountCodeGenCode-gen: model implements SCC count; graded vs exact oracle.3 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN28 718 OUT19 088 |
Algorithms - Subset sum reachability
1 passed |
100.0% |
IN8 719 OUT3 076 | |
| ✓ | SubsetSumCodeGenCode-gen: model implements subset-sum reachability; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 719 OUT3 076 |
Algorithms - Substring search (KMP)
1 failed |
0.0% |
IN8 741 OUT3 060 | |
| × | KmpCodeGenCode-gen: model implements KMP search; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: IndexOf([mississippi],[issip]) got -1 expected 4
Values:
r.ResultJson = "-1" |
Failed | IN8 741 OUT3 060 |
Algorithms - Sum of digits & digital root
1 passed |
100.0% |
IN8 720 OUT1 711 | |
| ✓ | DigitalRootCodeGenCode-gen: model implements sum-of-digits + digital root; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 720 OUT1 711 |
Algorithms - Temperature conversion
1 passed |
100.0% |
IN8 731 OUT714 | |
| ✓ | TemperatureCodeGenCode-gen: model implements temperature conversion; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 731 OUT714 |
Algorithms - Topological sort (lexicographically smallest)
1 passed |
100.0% |
IN17 972 OUT10 245 | |
| ✓ | TopoCodeGenCode-gen: model implements lexicographically-smallest topological sort.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN17 972 OUT10 245 |
Algorithms - Trailing zeros of factorial
1 passed |
100.0% |
IN8 732 OUT793 | |
| ✓ | TrailingZerosCodeGenCode-gen: model implements trailing zeros; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 732 OUT793 |
Algorithms - Trapping rain water
1 failed |
0.0% |
IN8 731 OUT3 741 | |
| × | TrappingRainCodeGenCode-gen: model implements trapping rain water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: Trapped([4 2 0 3 2 5]) got 3 expected 9
Values:
r.ResultJson = "3" |
Failed | IN8 731 OUT3 741 |
Algorithms - Two-sum indices
1 passed |
100.0% |
IN17 690 OUT3 907 | |
| ✓ | TwoSumCodeGenCode-gen: model implements two-sum; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN17 690 OUT3 907 |
Algorithms - Unique grid paths
1 failed |
0.0% |
IN8 713 OUT3 121 | |
| × | UniquePathsCodeGenCode-gen: model implements unique paths; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users Assertion expression evaluated to false: r.ResultJson == RefUnique(w, h).ToString()
Context: UniquePaths(3,2) got 2
Values:
r.ResultJson = "2" |
Failed | IN8 713 OUT3 121 |
Algorithms - Valid Sudoku unit
1 passed |
100.0% |
IN8 718 OUT972 | |
| ✓ | SudokuUnitCodeGenCode-gen: model implements Sudoku unit validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 718 OUT972 |
Algorithms - Wildcard string matching
1 passed |
100.0% |
IN8 736 OUT2 931 | |
| ✓ | WildcardMatchCodeGenCode-gen: model implements wildcard match; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 736 OUT2 931 |
Algorithms - Z-function
1 passed |
100.0% |
IN8 737 OUT3 357 | |
| ✓ | ZFunctionCodeGenCode-gen: model implements the Z-function; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 737 OUT3 357 |
Database - Conditional aggregation pivot
1 passed |
100.0% |
IN945 OUT336 | |
| ✓ | PivotCountsCodeGenCode-gen: model writes the pivot query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN945 OUT336 |
Database - Count distinct opponents
1 passed |
100.0% |
IN925 OUT265 | |
| ✓ | DistinctCountCodeGenCode-gen: model writes the COUNT(DISTINCT) query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Easysource:Ai4Users |
Passed | IN925 OUT265 |
Database - Date-range filter (BETWEEN)
1 passed |
100.0% |
IN955 OUT614 | |
| ✓ | DateRangeCodeGenCode-gen: model writes the date-range query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN955 OUT614 |
Database - EXISTS subquery (teams with matches)
1 passed |
100.0% |
IN932 OUT855 | |
| ✓ | ExistsSubqueryCodeGenCode-gen: model writes the EXISTS query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN932 OUT855 |
Database - JOIN player totals (two tables)
1 passed |
100.0% |
IN954 OUT515 | |
| ✓ | JoinScoresCodeGenCode-gen: model writes a JOIN + aggregate query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN954 OUT515 |
Database - LEFT JOIN anti-join (no items)
1 passed |
100.0% |
IN928 OUT690 | |
| ✓ | LeftJoinNullsCodeGenCode-gen: model writes the anti-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN928 OUT690 |
Database - Player stats SQL (multi-aggregate, QueryJson)
1 passed |
100.0% |
IN958 OUT194 | |
| ✓ | PlayerStatsCodeGenCode-gen: model writes a multi-aggregate stats query; graded by comparing its QueryJson output to the oracle query's (arbitrary shape, no struct).1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN958 OUT194 |
Database - Players above average score (subquery)
1 passed |
100.0% |
IN923 OUT300 | |
| ✓ | AboveAverageCodeGenCode-gen: model writes an above-average query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN923 OUT300 |
Database - Players who never played (anti-join)
1 passed |
100.0% |
IN920 OUT461 | |
| ✓ | NeverPlayedCodeGenCode-gen: model writes an anti-join query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN920 OUT461 |
Database - RANK window function
1 passed |
100.0% |
IN950 OUT925 | |
| ✓ | WindowRankCodeGenCode-gen: model writes the RANK() query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN950 OUT925 |
Database - Running total (window function)
1 passed |
100.0% |
IN947 OUT266 | |
| ✓ | RunningTotalCodeGenCode-gen: model writes the window running-total query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN947 OUT266 |
Database - Scoreboard SQL (players with >1 win, HAVING)
1 passed |
100.0% |
IN954 OUT663 | |
| ✓ | WinFilterCodeGenCode-gen: model writes a HAVING query; runs under try/catch with self-repair; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN954 OUT663 |
Database - Scoreboard SQL (top scorers query)
1 passed |
100.0% |
IN955 OUT267 | |
| ✓ | ScoreboardTopScorersCodeGenCode-gen: model writes a scoreboard SQL query; runs under try/catch with a self-repair loop on SQL errors; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN955 OUT267 |
Database - Self-join same-team pairs
1 passed |
100.0% |
IN938 OUT402 | |
| ✓ | SelfJoinCodeGenCode-gen: model writes the self-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN938 OUT402 |
Database - Top scorer per region
1 passed |
100.0% |
IN940 OUT1 069 | |
| ✓ | TopPerGroupCodeGenCode-gen: model writes the per-group top query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN940 OUT1 069 |
Database - UPDATE restock (SQL write)
1 passed |
100.0% |
IN932 OUT212 | |
| ✓ | UpdateInventoryCodeGenCode-gen: model writes an UPDATE; applied under try/catch, resulting state compared (via QueryJson) to the oracle update's state.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN932 OUT212 |
Database - Upsert with max merge
1 passed |
100.0% |
IN948 OUT486 | |
| ✓ | UpsertCodeGenCode-gen: model writes the upsert; executed on a fresh DB and the table compared.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN948 OUT486 |
SDLC - Acceptance criteria check
1 passed |
100.0% |
IN8 761 OUT4 459 | |
| ✓ | SdlcAcceptanceCodeGenCode-gen: model implements acceptance-criteria check; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT4 459 |
SDLC - Breaking API change detection
1 passed |
100.0% |
IN8 758 OUT4 292 | |
| ✓ | SdlcApiBreakingCodeGenCode-gen: model implements API-breaking detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 758 OUT4 292 |
SDLC - Condense without losing facts
1 passed |
100.0% |
IN1 046 OUT963 | |
| ✓ | NoInfoLossCodeGenRedaction test: model condenses keeping facts; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 046 OUT963 |
SDLC - Config merge with override
1 passed |
100.0% |
IN17 869 OUT7 293 | |
| ✓ | ConfigMergeCodeGenCode-gen: model implements config merge; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN17 869 OUT7 293 |
SDLC - Extract fields into fixed template
1 passed |
100.0% |
IN1 002 OUT335 | |
| ✓ | FieldExtractCodeGenRedaction test: model extracts fields into the template; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 002 OUT335 |
SDLC - Locate conflicting requirement pair
1 passed |
100.0% |
IN28 157 OUT15 468 | |
| ✓ | SdlcConstraintConflictCodeGenCode-gen: model locates the conflicting requirement pair; graded vs exact oracle.3 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN28 157 OUT15 468 |
SDLC - Locate redundant requirement
1 failed |
0.0% |
IN8 799 OUT7 912 | |
| × | SdlcRedundantRequirementCodeGenCode-gen: model locates a redundant requirement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: FindRedundant([x >= 5;x >= 10]) got 1 expected 0
Values:
r.ResultJson = "1" |
Failed | IN8 799 OUT7 912 |
SDLC - Mask PII to placeholder
1 passed |
100.0% |
IN970 OUT315 | |
| ✓ | MaskPiiCodeGenRedaction test: model masks PII to a placeholder; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN970 OUT315 |
SDLC - Modules on a dependency cycle
1 passed |
100.0% |
IN8 764 OUT6 940 | |
| ✓ | DependencyCycleModulesCodeGenCode-gen: model implements dependency-cycle detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN8 764 OUT6 940 |
SDLC - Remove duplicate requirements
1 passed |
100.0% |
IN943 OUT230 | |
| ✓ | DeduplicateCodeGenRedaction test: model removes duplicate requirements; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN943 OUT230 |
SDLC - Renumber requirements sequentially
1 passed |
100.0% |
IN1 010 OUT1 126 | |
| ✓ | RenumberCodeGenRedaction test: model renumbers sequentially; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 010 OUT1 126 |
SDLC - Reorganize notes into section template
1 passed |
100.0% |
IN1 124 OUT684 | |
| ✓ | RewriteSectionsCodeGenRedaction test: model reorganizes the notes; deterministic checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 124 OUT684 |
SDLC - Required semver bump from changelog
1 passed |
100.0% |
IN8 757 OUT2 444 | |
| ✓ | SemverBumpCodeGenCode-gen: model implements the semver-bump rule; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Easysource:Ai4Users |
Passed | IN8 757 OUT2 444 |
SDLC - Requirement-to-test traceability
1 passed |
100.0% |
IN8 779 OUT2 821 | |
| ✓ | SdlcTraceabilityCodeGenCode-gen: model implements traceability gap; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 779 OUT2 821 |
SDLC - Requirements completeness (spec coverage gap)
1 passed |
100.0% |
IN8 774 OUT6 424 | |
| ✓ | SdlcSpecCoverageCodeGenCode-gen: model implements spec-gap detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 774 OUT6 424 |
SDLC - Requirements consistency (contradiction detection)
1 passed |
100.0% |
IN18 149 OUT10 464 | |
| ✓ | ConsistencyCodeGenCode-gen: model implements requirements-consistency checking; graded vs oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN18 149 OUT10 464 |
SDLC - Reshape prose into JSON template
1 passed |
100.0% |
IN2 112 OUT796 | |
| ✓ | JsonShapeCodeGenRedaction test: model reshapes prose into JSON; checker + self-review loop.2 AI queries
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN2 112 OUT796 |
SDLC - Reshape rows into Markdown table
1 passed |
100.0% |
IN966 OUT265 | |
| ✓ | TableFormatCodeGenRedaction test: model builds a Markdown table; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN966 OUT265 |
SDLC - Rewrite in imperative requirement style
1 passed |
100.0% |
IN989 OUT1 055 | |
| ✓ | StyleRewriteCodeGenRedaction test: model rewrites notes in imperative style; checker + self-review loop.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN989 OUT1 055 |
SDLC - Roll up estimates over task tree
1 passed |
100.0% |
IN17 922 OUT8 007 | |
| ✓ | EstimateRollupCodeGenCode-gen: model implements estimate roll-up over a task tree; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN17 922 OUT8 007 |
SDLC - State-machine reachability
1 passed |
100.0% |
IN8 756 OUT2 164 | |
| ✓ | SdlcStateReachableCodeGenCode-gen: model implements state reachability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT2 164 |
SDLC - Ticket priority sort
1 passed |
100.0% |
IN8 749 OUT3 523 | |
| ✓ | SdlcPrioritySortCodeGenCode-gen: model implements priority sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 749 OUT3 523 |
SDLC - Unify glossary term consistency
1 passed |
100.0% |
IN970 OUT285 | |
| ✓ | GlossaryCodeGenRedaction test: model unifies synonyms to the canonical term; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN970 OUT285 |