Benchmark summary
A custom Ai4Users benchmark suite for SC#, covering code generation, reasoning, and stateful agentic workflows where models use tools, observe results, adapt their actions, recover from failures, and verify the final state.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Agentic - Blue-Green Deploy (health-gated cutover)
1 passed |
100.0% |
IN18 755 OUT695 | |
| ✓ | AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query
6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN18 755 OUT695 |
Agentic - Bugfix (root-cause vs regression)
1 passed |
100.0% |
IN20 329 OUT513 | |
| ✓ | AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query
1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN20 329 OUT513 |
Agentic - Cache Invalidation (exact set)
1 passed |
100.0% |
IN27 127 OUT965 | |
| ✓ | AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query
8 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN27 127 OUT965 |
Agentic - Canary (metric-gated release)
1 passed |
100.0% |
IN38 417 OUT2 609 | |
| ✓ | AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query
11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN38 417 OUT2 609 |
Agentic - Canary variants (5 metric cases)
5 passed |
100.0% |
IN142 949 OUT7 191 | |
| ✓ | CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN22 197 OUT1 410 |
| ✓ | CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN29 865 OUT1 571 |
| ✓ | CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN25 957 OUT1 180 |
| ✓ | CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query
11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN38 978 OUT1 888 |
| ✓ | CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN25 952 OUT1 142 |
Agentic - Code Review (find injected defect)
1 passed |
100.0% |
IN12 903 OUT1 449 | |
| ✓ | AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN12 903 OUT1 449 |
Agentic - Config Drift (remediate exact)
1 passed |
100.0% |
IN11 823 OUT576 | |
| ✓ | AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query
6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN11 823 OUT576 |
Agentic - Constraints (multi-section satisfaction)
1 passed |
100.0% |
IN11 149 OUT945 | |
| ✓ | AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 149 OUT945 |
Agentic - Constraints variants (5 cases)
5 passed |
100.0% |
IN47 555 OUT3 535 | |
| ✓ | ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 558 OUT1 024 |
| ✓ | ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 447 OUT454 |
| ✓ | ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 490 OUT498 |
| ✓ | ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 555 OUT875 |
| ✓ | ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 505 OUT684 |
Agentic - Data Reconciliation (per-field precedence)
1 passed |
100.0% |
IN13 382 OUT1 374 | |
| ✓ | AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN13 382 OUT1 374 |
Agentic - Diagnosis (root cause under budget)
1 passed |
100.0% |
IN12 213 OUT933 | |
| ✓ | AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN12 213 OUT933 |
Agentic - Diagnosis variants (5 cases)
5 passed |
100.0% |
IN74 848 OUT5 360 | |
| ✓ | DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN14 805 OUT1 209 |
| ✓ | DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN14 952 OUT1 075 |
| ✓ | DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 028 OUT1 189 |
| ✓ | DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN14 565 OUT898 |
| ✓ | DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 498 OUT989 |
Agentic - ETL Repair (root-cause in stage 2)
1 passed |
100.0% |
IN31 262 OUT1 203 | |
| ✓ | AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN31 262 OUT1 203 |
Agentic - Feature Flag (staged rollout + kill-switch)
1 passed |
100.0% |
IN30 309 OUT1 058 | |
| ✓ | AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN30 309 OUT1 058 |
Agentic - Game Backend: 1D bouncing ball
1 passed |
100.0% |
IN8 911 OUT3 267 | |
| ✓ | GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 911 OUT3 267 |
Agentic - Game Backend: 1D swept anti-tunneling
1 passed |
100.0% |
IN8 922 OUT2 517 | |
| ✓ | GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 922 OUT2 517 |
Agentic - Game Backend: 2048 left-move (slide + merge)
1 passed |
100.0% |
IN8 931 OUT4 263 | |
| ✓ | GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 931 OUT4 263 |
Agentic - Game Backend: 8-direction from vector
1 passed |
100.0% |
IN8 938 OUT2 146 | |
| ✓ | GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 938 OUT2 146 |
Agentic - Game Backend: AABB collision resolution (MTV)
1 passed |
100.0% |
IN9 027 OUT2 637 | |
| ✓ | GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN9 027 OUT2 637 |
Agentic - Game Backend: AABB containment
1 passed |
100.0% |
IN8 867 OUT571 | |
| ✓ | GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 867 OUT571 |
Agentic - Game Backend: Acceleration to max speed
1 passed |
100.0% |
IN8 869 OUT645 | |
| ✓ | GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 869 OUT645 |
Agentic - Game Backend: Achievement thresholds
1 passed |
100.0% |
IN8 861 OUT1 803 | |
| ✓ | GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 861 OUT1 803 |
Agentic - Game Backend: Active status effects
1 passed |
100.0% |
IN8 883 OUT2 273 | |
| ✓ | GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 883 OUT2 273 |
Agentic - Game Backend: Animation frame at time
1 passed |
100.0% |
IN8 901 OUT849 | |
| ✓ | GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 901 OUT849 |
Agentic - Game Backend: Army upkeep income
1 passed |
100.0% |
IN8 849 OUT1 398 | |
| ✓ | GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 849 OUT1 398 |
Agentic - Game Backend: Axis-aligned path follow
1 passed |
100.0% |
IN8 892 OUT2 793 | |
| ✓ | GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 892 OUT2 793 |
Agentic - Game Backend: Battleship shot resolution
1 passed |
100.0% |
IN8 951 OUT992 | |
| ✓ | GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 951 OUT992 |
Agentic - Game Backend: Bounce restitution
1 passed |
100.0% |
IN8 888 OUT1 120 | |
| ✓ | GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 888 OUT1 120 |
Agentic - Game Backend: Bresenham line rasterization
1 passed |
100.0% |
IN8 978 OUT2 328 | |
| ✓ | GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 978 OUT2 328 |
Agentic - Game Backend: Buff/debuff stacking
1 passed |
100.0% |
IN8 914 OUT3 395 | |
| ✓ | GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 914 OUT3 395 |
Agentic - Game Backend: Build queue completions
1 passed |
100.0% |
IN8 882 OUT1 611 | |
| ✓ | GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 882 OUT1 611 |
Agentic - Game Backend: Camera-relative movement
1 passed |
100.0% |
IN8 969 OUT1 706 | |
| ✓ | GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 969 OUT1 706 |
Agentic - Game Backend: Card hand sort
1 passed |
100.0% |
IN8 890 OUT3 451 | |
| ✓ | GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 890 OUT3 451 |
Agentic - Game Backend: Card-game rules (blackjack/deck/compare)
1 passed |
100.0% |
IN18 743 OUT8 071 | |
| ✓ | GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 743 OUT8 071 |
Agentic - Game Backend: Chart scaling & histogram bins
1 passed |
100.0% |
IN8 954 OUT2 311 | |
| ✓ | GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 954 OUT2 311 |
Agentic - Game Backend: Checkers captures
1 failed |
0.0% |
IN8 924 OUT5 178 | |
| × | GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Captures threw: String index 24 is out of range. The string has 23 character(s). |
Failed | IN8 924 OUT5 178 |
Agentic - Game Backend: Collision primitives (circle/aabb/segment)
1 passed |
100.0% |
IN9 127 OUT4 943 | |
| ✓ | GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN9 127 OUT4 943 |
Agentic - Game Backend: Combat turn order (initiative)
1 passed |
100.0% |
IN8 863 OUT2 749 | |
| ✓ | GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 863 OUT2 749 |
Agentic - Game Backend: Combo scoring (rising multiplier)
1 passed |
100.0% |
IN8 886 OUT1 133 | |
| ✓ | GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 886 OUT1 133 |
Agentic - Game Backend: Control-point capture
1 passed |
100.0% |
IN8 899 OUT2 138 | |
| ✓ | GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 899 OUT2 138 |
Agentic - Game Backend: Controls & Inventory (spec-driven function battery)
1 passed |
100.0% |
IN9 040 OUT2 206 | |
| ✓ | GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN9 040 OUT2 206 |
Agentic - Game Backend: Cooldowns & resource regen
1 passed |
100.0% |
IN8 957 OUT1 276 | |
| ✓ | GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 957 OUT1 276 |
Agentic - Game Backend: Coyote-time jump
1 passed |
100.0% |
IN8 878 OUT320 | |
| ✓ | GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 878 OUT320 |
Agentic - Game Backend: Crafting capacity
1 passed |
100.0% |
IN8 902 OUT3 893 | |
| ✓ | GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 902 OUT3 893 |
Agentic - Game Backend: Cribbage fifteens
1 passed |
100.0% |
IN8 879 OUT2 731 | |
| ✓ | GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 879 OUT2 731 |
Agentic - Game Backend: Critical-hit damage
1 passed |
100.0% |
IN8 869 OUT559 | |
| ✓ | GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 869 OUT559 |
Agentic - Game Backend: Damage over time
1 passed |
100.0% |
IN8 895 OUT1 894 | |
| ✓ | GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 895 OUT1 894 |
Agentic - Game Backend: Day/night cycle phase
1 passed |
100.0% |
IN8 881 OUT521 | |
| ✓ | GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 881 OUT521 |
Agentic - Game Backend: Dice notation max roll
1 passed |
100.0% |
IN8 896 OUT3 038 | |
| ✓ | GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 896 OUT3 038 |
Agentic - Game Backend: Enemy AI state machine
1 passed |
100.0% |
IN8 982 OUT2 555 | |
| ✓ | GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 982 OUT2 555 |
Agentic - Game Backend: Equipment slot validation
1 passed |
100.0% |
IN8 896 OUT1 425 | |
| ✓ | GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 896 OUT1 425 |
Agentic - Game Backend: Flood-fill region size
1 passed |
100.0% |
IN8 884 OUT4 589 | |
| ✓ | GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 884 OUT4 589 |
Agentic - Game Backend: Fog of war visibility
1 passed |
100.0% |
IN8 924 OUT1 916 | |
| ✓ | GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 924 OUT1 916 |
Agentic - Game Backend: Fog reveal (Chebyshev radius)
1 passed |
100.0% |
IN8 897 OUT3 021 | |
| ✓ | GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 897 OUT3 021 |
Agentic - Game Backend: Friction stop frame
1 passed |
100.0% |
IN8 877 OUT1 011 | |
| ✓ | GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 877 OUT1 011 |
Agentic - Game Backend: Game of Life (code-gen vs oracle)
1 passed |
100.0% |
IN8 936 OUT1 227 | |
| ✓ | GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 936 OUT1 227 |
Agentic - Game Backend: Greedy coin change
1 passed |
100.0% |
IN8 878 OUT1 942 | |
| ✓ | GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 878 OUT1 942 |
Agentic - Game Backend: Grid inventory & stacking (Diablo-style)
1 passed |
100.0% |
IN9 011 OUT2 924 | |
| ✓ | GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN9 011 OUT2 924 |
Agentic - Game Backend: Grid pathfinding (shortest path, BFS)
1 passed |
100.0% |
IN8 966 OUT5 263 | |
| ✓ | GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.1 AI query
0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 966 OUT5 263 |
Agentic - Game Backend: Grid raycast
1 passed |
100.0% |
IN8 934 OUT1 640 | |
| ✓ | GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 934 OUT1 640 |
Agentic - Game Backend: Health bar segments
1 passed |
100.0% |
IN8 935 OUT2 495 | |
| ✓ | GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 935 OUT2 495 |
Agentic - Game Backend: High-score table insert
1 passed |
100.0% |
IN18 055 OUT7 066 | |
| ✓ | GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 055 OUT7 066 |
Agentic - Game Backend: Hit chance
1 passed |
100.0% |
IN8 873 OUT590 | |
| ✓ | GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 873 OUT590 |
Agentic - Game Backend: Inventory encumbrance
1 passed |
100.0% |
IN8 870 OUT1 007 | |
| ✓ | GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 870 OUT1 007 |
Agentic - Game Backend: Inventory first-fit placement
1 passed |
100.0% |
IN8 936 OUT1 616 | |
| ✓ | GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 936 OUT1 616 |
Agentic - Game Backend: Inventory stack split
1 passed |
100.0% |
IN8 885 OUT518 | |
| ✓ | GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 885 OUT518 |
Agentic - Game Backend: Isometric projection
1 passed |
100.0% |
IN17 939 OUT1 692 | |
| ✓ | GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN17 939 OUT1 692 |
Agentic - Game Backend: Jump buffer
1 passed |
100.0% |
IN8 891 OUT1 793 | |
| ✓ | GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 891 OUT1 793 |
Agentic - Game Backend: Knockback with arena clamp
1 passed |
100.0% |
IN8 930 OUT1 198 | |
| ✓ | GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 930 OUT1 198 |
Agentic - Game Backend: Leaderboard rank
1 passed |
100.0% |
IN8 863 OUT1 878 | |
| ✓ | GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 863 OUT1 878 |
Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over)
1 passed |
100.0% |
IN9 014 OUT1 741 | |
| ✓ | GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN9 014 OUT1 741 |
Agentic - Game Backend: Lifesteal healing
1 passed |
100.0% |
IN8 874 OUT1 868 | |
| ✓ | GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 874 OUT1 868 |
Agentic - Game Backend: Lights Out toggle
1 passed |
100.0% |
IN8 890 OUT3 366 | |
| ✓ | GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 890 OUT3 366 |
Agentic - Game Backend: Match-3 cascade
1 passed |
100.0% |
IN8 948 OUT5 544 | |
| ✓ | GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 948 OUT5 544 |
Agentic - Game Backend: Match-3 detection
1 passed |
100.0% |
IN8 912 OUT2 513 | |
| ✓ | GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 912 OUT2 513 |
Agentic - Game Backend: Match-3 swap validity
1 passed |
100.0% |
IN8 921 OUT4 600 | |
| ✓ | GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 921 OUT4 600 |
Agentic - Game Backend: Memory match pairs
1 passed |
100.0% |
IN8 875 OUT1 737 | |
| ✓ | GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 875 OUT1 737 |
Agentic - Game Backend: Mercy invincibility (i-frames)
1 passed |
100.0% |
IN8 930 OUT2 858 | |
| ✓ | GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 930 OUT2 858 |
Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal)
1 passed |
100.0% |
IN9 012 OUT3 411 | |
| ✓ | GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN9 012 OUT3 411 |
Agentic - Game Backend: Minimap downsample
1 passed |
100.0% |
IN8 906 OUT2 548 | |
| ✓ | GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 906 OUT2 548 |
Agentic - Game Backend: Minimap marker position
1 passed |
100.0% |
IN8 909 OUT1 156 | |
| ✓ | GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 909 OUT1 156 |
Agentic - Game Backend: N-in-a-row win detection (Connect-4)
1 passed |
100.0% |
IN8 924 OUT3 277 | |
| ✓ | GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 924 OUT3 277 |
Agentic - Game Backend: Nonogram row validation
1 passed |
100.0% |
IN8 886 OUT2 083 | |
| ✓ | GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 886 OUT2 083 |
Agentic - Game Backend: OBB collision (rotating rectangles, SAT)
1 passed |
100.0% |
IN18 858 OUT9 963 | |
| ✓ | GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 858 OUT9 963 |
Agentic - Game Backend: Particle lifetime & expiry
1 passed |
100.0% |
IN8 968 OUT2 037 | |
| ✓ | GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 968 OUT2 037 |
Agentic - Game Backend: Party XP share
1 passed |
100.0% |
IN8 886 OUT1 349 | |
| ✓ | GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 886 OUT1 349 |
Agentic - Game Backend: Patrol waypoints (ping-pong)
1 passed |
100.0% |
IN8 909 OUT1 823 | |
| ✓ | GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 909 OUT1 823 |
Agentic - Game Backend: Platformer jump arc
1 passed |
100.0% |
IN8 900 OUT2 962 | |
| ✓ | GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 900 OUT2 962 |
Agentic - Game Backend: Point on circle (trig)
1 passed |
100.0% |
IN8 933 OUT1 229 | |
| ✓ | GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 933 OUT1 229 |
Agentic - Game Backend: Poker hand comparison
1 passed |
100.0% |
IN9 009 OUT11 080 | |
| ✓ | GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users |
Passed | IN9 009 OUT11 080 |
Agentic - Game Backend: Poker hand ranking
1 passed |
100.0% |
IN8 952 OUT5 490 | |
| ✓ | GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 952 OUT5 490 |
Agentic - Game Backend: Pop-cap training
1 passed |
100.0% |
IN8 887 OUT595 | |
| ✓ | GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 887 OUT595 |
Agentic - Game Backend: Progress bar fill
1 passed |
100.0% |
IN8 869 OUT888 | |
| ✓ | GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 869 OUT888 |
Agentic - Game Backend: Projectile range (physics/trig)
1 passed |
100.0% |
IN8 891 OUT1 296 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 891 OUT1 296 |
Agentic - Game Backend: Projectile with ground bounce
1 passed |
100.0% |
IN8 925 OUT2 208 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 925 OUT2 208 |
Agentic - Game Backend: Resource gathering
1 passed |
100.0% |
IN8 886 OUT739 | |
| ✓ | GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 886 OUT739 |
Agentic - Game Backend: Reversi flip count
1 passed |
100.0% |
IN8 957 OUT4 337 | |
| ✓ | GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 957 OUT4 337 |
Agentic - Game Backend: Rotate 2D vector (trig)
1 passed |
100.0% |
IN8 902 OUT1 137 | |
| ✓ | GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 902 OUT1 137 |
Agentic - Game Backend: Round-robin deal
1 passed |
100.0% |
IN8 891 OUT1 570 | |
| ✓ | GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 891 OUT1 570 |
Agentic - Game Backend: RTS resource economy
1 passed |
100.0% |
IN8 882 OUT873 | |
| ✓ | GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 882 OUT873 |
Agentic - Game Backend: RTS unit counters & damage
1 passed |
100.0% |
IN8 934 OUT993 | |
| ✓ | GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 934 OUT993 |
Agentic - Game Backend: Safest neighbour (danger map)
1 passed |
100.0% |
IN18 355 OUT6 602 | |
| ✓ | GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 355 OUT6 602 |
Agentic - Game Backend: Scroll viewport
1 passed |
100.0% |
IN8 904 OUT920 | |
| ✓ | GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 904 OUT920 |
Agentic - Game Backend: Set card validity
1 passed |
100.0% |
IN8 883 OUT1 678 | |
| ✓ | GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 883 OUT1 678 |
Agentic - Game Backend: Shop affordability
1 passed |
100.0% |
IN8 863 OUT794 | |
| ✓ | GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 863 OUT794 |
Agentic - Game Backend: Sliding puzzle move
1 passed |
100.0% |
IN8 894 OUT4 067 | |
| ✓ | GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 894 OUT4 067 |
Agentic - Game Backend: Snake head movement
1 passed |
100.0% |
IN8 951 OUT1 785 | |
| ✓ | GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 951 OUT1 785 |
Agentic - Game Backend: Snake tick
1 passed |
100.0% |
IN9 021 OUT2 930 | |
| ✓ | GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN9 021 OUT2 930 |
Agentic - Game Backend: Snap to grid
1 passed |
100.0% |
IN8 879 OUT832 | |
| ✓ | GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 879 OUT832 |
Agentic - Game Backend: Sokoban move/push legality
1 passed |
100.0% |
IN8 982 OUT1 284 | |
| ✓ | GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 982 OUT1 284 |
Agentic - Game Backend: Solitaire tableau move
1 passed |
100.0% |
IN8 921 OUT2 411 | |
| ✓ | GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 921 OUT2 411 |
Agentic - Game Backend: Spawn scheduler count
1 passed |
100.0% |
IN8 878 OUT1 381 | |
| ✓ | GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 878 OUT1 381 |
Agentic - Game Backend: Spawn-wave alive count
1 passed |
100.0% |
IN8 890 OUT1 458 | |
| ✓ | GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 890 OUT1 458 |
Agentic - Game Backend: Stat level scaling
1 passed |
100.0% |
IN8 882 OUT1 191 | |
| ✓ | GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 882 OUT1 191 |
Agentic - Game Backend: Steering seek (greedy pursuit)
1 passed |
100.0% |
IN8 908 OUT1 770 | |
| ✓ | GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 908 OUT1 770 |
Agentic - Game Backend: Tactical movement range
1 passed |
100.0% |
IN8 920 OUT3 873 | |
| ✓ | GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 920 OUT3 873 |
Agentic - Game Backend: Tech tree eligibility
1 passed |
100.0% |
IN8 902 OUT2 196 | |
| ✓ | GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 902 OUT2 196 |
Agentic - Game Backend: Tetris line clearing
1 passed |
100.0% |
IN8 893 OUT2 374 | |
| ✓ | GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 893 OUT2 374 |
Agentic - Game Backend: Threat table top target
1 passed |
100.0% |
IN8 887 OUT2 700 | |
| ✓ | GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 887 OUT2 700 |
Agentic - Game Backend: Tic-tac-toe result
1 passed |
100.0% |
IN8 878 OUT2 427 | |
| ✓ | GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 878 OUT2 427 |
Agentic - Game Backend: Toroidal wrap (Pac-Man)
1 passed |
100.0% |
IN8 914 OUT909 | |
| ✓ | GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 914 OUT909 |
Agentic - Game Backend: Tower targeting (nearest in range)
1 passed |
100.0% |
IN8 902 OUT2 912 | |
| ✓ | GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 902 OUT2 912 |
Agentic - Game Backend: Transaction ledger
1 passed |
100.0% |
IN8 905 OUT2 639 | |
| ✓ | GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 905 OUT2 639 |
Agentic - Game Backend: Trick-taking winner
1 passed |
100.0% |
IN8 933 OUT3 631 | |
| ✓ | GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 933 OUT3 631 |
Agentic - Game Backend: Turn-based duel resolution
1 passed |
100.0% |
IN8 965 OUT2 176 | |
| ✓ | GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 965 OUT2 176 |
Agentic - Game Backend: Typed armor mitigation
1 passed |
100.0% |
IN8 889 OUT2 206 | |
| ✓ | GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 889 OUT2 206 |
Agentic - Game Backend: UI menu (wrap/clamp/paginate)
1 passed |
100.0% |
IN8 981 OUT2 757 | |
| ✓ | GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 981 OUT2 757 |
Agentic - Game Backend: Uno playable cards
1 passed |
100.0% |
IN8 917 OUT1 831 | |
| ✓ | GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 917 OUT1 831 |
Agentic - Game Backend: Velocity reflection
1 passed |
100.0% |
IN8 861 OUT1 099 | |
| ✓ | GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 861 OUT1 099 |
Agentic - Game Backend: Wall-slide fall
1 passed |
100.0% |
IN8 869 OUT1 738 | |
| ✓ | GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 869 OUT1 738 |
Agentic - Game Backend: War card compare
1 passed |
100.0% |
IN8 889 OUT1 418 | |
| ✓ | GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 889 OUT1 418 |
Agentic - Game Backend: Weighted loot table
1 passed |
100.0% |
IN8 892 OUT1 850 | |
| ✓ | GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 892 OUT1 850 |
Agentic - Game Backend: Weighted terrain pathfinding
1 passed |
100.0% |
IN8 942 OUT6 763 | |
| ✓ | GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 942 OUT6 763 |
Agentic - Game Backend: Word-wrap line count
1 passed |
100.0% |
IN8 874 OUT4 613 | |
| ✓ | GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 874 OUT4 613 |
Agentic - Game Backend: XP to character level
1 passed |
100.0% |
IN8 882 OUT1 563 | |
| ✓ | GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 882 OUT1 563 |
Agentic - Idempotency (poison-retry / no double charge)
1 passed |
100.0% |
IN17 758 OUT710 | |
| ✓ | AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query
8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN17 758 OUT710 |
Agentic - Irreversible Commit (measure twice)
1 passed |
100.0% |
IN15 630 OUT591 | |
| ✓ | AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query
7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN15 630 OUT591 |
Agentic - Least Privilege (exact grants)
1 passed |
100.0% |
IN11 828 OUT478 | |
| ✓ | AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query
6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN11 828 OUT478 |
Agentic - Ledger (simulated DB read-modify-write)
1 passed |
100.0% |
IN12 894 OUT1 352 | |
| ✓ | AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 894 OUT1 352 |
Agentic - Ledger variants (5 RMW cases)
5 passed |
100.0% |
IN123 208 OUT17 079 | |
| ✓ | LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query
19 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN35 478 OUT7 021 |
| ✓ | LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query
16 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN43 439 OUT2 602 |
| ✓ | LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 906 OUT2 277 |
| ✓ | LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 448 OUT2 114 |
| ✓ | LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN18 937 OUT3 065 |
Agentic - Lock Ordering (deadlock avoidance)
1 passed |
100.0% |
IN83 994 OUT2 740 | |
| ✓ | AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query
25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN83 994 OUT2 740 |
Agentic - Lock Ordering variants (5 cases)
5 passed |
100.0% |
IN296 171 OUT11 631 | |
| ✓ | LockVariantFiveResFourOps1 AI query
25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN70 651 OUT2 905 |
| ✓ | LockVariantFourResThreeOps1 AI query
18 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN47 213 OUT1 842 |
| ✓ | LockVariantHeavyReuse1 AI query
23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN77 389 OUT2 454 |
| ✓ | LockVariantReverseListed1 AI query
24 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN52 051 OUT2 311 |
| ✓ | LockVariantThreeLockOp1 AI query
20 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN48 867 OUT2 119 |
Agentic - Migration (cascading rollback + reapply)
1 passed |
100.0% |
IN41 737 OUT1 323 | |
| ✓ | AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query
12 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN41 737 OUT1 323 |
Agentic - Requirements (detect contradiction)
1 passed |
100.0% |
IN7 149 OUT458 | |
| ✓ | AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query
2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN7 149 OUT458 |
Agentic - Retry/Circuit-breaker (transient vs hard-down)
1 passed |
100.0% |
IN25 846 OUT937 | |
| ✓ | AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query
10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN25 846 OUT937 |
Agentic - Saga (compensation / distributed rollback)
1 passed |
100.0% |
IN20 985 OUT625 | |
| ✓ | AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN20 985 OUT625 |
Agentic - Saga variants (5 compensation cases)
5 passed |
100.0% |
IN90 456 OUT2 728 | |
| ✓ | SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query
4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN13 397 OUT418 |
| ✓ | SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query
4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN13 447 OUT331 |
| ✓ | SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 773 OUT562 |
| ✓ | SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 773 OUT555 |
| ✓ | SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query
9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN30 066 OUT862 |
Agentic - Schema Compat (expand-contract)
1 passed |
100.0% |
IN21 117 OUT564 | |
| ✓ | AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN21 117 OUT564 |
Agentic - Scratchpad (self-repair, Dyck, tool-decision)
6 passed1 broken |
85.7% |
IN152 220 OUT68 740 | |
| ✓ | AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query
1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN21 682 OUT2 722 |
| ✓ | AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query
2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN32 660 OUT1 015 |
| ✓ | ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query
0 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN10 656 OUT72 |
| ✓ | ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query
1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN21 410 OUT687 |
| ✓ | ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query
1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN21 448 OUT332 |
| ! | ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query
0 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: response.Completed
Context: ToolDecisionLongDyck: AI response did not complete.
Values:
response.Completed = false |
Broken | IN10 767 OUT62 144 |
| ✓ | ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query
2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN33 597 OUT1 768 |
Agentic - Secret Rotation (zero-downtime ordering)
1 passed |
100.0% |
IN16 327 OUT576 | |
| ✓ | AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 327 OUT576 |
Agentic - StatefulOps (incident recovery, concurrency replan)
2 passed |
100.0% |
IN52 639 OUT1 802 | |
| ✓ | AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query
7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN21 889 OUT836 |
| ✓ | AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query
8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN30 750 OUT966 |
Agentic - Test Authoring (write a test that catches the bug)
1 passed |
100.0% |
IN11 558 OUT849 | |
| ✓ | AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 558 OUT849 |
Agentic - Topological Planning (deps + prereq)
1 passed |
100.0% |
IN28 038 OUT1 019 | |
| ✓ | AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN28 038 OUT1 019 |
Algorithms - 0/1 Knapsack (dynamic programming)
1 passed |
100.0% |
IN8 896 OUT3 109 | |
| ✓ | KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 896 OUT3 109 |
Algorithms - 1D elastic collision
1 passed |
100.0% |
IN8 936 OUT2 215 | |
| ✓ | ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 936 OUT2 215 |
Algorithms - All-pairs shortest path pair
1 passed |
100.0% |
IN8 904 OUT4 680 | |
| ✓ | FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 904 OUT4 680 |
Algorithms - Balanced brackets (stack)
1 passed |
100.0% |
IN8 867 OUT2 529 | |
| ✓ | ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 867 OUT2 529 |
Algorithms - Bellman-Ford shortest path (neg weights/cycles)
1 passed |
100.0% |
IN8 913 OUT6 480 | |
| ✓ | BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 913 OUT6 480 |
Algorithms - Binary Gray code
1 passed |
100.0% |
IN8 840 OUT599 | |
| ✓ | GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 840 OUT599 |
Algorithms - Bipartite graph check
1 passed |
100.0% |
IN8 886 OUT3 249 | |
| ✓ | BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 886 OUT3 249 |
Algorithms - Bit tricks (popcount/pow2/lowbit)
1 passed |
100.0% |
IN8 914 OUT1 500 | |
| ✓ | BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 914 OUT1 500 |
Algorithms - Caesar cipher
1 passed |
100.0% |
IN8 863 OUT2 098 | |
| ✓ | CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 863 OUT2 098 |
Algorithms - Catalan number
1 passed |
100.0% |
IN8 885 OUT1 628 | |
| ✓ | CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 885 OUT1 628 |
Algorithms - Chinese remainder theorem
1 passed |
100.0% |
IN8 903 OUT6 206 | |
| ✓ | CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 903 OUT6 206 |
Algorithms - Circle-circle MTV
1 passed |
100.0% |
IN8 980 OUT1 926 | |
| ✓ | CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 980 OUT1 926 |
Algorithms - Coin change combinations
1 passed |
100.0% |
IN8 870 OUT2 012 | |
| ✓ | CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 870 OUT2 012 |
Algorithms - Collatz step count
1 passed |
100.0% |
IN8 866 OUT691 | |
| ✓ | CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 866 OUT691 |
Algorithms - Connected components (union-find)
1 passed |
100.0% |
IN8 882 OUT3 720 | |
| ✓ | UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 882 OUT3 720 |
Algorithms - Container with most water
1 passed |
100.0% |
IN8 874 OUT1 835 | |
| ✓ | ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 874 OUT1 835 |
Algorithms - Count anagram groups
1 passed |
100.0% |
IN8 857 OUT1 897 | |
| ✓ | AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 857 OUT1 897 |
Algorithms - Count grid color regions
1 passed |
100.0% |
IN8 877 OUT2 076 | |
| ✓ | CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 877 OUT2 076 |
Algorithms - Count islands
1 passed |
100.0% |
IN8 880 OUT2 948 | |
| ✓ | CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 880 OUT2 948 |
Algorithms - Count overlapping occurrences
1 passed |
100.0% |
IN8 863 OUT1 248 | |
| ✓ | RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 863 OUT1 248 |
Algorithms - Count primes (Sieve of Eratosthenes)
1 passed |
100.0% |
IN8 842 OUT1 529 | |
| ✓ | CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 842 OUT1 529 |
Algorithms - Cumulative distribution
1 passed |
100.0% |
IN8 864 OUT2 131 | |
| ✓ | CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 864 OUT2 131 |
Algorithms - Dijkstra shortest path (weighted graph)
1 passed |
100.0% |
IN8 916 OUT3 458 | |
| ✓ | DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 916 OUT3 458 |
Algorithms - Directed cycle detection
1 passed |
100.0% |
IN8 873 OUT3 711 | |
| ✓ | DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 873 OUT3 711 |
Algorithms - Distinct subsequences
1 passed |
100.0% |
IN8 859 OUT3 596 | |
| ✓ | DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 859 OUT3 596 |
Algorithms - Divisor count
1 passed |
100.0% |
IN8 847 OUT1 160 | |
| ✓ | DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 847 OUT1 160 |
Algorithms - Downsample by block average
1 passed |
100.0% |
IN8 888 OUT2 109 | |
| ✓ | ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 888 OUT2 109 |
Algorithms - Dutch national flag sort
1 passed |
100.0% |
IN8 855 OUT2 802 | |
| ✓ | DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 855 OUT2 802 |
Algorithms - Egg drop trials
1 passed |
100.0% |
IN8 916 OUT3 083 | |
| ✓ | EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 916 OUT3 083 |
Algorithms - Equal subset-sum partition
1 passed |
100.0% |
IN8 858 OUT1 354 | |
| ✓ | PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 858 OUT1 354 |
Algorithms - Euler totient
1 passed |
100.0% |
IN8 847 OUT760 | |
| ✓ | EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 847 OUT760 |
Algorithms - Eulerian path existence
1 passed |
100.0% |
IN8 906 OUT4 503 | |
| ✓ | EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 906 OUT4 503 |
Algorithms - Extended Euclidean
1 passed |
100.0% |
IN8 959 OUT2 025 | |
| ✓ | ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 959 OUT2 025 |
Algorithms - Format duration H:MM:SS
1 passed |
100.0% |
IN8 874 OUT1 446 | |
| ✓ | FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 874 OUT1 446 |
Algorithms - GCD and LCM
1 passed |
100.0% |
IN8 851 OUT1 096 | |
| ✓ | GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 851 OUT1 096 |
Algorithms - Grid transpose
1 failed |
0.0% |
IN8 882 OUT1 486 | |
| × | TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Transpose threw: String index 6 is out of range. The string has 6 character(s). |
Failed | IN8 882 OUT1 486 |
Algorithms - House robber (max non-adjacent sum)
1 passed |
100.0% |
IN8 846 OUT1 314 | |
| ✓ | HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 846 OUT1 314 |
Algorithms - Integer linear interpolation
1 passed |
100.0% |
IN8 880 OUT309 | |
| ✓ | LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 880 OUT309 |
Algorithms - Integer square root
1 passed |
100.0% |
IN8 862 OUT1 777 | |
| ✓ | IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 862 OUT1 777 |
Algorithms - Integer to base-N string
1 passed |
100.0% |
IN8 868 OUT1 585 | |
| ✓ | BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 868 OUT1 585 |
Algorithms - Integer to Roman numeral
1 passed |
100.0% |
IN8 897 OUT1 562 | |
| ✓ | RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 897 OUT1 562 |
Algorithms - Josephus survivor
1 passed |
100.0% |
IN8 873 OUT1 451 | |
| ✓ | JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 873 OUT1 451 |
Algorithms - Jump game (min jumps)
1 passed |
100.0% |
IN8 881 OUT4 132 | |
| ✓ | JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 881 OUT4 132 |
Algorithms - Kth largest element
1 passed |
100.0% |
IN8 871 OUT1 149 | |
| ✓ | KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 871 OUT1 149 |
Algorithms - Least-squares line fit
1 passed |
100.0% |
IN8 913 OUT3 514 | |
| ✓ | LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 913 OUT3 514 |
Algorithms - Levenshtein edit distance (DP)
1 passed |
100.0% |
IN8 841 OUT2 544 | |
| ✓ | EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 841 OUT2 544 |
Algorithms - Log-scale bucket index
1 passed |
100.0% |
IN8 906 OUT2 135 | |
| ✓ | LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 906 OUT2 135 |
Algorithms - Logarithmic bin index
1 passed |
100.0% |
IN8 895 OUT1 717 | |
| ✓ | LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 895 OUT1 717 |
Algorithms - Longest common prefix
1 passed |
100.0% |
IN8 854 OUT3 272 | |
| ✓ | CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 854 OUT3 272 |
Algorithms - Longest common subsequence
1 passed |
100.0% |
IN8 839 OUT3 369 | |
| ✓ | LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 839 OUT3 369 |
Algorithms - Longest common substring
1 passed |
100.0% |
IN8 856 OUT3 354 | |
| ✓ | CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 856 OUT3 354 |
Algorithms - Longest increasing subsequence (DP)
1 passed |
100.0% |
IN8 873 OUT2 284 | |
| ✓ | LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 873 OUT2 284 |
Algorithms - Longest palindromic substring
1 passed |
100.0% |
IN8 851 OUT1 746 | |
| ✓ | LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 851 OUT1 746 |
Algorithms - Longest repeated substring
1 passed |
100.0% |
IN8 856 OUT2 591 | |
| ✓ | LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 856 OUT2 591 |
Algorithms - Lower bound (binary search)
1 passed |
100.0% |
IN8 884 OUT1 707 | |
| ✓ | LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 884 OUT1 707 |
Algorithms - Luhn checksum validation
1 passed |
100.0% |
IN8 885 OUT1 468 | |
| ✓ | LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 885 OUT1 468 |
Algorithms - Majority element
1 passed |
100.0% |
IN8 849 OUT2 521 | |
| ✓ | MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 849 OUT2 521 |
Algorithms - Matrix chain multiplication
1 passed |
100.0% |
IN8 884 OUT4 345 | |
| ✓ | MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 884 OUT4 345 |
Algorithms - Maximum subarray sum (Kadane)
1 passed |
100.0% |
IN8 874 OUT3 282 | |
| ✓ | MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 874 OUT3 282 |
Algorithms - Merge K sorted lists
1 passed |
100.0% |
IN8 867 OUT3 215 | |
| ✓ | MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 867 OUT3 215 |
Algorithms - Merge overlapping intervals
1 passed |
100.0% |
IN8 899 OUT3 442 | |
| ✓ | MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 899 OUT3 442 |
Algorithms - Min-max normalization
1 passed |
100.0% |
IN8 876 OUT2 348 | |
| ✓ | MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 876 OUT2 348 |
Algorithms - Minimum coin change (DP)
1 passed |
100.0% |
IN8 867 OUT2 323 | |
| ✓ | CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 867 OUT2 323 |
Algorithms - Minimum knight moves (BFS)
1 passed |
100.0% |
IN8 905 OUT4 373 | |
| ✓ | KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 905 OUT4 373 |
Algorithms - Minimum path sum (grid DP)
1 passed |
100.0% |
IN8 880 OUT3 256 | |
| ✓ | MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 880 OUT3 256 |
Algorithms - Minimum spanning tree (Kruskal)
1 passed |
100.0% |
IN8 880 OUT5 690 | |
| ✓ | KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 880 OUT5 690 |
Algorithms - Minimum window substring
1 passed |
100.0% |
IN8 888 OUT3 617 | |
| ✓ | MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 888 OUT3 617 |
Algorithms - Modular exponentiation (fast pow)
1 passed |
100.0% |
IN8 899 OUT1 761 | |
| ✓ | ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 899 OUT1 761 |
Algorithms - Modular inverse (extended Euclid)
1 passed |
100.0% |
IN8 887 OUT3 143 | |
| ✓ | ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 887 OUT3 143 |
Algorithms - Moving average
1 passed |
100.0% |
IN8 878 OUT2 280 | |
| ✓ | MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 878 OUT2 280 |
Algorithms - Next greater element
1 passed |
100.0% |
IN8 861 OUT2 143 | |
| ✓ | NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 861 OUT2 143 |
Algorithms - Next lexicographic permutation
1 passed |
100.0% |
IN8 869 OUT3 724 | |
| ✓ | NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 869 OUT3 724 |
Algorithms - Nth Fibonacci mod m
1 passed |
100.0% |
IN8 868 OUT1 068 | |
| ✓ | FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 868 OUT1 068 |
Algorithms - Overlapping substring count
1 passed |
100.0% |
IN8 853 OUT1 102 | |
| ✓ | SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 853 OUT1 102 |
Algorithms - Palindrome & anagram checks
1 passed |
100.0% |
IN8 883 OUT2 805 | |
| ✓ | PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 883 OUT2 805 |
Algorithms - Pascal triangle row
1 passed |
100.0% |
IN8 861 OUT2 884 | |
| ✓ | PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 861 OUT2 884 |
Algorithms - Point-to-segment distance
1 passed |
100.0% |
IN8 901 OUT2 476 | |
| ✓ | PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 901 OUT2 476 |
Algorithms - Prime factorization
1 passed |
100.0% |
IN8 872 OUT2 325 | |
| ✓ | PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 872 OUT2 325 |
Algorithms - Product of array except self
1 passed |
100.0% |
IN8 878 OUT3 981 | |
| ✓ | ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 878 OUT3 981 |
Algorithms - Quadratic Bezier point
1 passed |
100.0% |
IN8 938 OUT1 685 | |
| ✓ | BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 938 OUT1 685 |
Algorithms - Range sum via prefix sums
1 passed |
100.0% |
IN8 860 OUT1 056 | |
| ✓ | PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 860 OUT1 056 |
Algorithms - Reverse words & Hamming distance
1 passed |
100.0% |
IN8 864 OUT1 877 | |
| ✓ | ReverseWordsCodeGenCode-gen: model implements reverse-words + Hamming; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 864 OUT1 877 |
Algorithms - Rod cutting
1 passed |
100.0% |
IN8 894 OUT2 822 | |
| ✓ | RodCuttingCodeGenCode-gen: model implements rod cutting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 894 OUT2 822 |
Algorithms - Roman numeral to integer
1 passed |
100.0% |
IN8 883 OUT2 782 | |
| ✓ | RomanToIntCodeGenCode-gen: model implements Roman-to-int; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 883 OUT2 782 |
Algorithms - Rotate array right by k
1 passed |
100.0% |
IN8 887 OUT1 928 | |
| ✓ | RotateArrayCodeGenCode-gen: model implements right rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 887 OUT1 928 |
Algorithms - Rotate grid 90 CW (discriminator)
1 passed |
100.0% |
IN8 890 OUT4 645 | |
| ✓ | RotateMatrixCodeGenCode-gen: model implements 90-degree CW rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 890 OUT4 645 |
Algorithms - RPN evaluator
1 passed |
100.0% |
IN8 892 OUT2 851 | |
| ✓ | RpnCodeGenCode-gen: model implements an RPN evaluator; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 892 OUT2 851 |
Algorithms - Run-length compression (if shorter)
1 passed |
100.0% |
IN8 864 OUT3 548 | |
| ✓ | StringCompressCodeGenCode-gen: model implements string compression; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 864 OUT3 548 |
Algorithms - Run-length encoding (encode/decode)
1 passed |
100.0% |
IN8 910 OUT3 306 | |
| ✓ | RunLengthCodeGenCode-gen: model implements RLE encode+decode; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 910 OUT3 306 |
Algorithms - Semantic versioning (compare + bump)
1 passed |
100.0% |
IN8 940 OUT1 652 | |
| ✓ | SemVerCodeGenCode-gen: model implements semver compare + bump; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:InstructionFollowinglevel:Easysource:Ai4Users |
Passed | IN8 940 OUT1 652 |
Algorithms - Single-pile Nim (optimal play)
1 passed |
100.0% |
IN8 872 OUT763 | |
| ✓ | NimCodeGenCode-gen: model implements optimal Nim; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 872 OUT763 |
Algorithms - Sliding window maximum
1 passed |
100.0% |
IN8 864 OUT2 788 | |
| ✓ | SlidingWindowMaxCodeGenCode-gen: model implements sliding window maximum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 864 OUT2 788 |
Algorithms - Spiral matrix traversal
1 passed |
100.0% |
IN8 873 OUT2 536 | |
| ✓ | SpiralMatrixCodeGenCode-gen: model implements spiral traversal; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 873 OUT2 536 |
Algorithms - Square matrix multiply
1 passed |
100.0% |
IN8 865 OUT2 917 | |
| ✓ | MatMulCodeGenCode-gen: model implements square matrix multiply; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 865 OUT2 917 |
Algorithms - Strongly connected components
1 passed |
100.0% |
IN8 876 OUT4 198 | |
| ✓ | SccCountCodeGenCode-gen: model implements SCC count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 876 OUT4 198 |
Algorithms - Subset sum reachability
1 passed |
100.0% |
IN8 849 OUT2 998 | |
| ✓ | SubsetSumCodeGenCode-gen: model implements subset-sum reachability; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 849 OUT2 998 |
Algorithms - Substring search (KMP)
1 passed |
100.0% |
IN8 871 OUT2 510 | |
| ✓ | KmpCodeGenCode-gen: model implements KMP search; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 871 OUT2 510 |
Algorithms - Sum of digits & digital root
1 passed |
100.0% |
IN8 850 OUT864 | |
| ✓ | DigitalRootCodeGenCode-gen: model implements sum-of-digits + digital root; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 850 OUT864 |
Algorithms - Temperature conversion
1 passed |
100.0% |
IN8 861 OUT710 | |
| ✓ | TemperatureCodeGenCode-gen: model implements temperature conversion; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 861 OUT710 |
Algorithms - Topological sort (lexicographically smallest)
1 passed |
100.0% |
IN8 902 OUT2 975 | |
| ✓ | TopoCodeGenCode-gen: model implements lexicographically-smallest topological sort.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 902 OUT2 975 |
Algorithms - Trailing zeros of factorial
1 passed |
100.0% |
IN8 862 OUT489 | |
| ✓ | TrailingZerosCodeGenCode-gen: model implements trailing zeros; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 862 OUT489 |
Algorithms - Trapping rain water
1 passed |
100.0% |
IN8 861 OUT2 995 | |
| ✓ | TrappingRainCodeGenCode-gen: model implements trapping rain water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 861 OUT2 995 |
Algorithms - Two-sum indices
1 passed |
100.0% |
IN8 868 OUT2 960 | |
| ✓ | TwoSumCodeGenCode-gen: model implements two-sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 868 OUT2 960 |
Algorithms - Unique grid paths
1 passed |
100.0% |
IN8 843 OUT2 432 | |
| ✓ | UniquePathsCodeGenCode-gen: model implements unique paths; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 843 OUT2 432 |
Algorithms - Valid Sudoku unit
1 passed |
100.0% |
IN8 848 OUT1 033 | |
| ✓ | SudokuUnitCodeGenCode-gen: model implements Sudoku unit validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 848 OUT1 033 |
Algorithms - Wildcard string matching
1 passed |
100.0% |
IN8 866 OUT3 733 | |
| ✓ | WildcardMatchCodeGenCode-gen: model implements wildcard match; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 866 OUT3 733 |
Algorithms - Z-function
1 passed |
100.0% |
IN8 867 OUT3 402 | |
| ✓ | ZFunctionCodeGenCode-gen: model implements the Z-function; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 867 OUT3 402 |
Database - Conditional aggregation pivot
1 passed |
100.0% |
IN945 OUT277 | |
| ✓ | PivotCountsCodeGenCode-gen: model writes the pivot query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN945 OUT277 |
Database - Count distinct opponents
1 passed |
100.0% |
IN925 OUT206 | |
| ✓ | DistinctCountCodeGenCode-gen: model writes the COUNT(DISTINCT) query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Easysource:Ai4Users |
Passed | IN925 OUT206 |
Database - Date-range filter (BETWEEN)
1 passed |
100.0% |
IN955 OUT169 | |
| ✓ | DateRangeCodeGenCode-gen: model writes the date-range query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN955 OUT169 |
Database - EXISTS subquery (teams with matches)
1 passed |
100.0% |
IN932 OUT514 | |
| ✓ | ExistsSubqueryCodeGenCode-gen: model writes the EXISTS query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN932 OUT514 |
Database - JOIN player totals (two tables)
1 passed |
100.0% |
IN954 OUT221 | |
| ✓ | JoinScoresCodeGenCode-gen: model writes a JOIN + aggregate query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN954 OUT221 |
Database - LEFT JOIN anti-join (no items)
1 passed |
100.0% |
IN928 OUT614 | |
| ✓ | LeftJoinNullsCodeGenCode-gen: model writes the anti-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN928 OUT614 |
Database - Player stats SQL (multi-aggregate, QueryJson)
1 passed |
100.0% |
IN958 OUT168 | |
| ✓ | PlayerStatsCodeGenCode-gen: model writes a multi-aggregate stats query; graded by comparing its QueryJson output to the oracle query's (arbitrary shape, no struct).1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN958 OUT168 |
Database - Players above average score (subquery)
1 passed |
100.0% |
IN923 OUT230 | |
| ✓ | AboveAverageCodeGenCode-gen: model writes an above-average query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN923 OUT230 |
Database - Players who never played (anti-join)
1 passed |
100.0% |
IN920 OUT425 | |
| ✓ | NeverPlayedCodeGenCode-gen: model writes an anti-join query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN920 OUT425 |
Database - RANK window function
1 passed |
100.0% |
IN950 OUT750 | |
| ✓ | WindowRankCodeGenCode-gen: model writes the RANK() query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN950 OUT750 |
Database - Running total (window function)
1 passed |
100.0% |
IN947 OUT690 | |
| ✓ | RunningTotalCodeGenCode-gen: model writes the window running-total query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN947 OUT690 |
Database - Scoreboard SQL (players with >1 win, HAVING)
1 passed |
100.0% |
IN954 OUT320 | |
| ✓ | WinFilterCodeGenCode-gen: model writes a HAVING query; runs under try/catch with self-repair; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN954 OUT320 |
Database - Scoreboard SQL (top scorers query)
1 passed |
100.0% |
IN955 OUT112 | |
| ✓ | ScoreboardTopScorersCodeGenCode-gen: model writes a scoreboard SQL query; runs under try/catch with a self-repair loop on SQL errors; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN955 OUT112 |
Database - Self-join same-team pairs
1 passed |
100.0% |
IN938 OUT626 | |
| ✓ | SelfJoinCodeGenCode-gen: model writes the self-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN938 OUT626 |
Database - Top scorer per region
1 passed |
100.0% |
IN940 OUT1 242 | |
| ✓ | TopPerGroupCodeGenCode-gen: model writes the per-group top query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN940 OUT1 242 |
Database - UPDATE restock (SQL write)
1 passed |
100.0% |
IN932 OUT168 | |
| ✓ | UpdateInventoryCodeGenCode-gen: model writes an UPDATE; applied under try/catch, resulting state compared (via QueryJson) to the oracle update's state.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN932 OUT168 |
Database - Upsert with max merge
1 passed |
100.0% |
IN948 OUT1 557 | |
| ✓ | UpsertCodeGenCode-gen: model writes the upsert; executed on a fresh DB and the table compared.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN948 OUT1 557 |
SDLC - Acceptance criteria check
1 passed |
100.0% |
IN8 891 OUT5 252 | |
| ✓ | SdlcAcceptanceCodeGenCode-gen: model implements acceptance-criteria check; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 891 OUT5 252 |
SDLC - Breaking API change detection
1 failed |
0.0% |
IN8 888 OUT1 916 | |
| × | SdlcApiBreakingCodeGenCode-gen: model implements API-breaking detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Breaking([a:x;b:y],[b:z]) got "ab" expected "a b"
Values:
r.ResultJson = "\"ab\""
exp = "\"a b\"" |
Failed | IN8 888 OUT1 916 |
SDLC - Condense without losing facts
1 passed |
100.0% |
IN1 046 OUT2 372 | |
| ✓ | NoInfoLossCodeGenRedaction test: model condenses keeping facts; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 046 OUT2 372 |
SDLC - Config merge with override
1 passed |
100.0% |
IN8 882 OUT3 631 | |
| ✓ | ConfigMergeCodeGenCode-gen: model implements config merge; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 882 OUT3 631 |
SDLC - Extract fields into fixed template
1 passed |
100.0% |
IN1 002 OUT217 | |
| ✓ | FieldExtractCodeGenRedaction test: model extracts fields into the template; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 002 OUT217 |
SDLC - Locate conflicting requirement pair
1 passed |
100.0% |
IN8 944 OUT5 002 | |
| ✓ | SdlcConstraintConflictCodeGenCode-gen: model locates the conflicting requirement pair; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN8 944 OUT5 002 |
SDLC - Locate redundant requirement
1 passed |
100.0% |
IN18 620 OUT7 824 | |
| ✓ | SdlcRedundantRequirementCodeGenCode-gen: model locates a redundant requirement; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN18 620 OUT7 824 |
SDLC - Mask PII to placeholder
1 passed |
100.0% |
IN970 OUT304 | |
| ✓ | MaskPiiCodeGenRedaction test: model masks PII to a placeholder; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN970 OUT304 |
SDLC - Modules on a dependency cycle
1 passed |
100.0% |
IN8 894 OUT5 530 | |
| ✓ | DependencyCycleModulesCodeGenCode-gen: model implements dependency-cycle detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN8 894 OUT5 530 |
SDLC - Remove duplicate requirements
1 passed |
100.0% |
IN943 OUT147 | |
| ✓ | DeduplicateCodeGenRedaction test: model removes duplicate requirements; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN943 OUT147 |
SDLC - Renumber requirements sequentially
1 passed |
100.0% |
IN1 010 OUT516 | |
| ✓ | RenumberCodeGenRedaction test: model renumbers sequentially; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 010 OUT516 |
SDLC - Reorganize notes into section template
1 passed |
100.0% |
IN1 124 OUT458 | |
| ✓ | RewriteSectionsCodeGenRedaction test: model reorganizes the notes; deterministic checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 124 OUT458 |
SDLC - Required semver bump from changelog
1 passed |
100.0% |
IN8 887 OUT1 746 | |
| ✓ | SemverBumpCodeGenCode-gen: model implements the semver-bump rule; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Easysource:Ai4Users |
Passed | IN8 887 OUT1 746 |
SDLC - Requirement-to-test traceability
1 passed |
100.0% |
IN8 909 OUT2 633 | |
| ✓ | SdlcTraceabilityCodeGenCode-gen: model implements traceability gap; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 909 OUT2 633 |
SDLC - Requirements completeness (spec coverage gap)
1 passed |
100.0% |
IN18 228 OUT8 621 | |
| ✓ | SdlcSpecCoverageCodeGenCode-gen: model implements spec-gap detection; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN18 228 OUT8 621 |
SDLC - Requirements consistency (contradiction detection)
1 passed |
100.0% |
IN8 889 OUT8 282 | |
| ✓ | ConsistencyCodeGenCode-gen: model implements requirements-consistency checking; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 889 OUT8 282 |
SDLC - Reshape prose into JSON template
1 passed |
100.0% |
IN1 012 OUT150 | |
| ✓ | JsonShapeCodeGenRedaction test: model reshapes prose into JSON; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 012 OUT150 |
SDLC - Reshape rows into Markdown table
1 passed |
100.0% |
IN966 OUT113 | |
| ✓ | TableFormatCodeGenRedaction test: model builds a Markdown table; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN966 OUT113 |
SDLC - Rewrite in imperative requirement style
1 passed |
100.0% |
IN989 OUT611 | |
| ✓ | StyleRewriteCodeGenRedaction test: model rewrites notes in imperative style; checker + self-review loop.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN989 OUT611 |
SDLC - Roll up estimates over task tree
1 passed |
100.0% |
IN8 900 OUT1 839 | |
| ✓ | EstimateRollupCodeGenCode-gen: model implements estimate roll-up over a task tree; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 900 OUT1 839 |
SDLC - State-machine reachability
1 passed |
100.0% |
IN8 886 OUT3 141 | |
| ✓ | SdlcStateReachableCodeGenCode-gen: model implements state reachability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 886 OUT3 141 |
SDLC - Ticket priority sort
1 passed |
100.0% |
IN8 879 OUT3 297 | |
| ✓ | SdlcPrioritySortCodeGenCode-gen: model implements priority sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 879 OUT3 297 |
SDLC - Unify glossary term consistency
1 passed |
100.0% |
IN970 OUT420 | |
| ✓ | GlossaryCodeGenRedaction test: model unifies synonyms to the canonical term; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN970 OUT420 |