Benchmark summary
A custom Ai4Users benchmark suite for SC#, covering code generation, reasoning, and stateful agentic workflows where models use tools, observe results, adapt their actions, recover from failures, and verify the final state.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Agentic - Blue-Green Deploy (health-gated cutover)
1 passed |
100.0% |
IN14 565 OUT690 | |
| ✓ | AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query
4 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN14 565 OUT690 |
Agentic - Bugfix (root-cause vs regression)
1 passed |
100.0% |
IN19 990 OUT415 | |
| ✓ | AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query
1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN19 990 OUT415 |
Agentic - Cache Invalidation (exact set)
1 passed |
100.0% |
IN14 813 OUT1 301 | |
| ✓ | AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query
8 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN14 813 OUT1 301 |
Agentic - Canary (metric-gated release)
1 passed |
100.0% |
IN30 287 OUT2 447 | |
| ✓ | AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN30 287 OUT2 447 |
Agentic - Canary variants (5 metric cases)
5 passed |
100.0% |
IN158 041 OUT8 371 | |
| ✓ | CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN29 732 OUT1 526 |
| ✓ | CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN31 538 OUT2 084 |
| ✓ | CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN25 957 OUT1 366 |
| ✓ | CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN44 862 OUT2 134 |
| ✓ | CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN25 952 OUT1 261 |
Agentic - Code Review (find injected defect)
1 passed |
100.0% |
IN11 435 OUT1 935 | |
| ✓ | AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query
4 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 435 OUT1 935 |
Agentic - Config Drift (remediate exact)
1 passed |
100.0% |
IN11 823 OUT673 | |
| ✓ | AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query
6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN11 823 OUT673 |
Agentic - Constraints (multi-section satisfaction)
1 passed |
100.0% |
IN11 275 OUT645 | |
| ✓ | AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 275 OUT645 |
Agentic - Constraints variants (5 cases)
5 passed |
100.0% |
IN47 771 OUT3 060 | |
| ✓ | ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 558 OUT784 |
| ✓ | ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 447 OUT563 |
| ✓ | ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 706 OUT531 |
| ✓ | ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 555 OUT595 |
| ✓ | ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 505 OUT587 |
Agentic - Data Reconciliation (per-field precedence)
1 passed |
100.0% |
IN13 382 OUT1 490 | |
| ✓ | AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN13 382 OUT1 490 |
Agentic - Diagnosis (root cause under budget)
1 passed |
100.0% |
IN12 271 OUT810 | |
| ✓ | AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN12 271 OUT810 |
Agentic - Diagnosis variants (5 cases)
5 passed |
100.0% |
IN68 153 OUT5 118 | |
| ✓ | DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN12 020 OUT1 186 |
| ✓ | DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 082 OUT1 027 |
| ✓ | DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 807 OUT825 |
| ✓ | DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query
17 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN13 359 OUT995 |
| ✓ | DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 885 OUT1 085 |
Agentic - ETL Repair (root-cause in stage 2)
1 passed |
100.0% |
IN32 477 OUT1 703 | |
| ✓ | AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN32 477 OUT1 703 |
Agentic - Feature Flag (staged rollout + kill-switch)
1 passed |
100.0% |
IN20 776 OUT807 | |
| ✓ | AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN20 776 OUT807 |
Agentic - Game Backend: 1D bouncing ball
1 passed |
100.0% |
IN8 781 OUT891 | |
| ✓ | GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 781 OUT891 |
Agentic - Game Backend: 1D swept anti-tunneling
1 passed |
100.0% |
IN8 792 OUT1 305 | |
| ✓ | GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 792 OUT1 305 |
Agentic - Game Backend: 2048 left-move (slide + merge)
1 passed |
100.0% |
IN8 801 OUT3 364 | |
| ✓ | GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 801 OUT3 364 |
Agentic - Game Backend: 8-direction from vector
1 passed |
100.0% |
IN8 808 OUT423 | |
| ✓ | GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 808 OUT423 |
Agentic - Game Backend: AABB collision resolution (MTV)
1 passed |
100.0% |
IN8 897 OUT1 180 | |
| ✓ | GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 897 OUT1 180 |
Agentic - Game Backend: AABB containment
1 passed |
100.0% |
IN8 737 OUT200 | |
| ✓ | GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 737 OUT200 |
Agentic - Game Backend: Acceleration to max speed
1 passed |
100.0% |
IN8 739 OUT2 148 | |
| ✓ | GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 739 OUT2 148 |
Agentic - Game Backend: Achievement thresholds
1 passed |
100.0% |
IN8 731 OUT925 | |
| ✓ | GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 731 OUT925 |
Agentic - Game Backend: Active status effects
1 passed |
100.0% |
IN8 753 OUT906 | |
| ✓ | GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT906 |
Agentic - Game Backend: Animation frame at time
1 passed |
100.0% |
IN8 771 OUT750 | |
| ✓ | GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 771 OUT750 |
Agentic - Game Backend: Army upkeep income
1 passed |
100.0% |
IN8 719 OUT319 | |
| ✓ | GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 719 OUT319 |
Agentic - Game Backend: Axis-aligned path follow
1 passed |
100.0% |
IN8 762 OUT2 578 | |
| ✓ | GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 762 OUT2 578 |
Agentic - Game Backend: Battleship shot resolution
1 passed |
100.0% |
IN8 821 OUT1 697 | |
| ✓ | GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 821 OUT1 697 |
Agentic - Game Backend: Bounce restitution
1 passed |
100.0% |
IN8 758 OUT440 | |
| ✓ | GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 758 OUT440 |
Agentic - Game Backend: Bresenham line rasterization
1 passed |
100.0% |
IN8 848 OUT2 236 | |
| ✓ | GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 848 OUT2 236 |
Agentic - Game Backend: Buff/debuff stacking
1 passed |
100.0% |
IN8 784 OUT3 171 | |
| ✓ | GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 784 OUT3 171 |
Agentic - Game Backend: Build queue completions
1 passed |
100.0% |
IN8 752 OUT663 | |
| ✓ | GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 752 OUT663 |
Agentic - Game Backend: Camera-relative movement
1 passed |
100.0% |
IN8 839 OUT1 099 | |
| ✓ | GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 839 OUT1 099 |
Agentic - Game Backend: Card hand sort
1 passed |
100.0% |
IN8 760 OUT1 493 | |
| ✓ | GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 760 OUT1 493 |
Agentic - Game Backend: Card-game rules (blackjack/deck/compare)
1 passed |
100.0% |
IN18 637 OUT5 622 | |
| ✓ | GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 637 OUT5 622 |
Agentic - Game Backend: Chart scaling & histogram bins
1 passed |
100.0% |
IN8 824 OUT6 593 | |
| ✓ | GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 824 OUT6 593 |
Agentic - Game Backend: Checkers captures
1 passed |
100.0% |
IN8 794 OUT1 095 | |
| ✓ | GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 794 OUT1 095 |
Agentic - Game Backend: Collision primitives (circle/aabb/segment)
1 passed |
100.0% |
IN8 997 OUT2 414 | |
| ✓ | GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 997 OUT2 414 |
Agentic - Game Backend: Combat turn order (initiative)
1 passed |
100.0% |
IN17 842 OUT3 274 | |
| ✓ | GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 842 OUT3 274 |
Agentic - Game Backend: Combo scoring (rising multiplier)
1 passed |
100.0% |
IN8 756 OUT979 | |
| ✓ | GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 756 OUT979 |
Agentic - Game Backend: Control-point capture
1 passed |
100.0% |
IN8 769 OUT1 512 | |
| ✓ | GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT1 512 |
Agentic - Game Backend: Controls & Inventory (spec-driven function battery)
1 passed |
100.0% |
IN8 910 OUT1 360 | |
| ✓ | GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 910 OUT1 360 |
Agentic - Game Backend: Cooldowns & resource regen
1 passed |
100.0% |
IN8 827 OUT600 | |
| ✓ | GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 827 OUT600 |
Agentic - Game Backend: Coyote-time jump
1 passed |
100.0% |
IN8 748 OUT264 | |
| ✓ | GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 748 OUT264 |
Agentic - Game Backend: Crafting capacity
1 passed |
100.0% |
IN8 772 OUT1 845 | |
| ✓ | GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT1 845 |
Agentic - Game Backend: Cribbage fifteens
1 passed |
100.0% |
IN8 749 OUT911 | |
| ✓ | GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 749 OUT911 |
Agentic - Game Backend: Critical-hit damage
1 passed |
100.0% |
IN8 739 OUT260 | |
| ✓ | GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 739 OUT260 |
Agentic - Game Backend: Damage over time
1 passed |
100.0% |
IN8 765 OUT741 | |
| ✓ | GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 765 OUT741 |
Agentic - Game Backend: Day/night cycle phase
1 passed |
100.0% |
IN8 751 OUT662 | |
| ✓ | GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 751 OUT662 |
Agentic - Game Backend: Dice notation max roll
1 passed |
100.0% |
IN8 766 OUT1 611 | |
| ✓ | GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 766 OUT1 611 |
Agentic - Game Backend: Enemy AI state machine
1 passed |
100.0% |
IN8 852 OUT1 427 | |
| ✓ | GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 852 OUT1 427 |
Agentic - Game Backend: Equipment slot validation
1 passed |
100.0% |
IN8 766 OUT593 | |
| ✓ | GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 766 OUT593 |
Agentic - Game Backend: Flood-fill region size
1 passed |
100.0% |
IN8 754 OUT1 532 | |
| ✓ | GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT1 532 |
Agentic - Game Backend: Fog of war visibility
1 passed |
100.0% |
IN8 794 OUT1 797 | |
| ✓ | GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 794 OUT1 797 |
Agentic - Game Backend: Fog reveal (Chebyshev radius)
1 passed |
100.0% |
IN8 767 OUT14 594 | |
| ✓ | GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 767 OUT14 594 |
Agentic - Game Backend: Friction stop frame
1 passed |
100.0% |
IN8 747 OUT499 | |
| ✓ | GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 747 OUT499 |
Agentic - Game Backend: Game of Life (code-gen vs oracle)
1 passed |
100.0% |
IN8 806 OUT1 097 | |
| ✓ | GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 806 OUT1 097 |
Agentic - Game Backend: Greedy coin change
1 passed |
100.0% |
IN8 748 OUT595 | |
| ✓ | GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 748 OUT595 |
Agentic - Game Backend: Grid inventory & stacking (Diablo-style)
1 passed |
100.0% |
IN8 881 OUT4 353 | |
| ✓ | GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 881 OUT4 353 |
Agentic - Game Backend: Grid pathfinding (shortest path, BFS)
1 failed |
0.0% |
IN8 836 OUT2 661 | |
| × | GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.1 AI query
0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: ShortestPath mismatch grid 6x5 (3,3)->(5,3) got -1 expected 2
Values:
r.ResultJson = "-1" |
Failed | IN8 836 OUT2 661 |
Agentic - Game Backend: Grid raycast
1 passed |
100.0% |
IN8 804 OUT1 001 | |
| ✓ | GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 804 OUT1 001 |
Agentic - Game Backend: Health bar segments
1 passed |
100.0% |
IN8 805 OUT1 389 | |
| ✓ | GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 805 OUT1 389 |
Agentic - Game Backend: High-score table insert
1 passed |
100.0% |
IN8 759 OUT3 346 | |
| ✓ | GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 759 OUT3 346 |
Agentic - Game Backend: Hit chance
1 passed |
100.0% |
IN8 743 OUT514 | |
| ✓ | GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 743 OUT514 |
Agentic - Game Backend: Inventory encumbrance
1 passed |
100.0% |
IN8 740 OUT570 | |
| ✓ | GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 740 OUT570 |
Agentic - Game Backend: Inventory first-fit placement
1 passed |
100.0% |
IN8 806 OUT1 316 | |
| ✓ | GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 806 OUT1 316 |
Agentic - Game Backend: Inventory stack split
1 passed |
100.0% |
IN8 755 OUT396 | |
| ✓ | GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 755 OUT396 |
Agentic - Game Backend: Isometric projection
1 passed |
100.0% |
IN8 773 OUT354 | |
| ✓ | GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 773 OUT354 |
Agentic - Game Backend: Jump buffer
1 passed |
100.0% |
IN8 761 OUT923 | |
| ✓ | GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT923 |
Agentic - Game Backend: Knockback with arena clamp
1 passed |
100.0% |
IN8 800 OUT472 | |
| ✓ | GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 800 OUT472 |
Agentic - Game Backend: Leaderboard rank
1 passed |
100.0% |
IN8 733 OUT547 | |
| ✓ | GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 733 OUT547 |
Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over)
1 passed |
100.0% |
IN8 884 OUT756 | |
| ✓ | GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 884 OUT756 |
Agentic - Game Backend: Lifesteal healing
1 passed |
100.0% |
IN8 744 OUT2 266 | |
| ✓ | GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 744 OUT2 266 |
Agentic - Game Backend: Lights Out toggle
1 passed |
100.0% |
IN8 760 OUT906 | |
| ✓ | GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 760 OUT906 |
Agentic - Game Backend: Match-3 cascade
1 passed |
100.0% |
IN8 818 OUT6 202 | |
| ✓ | GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 818 OUT6 202 |
Agentic - Game Backend: Match-3 detection
1 passed |
100.0% |
IN8 782 OUT1 805 | |
| ✓ | GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 782 OUT1 805 |
Agentic - Game Backend: Match-3 swap validity
1 passed |
100.0% |
IN8 791 OUT7 000 | |
| ✓ | GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 791 OUT7 000 |
Agentic - Game Backend: Memory match pairs
1 passed |
100.0% |
IN8 745 OUT736 | |
| ✓ | GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 745 OUT736 |
Agentic - Game Backend: Mercy invincibility (i-frames)
1 passed |
100.0% |
IN8 800 OUT1 361 | |
| ✓ | GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 800 OUT1 361 |
Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal)
1 passed |
100.0% |
IN8 882 OUT6 218 | |
| ✓ | GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 882 OUT6 218 |
Agentic - Game Backend: Minimap downsample
1 passed |
100.0% |
IN8 776 OUT939 | |
| ✓ | GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 776 OUT939 |
Agentic - Game Backend: Minimap marker position
1 passed |
100.0% |
IN8 779 OUT1 545 | |
| ✓ | GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 779 OUT1 545 |
Agentic - Game Backend: N-in-a-row win detection (Connect-4)
1 passed |
100.0% |
IN8 794 OUT2 142 | |
| ✓ | GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 794 OUT2 142 |
Agentic - Game Backend: Nonogram row validation
1 passed |
100.0% |
IN8 756 OUT2 139 | |
| ✓ | GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT2 139 |
Agentic - Game Backend: OBB collision (rotating rectangles, SAT)
1 passed |
100.0% |
IN8 903 OUT22 514 | |
| ✓ | GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 903 OUT22 514 |
Agentic - Game Backend: Particle lifetime & expiry
1 passed |
100.0% |
IN8 838 OUT1 736 | |
| ✓ | GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 838 OUT1 736 |
Agentic - Game Backend: Party XP share
1 passed |
100.0% |
IN8 756 OUT400 | |
| ✓ | GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 756 OUT400 |
Agentic - Game Backend: Patrol waypoints (ping-pong)
1 passed |
100.0% |
IN8 779 OUT1 756 | |
| ✓ | GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 779 OUT1 756 |
Agentic - Game Backend: Platformer jump arc
1 passed |
100.0% |
IN8 770 OUT581 | |
| ✓ | GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 770 OUT581 |
Agentic - Game Backend: Point on circle (trig)
1 passed |
100.0% |
IN8 803 OUT887 | |
| ✓ | GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 803 OUT887 |
Agentic - Game Backend: Poker hand comparison
1 failed |
0.0% |
IN8 879 OUT15 216 | |
| × | GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users Compare threw: List index 5 is out of range. The list has 5 element(s). |
Failed | IN8 879 OUT15 216 |
Agentic - Game Backend: Poker hand ranking
1 passed |
100.0% |
IN8 822 OUT3 229 | |
| ✓ | GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 822 OUT3 229 |
Agentic - Game Backend: Pop-cap training
1 passed |
100.0% |
IN8 757 OUT526 | |
| ✓ | GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 757 OUT526 |
Agentic - Game Backend: Progress bar fill
1 passed |
100.0% |
IN8 739 OUT346 | |
| ✓ | GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 739 OUT346 |
Agentic - Game Backend: Projectile range (physics/trig)
1 passed |
100.0% |
IN8 761 OUT892 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT892 |
Agentic - Game Backend: Projectile with ground bounce
1 passed |
100.0% |
IN8 795 OUT308 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 795 OUT308 |
Agentic - Game Backend: Resource gathering
1 passed |
100.0% |
IN8 756 OUT1 520 | |
| ✓ | GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 756 OUT1 520 |
Agentic - Game Backend: Reversi flip count
1 failed |
0.0% |
IN8 827 OUT2 145 | |
| × | GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users FlipCount threw: The script exceeded the timeout of 30000 ms. |
Failed | IN8 827 OUT2 145 |
Agentic - Game Backend: Rotate 2D vector (trig)
1 passed |
100.0% |
IN8 772 OUT711 | |
| ✓ | GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT711 |
Agentic - Game Backend: Round-robin deal
1 passed |
100.0% |
IN8 761 OUT1 565 | |
| ✓ | GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 761 OUT1 565 |
Agentic - Game Backend: RTS resource economy
1 passed |
100.0% |
IN8 752 OUT412 | |
| ✓ | GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 752 OUT412 |
Agentic - Game Backend: RTS unit counters & damage
1 passed |
100.0% |
IN8 804 OUT878 | |
| ✓ | GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 804 OUT878 |
Agentic - Game Backend: Safest neighbour (danger map)
1 passed |
100.0% |
IN8 792 OUT1 990 | |
| ✓ | GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 792 OUT1 990 |
Agentic - Game Backend: Scroll viewport
1 passed |
100.0% |
IN8 774 OUT946 | |
| ✓ | GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 774 OUT946 |
Agentic - Game Backend: Set card validity
1 passed |
100.0% |
IN8 753 OUT828 | |
| ✓ | GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT828 |
Agentic - Game Backend: Shop affordability
1 passed |
100.0% |
IN8 733 OUT591 | |
| ✓ | GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 733 OUT591 |
Agentic - Game Backend: Sliding puzzle move
1 passed |
100.0% |
IN8 764 OUT1 393 | |
| ✓ | GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 764 OUT1 393 |
Agentic - Game Backend: Snake head movement
1 passed |
100.0% |
IN8 821 OUT1 202 | |
| ✓ | GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 821 OUT1 202 |
Agentic - Game Backend: Snake tick
1 passed |
100.0% |
IN8 891 OUT1 975 | |
| ✓ | GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 891 OUT1 975 |
Agentic - Game Backend: Snap to grid
1 passed |
100.0% |
IN8 749 OUT276 | |
| ✓ | GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 749 OUT276 |
Agentic - Game Backend: Sokoban move/push legality
1 passed |
100.0% |
IN8 852 OUT1 492 | |
| ✓ | GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 852 OUT1 492 |
Agentic - Game Backend: Solitaire tableau move
1 passed |
100.0% |
IN8 791 OUT1 930 | |
| ✓ | GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 791 OUT1 930 |
Agentic - Game Backend: Spawn scheduler count
1 passed |
100.0% |
IN8 748 OUT1 492 | |
| ✓ | GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 748 OUT1 492 |
Agentic - Game Backend: Spawn-wave alive count
1 passed |
100.0% |
IN8 760 OUT593 | |
| ✓ | GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 760 OUT593 |
Agentic - Game Backend: Stat level scaling
1 passed |
100.0% |
IN8 752 OUT286 | |
| ✓ | GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT286 |
Agentic - Game Backend: Steering seek (greedy pursuit)
1 passed |
100.0% |
IN8 778 OUT3 061 | |
| ✓ | GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 778 OUT3 061 |
Agentic - Game Backend: Tactical movement range
1 passed |
100.0% |
IN8 790 OUT3 102 | |
| ✓ | GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 790 OUT3 102 |
Agentic - Game Backend: Tech tree eligibility
1 passed |
100.0% |
IN8 772 OUT1 759 | |
| ✓ | GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT1 759 |
Agentic - Game Backend: Tetris line clearing
1 passed |
100.0% |
IN8 763 OUT954 | |
| ✓ | GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 763 OUT954 |
Agentic - Game Backend: Threat table top target
1 passed |
100.0% |
IN8 757 OUT1 497 | |
| ✓ | GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 757 OUT1 497 |
Agentic - Game Backend: Tic-tac-toe result
1 passed |
100.0% |
IN8 748 OUT1 323 | |
| ✓ | GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 748 OUT1 323 |
Agentic - Game Backend: Toroidal wrap (Pac-Man)
1 passed |
100.0% |
IN8 784 OUT389 | |
| ✓ | GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 784 OUT389 |
Agentic - Game Backend: Tower targeting (nearest in range)
1 passed |
100.0% |
IN8 772 OUT1 556 | |
| ✓ | GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT1 556 |
Agentic - Game Backend: Transaction ledger
1 passed |
100.0% |
IN8 775 OUT968 | |
| ✓ | GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 775 OUT968 |
Agentic - Game Backend: Trick-taking winner
1 passed |
100.0% |
IN8 803 OUT3 304 | |
| ✓ | GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 803 OUT3 304 |
Agentic - Game Backend: Turn-based duel resolution
1 passed |
100.0% |
IN8 835 OUT1 489 | |
| ✓ | GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 835 OUT1 489 |
Agentic - Game Backend: Typed armor mitigation
1 passed |
100.0% |
IN8 759 OUT2 188 | |
| ✓ | GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 759 OUT2 188 |
Agentic - Game Backend: UI menu (wrap/clamp/paginate)
1 passed |
100.0% |
IN8 851 OUT1 242 | |
| ✓ | GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 851 OUT1 242 |
Agentic - Game Backend: Uno playable cards
1 passed |
100.0% |
IN8 787 OUT822 | |
| ✓ | GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 787 OUT822 |
Agentic - Game Backend: Velocity reflection
1 passed |
100.0% |
IN8 731 OUT422 | |
| ✓ | GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 731 OUT422 |
Agentic - Game Backend: Wall-slide fall
1 passed |
100.0% |
IN8 739 OUT349 | |
| ✓ | GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 739 OUT349 |
Agentic - Game Backend: War card compare
1 passed |
100.0% |
IN8 759 OUT898 | |
| ✓ | GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 759 OUT898 |
Agentic - Game Backend: Weighted loot table
1 passed |
100.0% |
IN8 762 OUT903 | |
| ✓ | GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 762 OUT903 |
Agentic - Game Backend: Weighted terrain pathfinding
1 passed |
100.0% |
IN8 812 OUT13 737 | |
| ✓ | GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 812 OUT13 737 |
Agentic - Game Backend: Word-wrap line count
1 passed |
100.0% |
IN8 744 OUT1 822 | |
| ✓ | GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT1 822 |
Agentic - Game Backend: XP to character level
1 passed |
100.0% |
IN8 752 OUT3 715 | |
| ✓ | GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT3 715 |
Agentic - Idempotency (poison-retry / no double charge)
1 passed |
100.0% |
IN15 809 OUT1 196 | |
| ✓ | AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query
10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN15 809 OUT1 196 |
Agentic - Irreversible Commit (measure twice)
1 passed |
100.0% |
IN19 961 OUT1 127 | |
| ✓ | AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query
10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN19 961 OUT1 127 |
Agentic - Least Privilege (exact grants)
1 passed |
100.0% |
IN11 780 OUT398 | |
| ✓ | AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query
6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN11 780 OUT398 |
Agentic - Ledger (simulated DB read-modify-write)
1 passed |
100.0% |
IN12 352 OUT1 528 | |
| ✓ | AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 352 OUT1 528 |
Agentic - Ledger variants (5 RMW cases)
5 passed |
100.0% |
IN77 799 OUT9 575 | |
| ✓ | LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN19 619 OUT2 839 |
| ✓ | LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 989 OUT1 277 |
| ✓ | LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN13 614 OUT1 544 |
| ✓ | LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query
6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN11 815 OUT1 955 |
| ✓ | LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN19 762 OUT1 960 |
Agentic - Lock Ordering (deadlock avoidance)
1 passed |
100.0% |
IN30 071 OUT2 156 | |
| ✓ | AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query
25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN30 071 OUT2 156 |
Agentic - Lock Ordering variants (5 cases)
5 passed |
100.0% |
IN190 551 OUT9 149 | |
| ✓ | LockVariantFiveResFourOps1 AI query
23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN29 416 OUT2 773 |
| ✓ | LockVariantFourResThreeOps1 AI query
18 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN57 901 OUT1 990 |
| ✓ | LockVariantHeavyReuse1 AI query
23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN80 962 OUT2 312 |
| ✓ | LockVariantReverseListed1 AI query
24 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN11 390 OUT1 087 |
| ✓ | LockVariantThreeLockOp1 AI query
20 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN10 882 OUT987 |
Agentic - Migration (cascading rollback + reapply)
1 passed |
100.0% |
IN22 519 OUT1 653 | |
| ✓ | AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query
11 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN22 519 OUT1 653 |
Agentic - Requirements (detect contradiction)
1 passed |
100.0% |
IN7 298 OUT806 | |
| ✓ | AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query
2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN7 298 OUT806 |
Agentic - Retry/Circuit-breaker (transient vs hard-down)
1 passed |
100.0% |
IN25 713 OUT899 | |
| ✓ | AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query
10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN25 713 OUT899 |
Agentic - Saga (compensation / distributed rollback)
1 passed |
100.0% |
IN20 985 OUT748 | |
| ✓ | AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN20 985 OUT748 |
Agentic - Saga variants (5 compensation cases)
5 passed |
100.0% |
IN109 591 OUT3 561 | |
| ✓ | SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 408 OUT654 |
| ✓ | SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 583 OUT463 |
| ✓ | SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query
7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN23 201 OUT981 |
| ✓ | SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query
7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN22 992 OUT618 |
| ✓ | SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query
9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN30 407 OUT845 |
Agentic - Schema Compat (expand-contract)
1 passed |
100.0% |
IN22 404 OUT1 002 | |
| ✓ | AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query
7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN22 404 OUT1 002 |
Agentic - Scratchpad (self-repair, Dyck, tool-decision)
7 passed |
100.0% |
IN149 914 OUT19 590 | |
| ✓ | AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query
1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN21 455 OUT4 064 |
| ✓ | AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query
2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN32 267 OUT3 396 |
| ✓ | ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query
0 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN10 526 OUT41 |
| ✓ | ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query
0 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN10 548 OUT224 |
| ✓ | ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query
1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN21 156 OUT878 |
| ✓ | ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query
1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN21 604 OUT8 571 |
| ✓ | ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query
2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN32 358 OUT2 416 |
Agentic - Secret Rotation (zero-downtime ordering)
1 passed |
100.0% |
IN17 190 OUT1 126 | |
| ✓ | AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN17 190 OUT1 126 |
Agentic - StatefulOps (incident recovery, concurrency replan)
2 passed |
100.0% |
IN58 446 OUT2 698 | |
| ✓ | AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query
7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN22 750 OUT1 525 |
| ✓ | AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query
9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN35 696 OUT1 173 |
Agentic - Test Authoring (write a test that catches the bug)
1 passed |
100.0% |
IN8 421 OUT484 | |
| ✓ | AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query
5 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN8 421 OUT484 |
Agentic - Topological Planning (deps + prereq)
1 passed |
100.0% |
IN28 465 OUT1 186 | |
| ✓ | AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN28 465 OUT1 186 |
Algorithms - 0/1 Knapsack (dynamic programming)
1 passed |
100.0% |
IN8 766 OUT13 183 | |
| ✓ | KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 766 OUT13 183 |
Algorithms - 1D elastic collision
1 passed |
100.0% |
IN8 806 OUT1 178 | |
| ✓ | ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 806 OUT1 178 |
Algorithms - All-pairs shortest path pair
1 passed |
100.0% |
IN8 774 OUT21 765 | |
| ✓ | FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 774 OUT21 765 |
Algorithms - Balanced brackets (stack)
1 passed |
100.0% |
IN8 737 OUT516 | |
| ✓ | ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 737 OUT516 |
Algorithms - Bellman-Ford shortest path (neg weights/cycles)
1 passed |
100.0% |
IN8 783 OUT20 968 | |
| ✓ | BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 783 OUT20 968 |
Algorithms - Binary Gray code
1 passed |
100.0% |
IN8 710 OUT193 | |
| ✓ | GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 710 OUT193 |
Algorithms - Bipartite graph check
1 passed |
100.0% |
IN8 756 OUT1 846 | |
| ✓ | BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT1 846 |
Algorithms - Bit tricks (popcount/pow2/lowbit)
1 passed |
100.0% |
IN8 784 OUT1 110 | |
| ✓ | BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 784 OUT1 110 |
Algorithms - Caesar cipher
1 passed |
100.0% |
IN8 733 OUT1 025 | |
| ✓ | CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT1 025 |
Algorithms - Catalan number
1 passed |
100.0% |
IN8 755 OUT548 | |
| ✓ | CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT548 |
Algorithms - Chinese remainder theorem
1 passed |
100.0% |
IN8 773 OUT11 420 | |
| ✓ | CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 773 OUT11 420 |
Algorithms - Circle-circle MTV
1 passed |
100.0% |
IN8 850 OUT3 515 | |
| ✓ | CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 850 OUT3 515 |
Algorithms - Coin change combinations
1 failed |
0.0% |
IN8 740 OUT5 606 | |
| × | CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: Ways(11,[3 3 4]) got 1 expected 2
Values:
r.ResultJson = "1" |
Failed | IN8 740 OUT5 606 |
Algorithms - Collatz step count
1 passed |
100.0% |
IN8 736 OUT531 | |
| ✓ | CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 736 OUT531 |
Algorithms - Connected components (union-find)
1 passed |
100.0% |
IN8 752 OUT1 734 | |
| ✓ | UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT1 734 |
Algorithms - Container with most water
1 passed |
100.0% |
IN8 744 OUT1 108 | |
| ✓ | ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT1 108 |
Algorithms - Count anagram groups
1 passed |
100.0% |
IN8 727 OUT816 | |
| ✓ | AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 727 OUT816 |
Algorithms - Count grid color regions
1 passed |
100.0% |
IN8 747 OUT1 372 | |
| ✓ | CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 747 OUT1 372 |
Algorithms - Count islands
1 passed |
100.0% |
IN8 750 OUT2 466 | |
| ✓ | CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 750 OUT2 466 |
Algorithms - Count overlapping occurrences
1 passed |
100.0% |
IN8 733 OUT517 | |
| ✓ | RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT517 |
Algorithms - Count primes (Sieve of Eratosthenes)
1 passed |
100.0% |
IN8 712 OUT1 373 | |
| ✓ | CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 712 OUT1 373 |
Algorithms - Cumulative distribution
1 passed |
100.0% |
IN8 734 OUT1 241 | |
| ✓ | CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 734 OUT1 241 |
Algorithms - Dijkstra shortest path (weighted graph)
1 passed |
100.0% |
IN8 786 OUT17 251 | |
| ✓ | DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 786 OUT17 251 |
Algorithms - Directed cycle detection
1 passed |
100.0% |
IN8 743 OUT2 733 | |
| ✓ | DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 743 OUT2 733 |
Algorithms - Distinct subsequences
1 passed |
100.0% |
IN8 729 OUT904 | |
| ✓ | DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 729 OUT904 |
Algorithms - Divisor count
1 passed |
100.0% |
IN8 717 OUT547 | |
| ✓ | DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 717 OUT547 |
Algorithms - Downsample by block average
1 passed |
100.0% |
IN8 758 OUT1 813 | |
| ✓ | ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 758 OUT1 813 |
Algorithms - Dutch national flag sort
1 passed |
100.0% |
IN8 725 OUT1 879 | |
| ✓ | DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 725 OUT1 879 |
Algorithms - Egg drop trials
1 passed |
100.0% |
IN8 786 OUT3 170 | |
| ✓ | EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 786 OUT3 170 |
Algorithms - Equal subset-sum partition
1 passed |
100.0% |
IN8 728 OUT3 085 | |
| ✓ | PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 728 OUT3 085 |
Algorithms - Euler totient
1 passed |
100.0% |
IN8 717 OUT1 006 | |
| ✓ | EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 717 OUT1 006 |
Algorithms - Eulerian path existence
1 passed |
100.0% |
IN18 191 OUT8 160 | |
| ✓ | EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN18 191 OUT8 160 |
Algorithms - Extended Euclidean
1 passed |
100.0% |
IN8 829 OUT2 843 | |
| ✓ | ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 829 OUT2 843 |
Algorithms - Format duration H:MM:SS
1 passed |
100.0% |
IN8 744 OUT571 | |
| ✓ | FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 744 OUT571 |
Algorithms - GCD and LCM
1 passed |
100.0% |
IN8 721 OUT1 693 | |
| ✓ | GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 721 OUT1 693 |
Algorithms - Grid transpose
1 passed |
100.0% |
IN8 752 OUT637 | |
| ✓ | TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT637 |
Algorithms - House robber (max non-adjacent sum)
1 passed |
100.0% |
IN8 716 OUT841 | |
| ✓ | HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 716 OUT841 |
Algorithms - Integer linear interpolation
1 passed |
100.0% |
IN8 750 OUT255 | |
| ✓ | LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 750 OUT255 |
Algorithms - Integer square root
1 passed |
100.0% |
IN8 732 OUT328 | |
| ✓ | IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 732 OUT328 |
Algorithms - Integer to base-N string
1 passed |
100.0% |
IN8 738 OUT954 | |
| ✓ | BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 738 OUT954 |
Algorithms - Integer to Roman numeral
1 failed |
0.0% |
IN8 767 OUT1 056 | |
| × | RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: ToRoman(3888) got "MMMCDCCCXLXXXVIII" expected "MMMDCCCLXXXVIII"
Values:
r.ResultJson = "\"MMMCDCCCXLXXXVIII\""
exp = "\"MMMDCCCLXXXVIII\"" |
Failed | IN8 767 OUT1 056 |
Algorithms - Josephus survivor
1 passed |
100.0% |
IN8 743 OUT454 | |
| ✓ | JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 743 OUT454 |
Algorithms - Jump game (min jumps)
1 passed |
100.0% |
IN8 751 OUT2 399 | |
| ✓ | JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 751 OUT2 399 |
Algorithms - Kth largest element
1 passed |
100.0% |
IN8 741 OUT362 | |
| ✓ | KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 741 OUT362 |
Algorithms - Least-squares line fit
1 passed |
100.0% |
IN8 783 OUT2 968 | |
| ✓ | LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 783 OUT2 968 |
Algorithms - Levenshtein edit distance (DP)
1 passed |
100.0% |
IN8 711 OUT737 | |
| ✓ | EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 711 OUT737 |
Algorithms - Log-scale bucket index
1 passed |
100.0% |
IN8 776 OUT1 449 | |
| ✓ | LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 776 OUT1 449 |
Algorithms - Logarithmic bin index
1 passed |
100.0% |
IN8 765 OUT2 070 | |
| ✓ | LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 765 OUT2 070 |
Algorithms - Longest common prefix
1 passed |
100.0% |
IN8 724 OUT865 | |
| ✓ | CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 724 OUT865 |
Algorithms - Longest common subsequence
1 passed |
100.0% |
IN8 709 OUT438 | |
| ✓ | LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 709 OUT438 |
Algorithms - Longest common substring
1 passed |
100.0% |
IN8 726 OUT11 471 | |
| ✓ | CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 726 OUT11 471 |
Algorithms - Longest increasing subsequence (DP)
1 passed |
100.0% |
IN8 743 OUT1 748 | |
| ✓ | LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 743 OUT1 748 |
Algorithms - Longest palindromic substring
1 passed |
100.0% |
IN8 721 OUT3 907 | |
| ✓ | LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 721 OUT3 907 |
Algorithms - Longest repeated substring
1 passed |
100.0% |
IN8 726 OUT20 774 | |
| ✓ | LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 726 OUT20 774 |
Algorithms - Lower bound (binary search)
1 passed |
100.0% |
IN8 754 OUT746 | |
| ✓ | LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT746 |
Algorithms - Luhn checksum validation
1 passed |
100.0% |
IN8 755 OUT1 201 | |
| ✓ | LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT1 201 |
Algorithms - Majority element
1 passed |
100.0% |
IN8 719 OUT1 991 | |
| ✓ | MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 719 OUT1 991 |
Algorithms - Matrix chain multiplication
1 failed |
0.0% |
IN18 137 OUT2 092 | |
| × | MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: MinMult([40 20 30 10 30]) got 0 expected 26000
Values:
r.ResultJson = "0" |
Failed | IN18 137 OUT2 092 |
Algorithms - Maximum subarray sum (Kadane)
1 passed |
100.0% |
IN8 744 OUT669 | |
| ✓ | MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT669 |
Algorithms - Merge K sorted lists
1 passed |
100.0% |
IN8 737 OUT1 139 | |
| ✓ | MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 737 OUT1 139 |
Algorithms - Merge overlapping intervals
1 passed |
100.0% |
IN8 769 OUT2 525 | |
| ✓ | MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT2 525 |
Algorithms - Min-max normalization
1 passed |
100.0% |
IN8 746 OUT1 198 | |
| ✓ | MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 746 OUT1 198 |
Algorithms - Minimum coin change (DP)
1 passed |
100.0% |
IN8 737 OUT3 328 | |
| ✓ | CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 737 OUT3 328 |
Algorithms - Minimum knight moves (BFS)
1 passed |
100.0% |
IN8 775 OUT1 767 | |
| ✓ | KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 775 OUT1 767 |
Algorithms - Minimum path sum (grid DP)
1 passed |
100.0% |
IN8 750 OUT1 899 | |
| ✓ | MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 750 OUT1 899 |
Algorithms - Minimum spanning tree (Kruskal)
1 passed |
100.0% |
IN18 250 OUT3 229 | |
| ✓ | KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN18 250 OUT3 229 |
Algorithms - Minimum window substring
1 passed |
100.0% |
IN8 758 OUT3 051 | |
| ✓ | MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 758 OUT3 051 |
Algorithms - Modular exponentiation (fast pow)
1 passed |
100.0% |
IN8 769 OUT922 | |
| ✓ | ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT922 |
Algorithms - Modular inverse (extended Euclid)
1 passed |
100.0% |
IN8 757 OUT1 465 | |
| ✓ | ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 757 OUT1 465 |
Algorithms - Moving average
1 passed |
100.0% |
IN8 748 OUT2 733 | |
| ✓ | MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 748 OUT2 733 |
Algorithms - Next greater element
1 passed |
100.0% |
IN8 731 OUT1 705 | |
| ✓ | NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 731 OUT1 705 |
Algorithms - Next lexicographic permutation
1 passed |
100.0% |
IN8 739 OUT2 132 | |
| ✓ | NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 739 OUT2 132 |
Algorithms - Nth Fibonacci mod m
1 passed |
100.0% |
IN8 738 OUT608 | |
| ✓ | FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 738 OUT608 |
Algorithms - Overlapping substring count
1 passed |
100.0% |
IN8 723 OUT403 | |
| ✓ | SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 723 OUT403 |
Algorithms - Palindrome & anagram checks
1 passed |
100.0% |
IN8 753 OUT2 324 | |
| ✓ | PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT2 324 |
Algorithms - Pascal triangle row
1 passed |
100.0% |
IN8 731 OUT821 | |
| ✓ | PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 731 OUT821 |
Algorithms - Point-to-segment distance
1 passed |
100.0% |
IN8 771 OUT8 027 | |
| ✓ | PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 771 OUT8 027 |
Algorithms - Prime factorization
1 passed |
100.0% |
IN8 742 OUT1 018 | |
| ✓ | PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 742 OUT1 018 |
Algorithms - Product of array except self
1 passed |
100.0% |
IN8 748 OUT1 333 | |
| ✓ | ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 748 OUT1 333 |
Algorithms - Quadratic Bezier point
1 passed |
100.0% |
IN8 808 OUT6 674 | |
| ✓ | BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 808 OUT6 674 |
Algorithms - Range sum via prefix sums
1 passed |
100.0% |
IN8 730 OUT683 | |
| ✓ | PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 730 OUT683 |
Algorithms - Reverse words & Hamming distance
1 passed |
100.0% |
IN8 734 OUT649 | |
| ✓ | ReverseWordsCodeGenCode-gen: model implements reverse-words + Hamming; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 734 OUT649 |
Algorithms - Rod cutting
1 passed |
100.0% |
IN8 764 OUT3 155 | |
| ✓ | RodCuttingCodeGenCode-gen: model implements rod cutting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 764 OUT3 155 |
Algorithms - Roman numeral to integer
1 passed |
100.0% |
IN8 753 OUT1 444 | |
| ✓ | RomanToIntCodeGenCode-gen: model implements Roman-to-int; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT1 444 |
Algorithms - Rotate array right by k
1 passed |
100.0% |
IN8 757 OUT930 | |
| ✓ | RotateArrayCodeGenCode-gen: model implements right rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 757 OUT930 |
Algorithms - Rotate grid 90 CW (discriminator)
1 passed |
100.0% |
IN8 760 OUT1 636 | |
| ✓ | RotateMatrixCodeGenCode-gen: model implements 90-degree CW rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 760 OUT1 636 |
Algorithms - RPN evaluator
1 passed |
100.0% |
IN8 762 OUT1 749 | |
| ✓ | RpnCodeGenCode-gen: model implements an RPN evaluator; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 762 OUT1 749 |
Algorithms - Run-length compression (if shorter)
1 passed |
100.0% |
IN8 734 OUT701 | |
| ✓ | StringCompressCodeGenCode-gen: model implements string compression; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 734 OUT701 |
Algorithms - Run-length encoding (encode/decode)
1 passed |
100.0% |
IN8 780 OUT2 742 | |
| ✓ | RunLengthCodeGenCode-gen: model implements RLE encode+decode; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 780 OUT2 742 |
Algorithms - Semantic versioning (compare + bump)
1 passed |
100.0% |
IN8 810 OUT1 384 | |
| ✓ | SemVerCodeGenCode-gen: model implements semver compare + bump; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:InstructionFollowinglevel:Easysource:Ai4Users |
Passed | IN8 810 OUT1 384 |
Algorithms - Single-pile Nim (optimal play)
1 passed |
100.0% |
IN8 742 OUT943 | |
| ✓ | NimCodeGenCode-gen: model implements optimal Nim; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 742 OUT943 |
Algorithms - Sliding window maximum
1 passed |
100.0% |
IN8 734 OUT2 356 | |
| ✓ | SlidingWindowMaxCodeGenCode-gen: model implements sliding window maximum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 734 OUT2 356 |
Algorithms - Spiral matrix traversal
1 passed |
100.0% |
IN8 743 OUT4 627 | |
| ✓ | SpiralMatrixCodeGenCode-gen: model implements spiral traversal; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 743 OUT4 627 |
Algorithms - Square matrix multiply
1 passed |
100.0% |
IN8 735 OUT2 403 | |
| ✓ | MatMulCodeGenCode-gen: model implements square matrix multiply; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 735 OUT2 403 |
Algorithms - Strongly connected components
1 passed |
100.0% |
IN8 746 OUT4 556 | |
| ✓ | SccCountCodeGenCode-gen: model implements SCC count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 746 OUT4 556 |
Algorithms - Subset sum reachability
1 passed |
100.0% |
IN8 719 OUT30 320 | |
| ✓ | SubsetSumCodeGenCode-gen: model implements subset-sum reachability; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 719 OUT30 320 |
Algorithms - Substring search (KMP)
1 passed |
100.0% |
IN8 741 OUT784 | |
| ✓ | KmpCodeGenCode-gen: model implements KMP search; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 741 OUT784 |
Algorithms - Sum of digits & digital root
1 passed |
100.0% |
IN8 720 OUT444 | |
| ✓ | DigitalRootCodeGenCode-gen: model implements sum-of-digits + digital root; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 720 OUT444 |
Algorithms - Temperature conversion
1 passed |
100.0% |
IN8 731 OUT240 | |
| ✓ | TemperatureCodeGenCode-gen: model implements temperature conversion; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 731 OUT240 |
Algorithms - Topological sort (lexicographically smallest)
1 passed |
100.0% |
IN8 772 OUT4 081 | |
| ✓ | TopoCodeGenCode-gen: model implements lexicographically-smallest topological sort.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 772 OUT4 081 |
Algorithms - Trailing zeros of factorial
1 passed |
100.0% |
IN8 732 OUT277 | |
| ✓ | TrailingZerosCodeGenCode-gen: model implements trailing zeros; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 732 OUT277 |
Algorithms - Trapping rain water
1 passed |
100.0% |
IN8 731 OUT1 638 | |
| ✓ | TrappingRainCodeGenCode-gen: model implements trapping rain water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 731 OUT1 638 |
Algorithms - Two-sum indices
1 passed |
100.0% |
IN8 738 OUT5 252 | |
| ✓ | TwoSumCodeGenCode-gen: model implements two-sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 738 OUT5 252 |
Algorithms - Unique grid paths
1 passed |
100.0% |
IN8 713 OUT1 014 | |
| ✓ | UniquePathsCodeGenCode-gen: model implements unique paths; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 713 OUT1 014 |
Algorithms - Valid Sudoku unit
1 passed |
100.0% |
IN8 718 OUT512 | |
| ✓ | SudokuUnitCodeGenCode-gen: model implements Sudoku unit validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 718 OUT512 |
Algorithms - Wildcard string matching
1 passed |
100.0% |
IN8 736 OUT3 161 | |
| ✓ | WildcardMatchCodeGenCode-gen: model implements wildcard match; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 736 OUT3 161 |
Algorithms - Z-function
1 passed |
100.0% |
IN8 737 OUT1 954 | |
| ✓ | ZFunctionCodeGenCode-gen: model implements the Z-function; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 737 OUT1 954 |
Database - Conditional aggregation pivot
1 passed |
100.0% |
IN945 OUT98 | |
| ✓ | PivotCountsCodeGenCode-gen: model writes the pivot query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN945 OUT98 |
Database - Count distinct opponents
1 passed |
100.0% |
IN925 OUT78 | |
| ✓ | DistinctCountCodeGenCode-gen: model writes the COUNT(DISTINCT) query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Easysource:Ai4Users |
Passed | IN925 OUT78 |
Database - Date-range filter (BETWEEN)
1 passed |
100.0% |
IN955 OUT110 | |
| ✓ | DateRangeCodeGenCode-gen: model writes the date-range query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN955 OUT110 |
Database - EXISTS subquery (teams with matches)
1 passed |
100.0% |
IN932 OUT167 | |
| ✓ | ExistsSubqueryCodeGenCode-gen: model writes the EXISTS query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN932 OUT167 |
Database - JOIN player totals (two tables)
1 passed |
100.0% |
IN954 OUT241 | |
| ✓ | JoinScoresCodeGenCode-gen: model writes a JOIN + aggregate query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN954 OUT241 |
Database - LEFT JOIN anti-join (no items)
1 passed |
100.0% |
IN928 OUT146 | |
| ✓ | LeftJoinNullsCodeGenCode-gen: model writes the anti-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN928 OUT146 |
Database - Player stats SQL (multi-aggregate, QueryJson)
1 passed |
100.0% |
IN958 OUT95 | |
| ✓ | PlayerStatsCodeGenCode-gen: model writes a multi-aggregate stats query; graded by comparing its QueryJson output to the oracle query's (arbitrary shape, no struct).1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN958 OUT95 |
Database - Players above average score (subquery)
1 passed |
100.0% |
IN923 OUT86 | |
| ✓ | AboveAverageCodeGenCode-gen: model writes an above-average query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN923 OUT86 |
Database - Players who never played (anti-join)
1 passed |
100.0% |
IN920 OUT112 | |
| ✓ | NeverPlayedCodeGenCode-gen: model writes an anti-join query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN920 OUT112 |
Database - RANK window function
1 passed |
100.0% |
IN950 OUT118 | |
| ✓ | WindowRankCodeGenCode-gen: model writes the RANK() query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN950 OUT118 |
Database - Running total (window function)
1 passed |
100.0% |
IN947 OUT136 | |
| ✓ | RunningTotalCodeGenCode-gen: model writes the window running-total query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN947 OUT136 |
Database - Scoreboard SQL (players with >1 win, HAVING)
1 passed |
100.0% |
IN954 OUT115 | |
| ✓ | WinFilterCodeGenCode-gen: model writes a HAVING query; runs under try/catch with self-repair; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN954 OUT115 |
Database - Scoreboard SQL (top scorers query)
1 passed |
100.0% |
IN955 OUT84 | |
| ✓ | ScoreboardTopScorersCodeGenCode-gen: model writes a scoreboard SQL query; runs under try/catch with a self-repair loop on SQL errors; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN955 OUT84 |
Database - Self-join same-team pairs
1 passed |
100.0% |
IN938 OUT222 | |
| ✓ | SelfJoinCodeGenCode-gen: model writes the self-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN938 OUT222 |
Database - Top scorer per region
1 passed |
100.0% |
IN940 OUT199 | |
| ✓ | TopPerGroupCodeGenCode-gen: model writes the per-group top query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN940 OUT199 |
Database - UPDATE restock (SQL write)
1 passed |
100.0% |
IN932 OUT60 | |
| ✓ | UpdateInventoryCodeGenCode-gen: model writes an UPDATE; applied under try/catch, resulting state compared (via QueryJson) to the oracle update's state.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN932 OUT60 |
Database - Upsert with max merge
1 passed |
100.0% |
IN948 OUT292 | |
| ✓ | UpsertCodeGenCode-gen: model writes the upsert; executed on a fresh DB and the table compared.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN948 OUT292 |
SDLC - Acceptance criteria check
1 passed |
100.0% |
IN8 761 OUT2 999 | |
| ✓ | SdlcAcceptanceCodeGenCode-gen: model implements acceptance-criteria check; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT2 999 |
SDLC - Breaking API change detection
1 passed |
100.0% |
IN8 758 OUT2 380 | |
| ✓ | SdlcApiBreakingCodeGenCode-gen: model implements API-breaking detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 758 OUT2 380 |
SDLC - Condense without losing facts
1 passed |
100.0% |
IN1 046 OUT1 204 | |
| ✓ | NoInfoLossCodeGenRedaction test: model condenses keeping facts; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 046 OUT1 204 |
SDLC - Config merge with override
1 passed |
100.0% |
IN8 752 OUT1 582 | |
| ✓ | ConfigMergeCodeGenCode-gen: model implements config merge; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT1 582 |
SDLC - Extract fields into fixed template
1 passed |
100.0% |
IN1 002 OUT65 | |
| ✓ | FieldExtractCodeGenRedaction test: model extracts fields into the template; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 002 OUT65 |
SDLC - Locate conflicting requirement pair
1 passed |
100.0% |
IN8 814 OUT14 667 | |
| ✓ | SdlcConstraintConflictCodeGenCode-gen: model locates the conflicting requirement pair; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN8 814 OUT14 667 |
SDLC - Locate redundant requirement
1 passed |
100.0% |
IN18 842 OUT22 601 | |
| ✓ | SdlcRedundantRequirementCodeGenCode-gen: model locates a redundant requirement; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN18 842 OUT22 601 |
SDLC - Mask PII to placeholder
1 passed |
100.0% |
IN970 OUT184 | |
| ✓ | MaskPiiCodeGenRedaction test: model masks PII to a placeholder; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN970 OUT184 |
SDLC - Modules on a dependency cycle
1 passed |
100.0% |
IN8 764 OUT13 087 | |
| ✓ | DependencyCycleModulesCodeGenCode-gen: model implements dependency-cycle detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN8 764 OUT13 087 |
SDLC - Remove duplicate requirements
1 passed |
100.0% |
IN943 OUT114 | |
| ✓ | DeduplicateCodeGenRedaction test: model removes duplicate requirements; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN943 OUT114 |
SDLC - Renumber requirements sequentially
1 passed |
100.0% |
IN1 010 OUT238 | |
| ✓ | RenumberCodeGenRedaction test: model renumbers sequentially; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 010 OUT238 |
SDLC - Reorganize notes into section template
1 passed |
100.0% |
IN1 124 OUT243 | |
| ✓ | RewriteSectionsCodeGenRedaction test: model reorganizes the notes; deterministic checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 124 OUT243 |
SDLC - Required semver bump from changelog
1 passed |
100.0% |
IN8 757 OUT825 | |
| ✓ | SemverBumpCodeGenCode-gen: model implements the semver-bump rule; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Easysource:Ai4Users |
Passed | IN8 757 OUT825 |
SDLC - Requirement-to-test traceability
1 passed |
100.0% |
IN8 779 OUT1 618 | |
| ✓ | SdlcTraceabilityCodeGenCode-gen: model implements traceability gap; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 779 OUT1 618 |
SDLC - Requirements completeness (spec coverage gap)
1 passed |
100.0% |
IN8 774 OUT6 243 | |
| ✓ | SdlcSpecCoverageCodeGenCode-gen: model implements spec-gap detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 774 OUT6 243 |
SDLC - Requirements consistency (contradiction detection)
1 passed |
100.0% |
IN8 759 OUT5 467 | |
| ✓ | ConsistencyCodeGenCode-gen: model implements requirements-consistency checking; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 759 OUT5 467 |
SDLC - Reshape prose into JSON template
1 passed |
100.0% |
IN1 012 OUT109 | |
| ✓ | JsonShapeCodeGenRedaction test: model reshapes prose into JSON; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 012 OUT109 |
SDLC - Reshape rows into Markdown table
1 passed |
100.0% |
IN966 OUT106 | |
| ✓ | TableFormatCodeGenRedaction test: model builds a Markdown table; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN966 OUT106 |
SDLC - Rewrite in imperative requirement style
1 passed |
100.0% |
IN989 OUT271 | |
| ✓ | StyleRewriteCodeGenRedaction test: model rewrites notes in imperative style; checker + self-review loop.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN989 OUT271 |
SDLC - Roll up estimates over task tree
1 passed |
100.0% |
IN8 770 OUT2 002 | |
| ✓ | EstimateRollupCodeGenCode-gen: model implements estimate roll-up over a task tree; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 770 OUT2 002 |
SDLC - State-machine reachability
1 passed |
100.0% |
IN8 756 OUT1 608 | |
| ✓ | SdlcStateReachableCodeGenCode-gen: model implements state reachability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT1 608 |
SDLC - Ticket priority sort
1 passed |
100.0% |
IN8 749 OUT1 545 | |
| ✓ | SdlcPrioritySortCodeGenCode-gen: model implements priority sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 749 OUT1 545 |
SDLC - Unify glossary term consistency
1 passed |
100.0% |
IN970 OUT214 | |
| ✓ | GlossaryCodeGenRedaction test: model unifies synonyms to the canonical term; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN970 OUT214 |