Benchmark summary
A custom Ai4Users benchmark suite for SC#, covering code generation, reasoning, and stateful agentic workflows where models use tools, observe results, adapt their actions, recover from failures, and verify the final state.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Agentic - Blue-Green Deploy (health-gated cutover)
1 passed |
100.0% |
IN17 750 OUT695 | |
| ✓ | AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query
5 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN17 750 OUT695 |
Agentic - Bugfix (root-cause vs regression)
1 passed |
100.0% |
IN20 017 OUT336 | |
| ✓ | AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query
1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN20 017 OUT336 |
Agentic - Cache Invalidation (exact set)
1 passed |
100.0% |
IN14 813 OUT954 | |
| ✓ | AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query
8 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN14 813 OUT954 |
Agentic - Canary (metric-gated release)
1 passed |
100.0% |
IN39 890 OUT1 787 | |
| ✓ | AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query
11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN39 890 OUT1 787 |
Agentic - Canary variants (5 metric cases)
5 passed |
100.0% |
IN147 011 OUT7 860 | |
| ✓ | CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN29 924 OUT1 316 |
| ✓ | CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN30 243 OUT1 237 |
| ✓ | CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN27 656 OUT1 923 |
| ✓ | CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query
23 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN28 812 OUT1 967 |
| ✓ | CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN30 376 OUT1 417 |
Agentic - Code Review (find injected defect)
1 passed |
100.0% |
IN11 829 OUT951 | |
| ✓ | AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query
5 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 829 OUT951 |
Agentic - Config Drift (remediate exact)
1 passed |
100.0% |
IN12 099 OUT731 | |
| ✓ | AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query
6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN12 099 OUT731 |
Agentic - Constraints (multi-section satisfaction)
1 passed |
100.0% |
IN11 149 OUT997 | |
| ✓ | AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 149 OUT997 |
Agentic - Constraints variants (5 cases)
5 passed |
100.0% |
IN47 795 OUT3 853 | |
| ✓ | ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 558 OUT1 127 |
| ✓ | ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 687 OUT628 |
| ✓ | ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 490 OUT568 |
| ✓ | ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 555 OUT754 |
| ✓ | ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 505 OUT776 |
Agentic - Data Reconciliation (per-field precedence)
1 passed |
100.0% |
IN13 610 OUT977 | |
| ✓ | AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN13 610 OUT977 |
Agentic - Diagnosis (root cause under budget)
1 passed |
100.0% |
IN12 406 OUT1 072 | |
| ✓ | AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN12 406 OUT1 072 |
Agentic - Diagnosis variants (5 cases)
5 passed |
100.0% |
IN67 747 OUT5 042 | |
| ✓ | DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN12 021 OUT1 113 |
| ✓ | DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN12 155 OUT951 |
| ✓ | DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 064 OUT949 |
| ✓ | DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN12 464 OUT849 |
| ✓ | DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN16 043 OUT1 180 |
Agentic - ETL Repair (root-cause in stage 2)
1 passed |
100.0% |
IN32 554 OUT2 592 | |
| ✓ | AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN32 554 OUT2 592 |
Agentic - Feature Flag (staged rollout + kill-switch)
1 passed |
100.0% |
IN26 672 OUT1 373 | |
| ✓ | AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN26 672 OUT1 373 |
Agentic - Game Backend: 1D bouncing ball
1 passed |
100.0% |
IN8 781 OUT1 959 | |
| ✓ | GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 781 OUT1 959 |
Agentic - Game Backend: 1D swept anti-tunneling
1 passed |
100.0% |
IN8 792 OUT1 069 | |
| ✓ | GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 792 OUT1 069 |
Agentic - Game Backend: 2048 left-move (slide + merge)
1 passed |
100.0% |
IN8 801 OUT2 351 | |
| ✓ | GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 801 OUT2 351 |
Agentic - Game Backend: 8-direction from vector
1 passed |
100.0% |
IN8 808 OUT667 | |
| ✓ | GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 808 OUT667 |
Agentic - Game Backend: AABB collision resolution (MTV)
1 passed |
100.0% |
IN8 897 OUT900 | |
| ✓ | GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 897 OUT900 |
Agentic - Game Backend: AABB containment
1 passed |
100.0% |
IN8 737 OUT372 | |
| ✓ | GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 737 OUT372 |
Agentic - Game Backend: Acceleration to max speed
1 passed |
100.0% |
IN8 739 OUT517 | |
| ✓ | GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 739 OUT517 |
Agentic - Game Backend: Achievement thresholds
1 passed |
100.0% |
IN8 731 OUT1 248 | |
| ✓ | GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 731 OUT1 248 |
Agentic - Game Backend: Active status effects
1 passed |
100.0% |
IN8 753 OUT992 | |
| ✓ | GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT992 |
Agentic - Game Backend: Animation frame at time
1 passed |
100.0% |
IN8 771 OUT503 | |
| ✓ | GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 771 OUT503 |
Agentic - Game Backend: Army upkeep income
1 passed |
100.0% |
IN8 719 OUT582 | |
| ✓ | GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 719 OUT582 |
Agentic - Game Backend: Axis-aligned path follow
1 passed |
100.0% |
IN18 008 OUT1 918 | |
| ✓ | GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 008 OUT1 918 |
Agentic - Game Backend: Battleship shot resolution
1 passed |
100.0% |
IN8 821 OUT1 550 | |
| ✓ | GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 821 OUT1 550 |
Agentic - Game Backend: Bounce restitution
1 passed |
100.0% |
IN8 758 OUT289 | |
| ✓ | GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 758 OUT289 |
Agentic - Game Backend: Bresenham line rasterization
1 passed |
100.0% |
IN8 848 OUT1 043 | |
| ✓ | GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 848 OUT1 043 |
Agentic - Game Backend: Buff/debuff stacking
1 passed |
100.0% |
IN8 784 OUT1 109 | |
| ✓ | GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 784 OUT1 109 |
Agentic - Game Backend: Build queue completions
1 passed |
100.0% |
IN8 752 OUT843 | |
| ✓ | GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 752 OUT843 |
Agentic - Game Backend: Camera-relative movement
1 passed |
100.0% |
IN8 839 OUT691 | |
| ✓ | GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 839 OUT691 |
Agentic - Game Backend: Card hand sort
1 passed |
100.0% |
IN8 760 OUT4 105 | |
| ✓ | GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 760 OUT4 105 |
Agentic - Game Backend: Card-game rules (blackjack/deck/compare)
1 passed |
100.0% |
IN8 935 OUT2 590 | |
| ✓ | GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 935 OUT2 590 |
Agentic - Game Backend: Chart scaling & histogram bins
1 passed |
100.0% |
IN8 824 OUT10 659 | |
| ✓ | GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 824 OUT10 659 |
Agentic - Game Backend: Checkers captures
1 passed |
100.0% |
IN8 794 OUT2 368 | |
| ✓ | GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 794 OUT2 368 |
Agentic - Game Backend: Collision primitives (circle/aabb/segment)
1 passed |
100.0% |
IN8 997 OUT1 783 | |
| ✓ | GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 997 OUT1 783 |
Agentic - Game Backend: Combat turn order (initiative)
1 passed |
100.0% |
IN8 733 OUT1 402 | |
| ✓ | GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT1 402 |
Agentic - Game Backend: Combo scoring (rising multiplier)
1 failed |
0.0% |
IN8 756 OUT620 | |
| × | GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users ComboScore threw: Cannot parse '' as int. |
Failed | IN8 756 OUT620 |
Agentic - Game Backend: Control-point capture
1 passed |
100.0% |
IN8 769 OUT2 092 | |
| ✓ | GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT2 092 |
Agentic - Game Backend: Controls & Inventory (spec-driven function battery)
1 passed |
100.0% |
IN8 910 OUT856 | |
| ✓ | GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 910 OUT856 |
Agentic - Game Backend: Cooldowns & resource regen
1 passed |
100.0% |
IN8 827 OUT771 | |
| ✓ | GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 827 OUT771 |
Agentic - Game Backend: Coyote-time jump
1 passed |
100.0% |
IN8 748 OUT203 | |
| ✓ | GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 748 OUT203 |
Agentic - Game Backend: Crafting capacity
1 passed |
100.0% |
IN8 772 OUT1 572 | |
| ✓ | GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT1 572 |
Agentic - Game Backend: Cribbage fifteens
1 passed |
100.0% |
IN8 749 OUT1 176 | |
| ✓ | GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 749 OUT1 176 |
Agentic - Game Backend: Critical-hit damage
1 passed |
100.0% |
IN8 739 OUT154 | |
| ✓ | GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 739 OUT154 |
Agentic - Game Backend: Damage over time
1 passed |
100.0% |
IN8 765 OUT889 | |
| ✓ | GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 765 OUT889 |
Agentic - Game Backend: Day/night cycle phase
1 passed |
100.0% |
IN8 751 OUT219 | |
| ✓ | GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 751 OUT219 |
Agentic - Game Backend: Dice notation max roll
1 passed |
100.0% |
IN8 766 OUT1 297 | |
| ✓ | GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 766 OUT1 297 |
Agentic - Game Backend: Enemy AI state machine
1 passed |
100.0% |
IN8 852 OUT1 007 | |
| ✓ | GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 852 OUT1 007 |
Agentic - Game Backend: Equipment slot validation
1 passed |
100.0% |
IN8 766 OUT490 | |
| ✓ | GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 766 OUT490 |
Agentic - Game Backend: Flood-fill region size
1 passed |
100.0% |
IN8 754 OUT1 522 | |
| ✓ | GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT1 522 |
Agentic - Game Backend: Fog of war visibility
1 passed |
100.0% |
IN8 794 OUT860 | |
| ✓ | GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 794 OUT860 |
Agentic - Game Backend: Fog reveal (Chebyshev radius)
1 passed |
100.0% |
IN8 767 OUT22 575 | |
| ✓ | GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 767 OUT22 575 |
Agentic - Game Backend: Friction stop frame
1 passed |
100.0% |
IN8 747 OUT399 | |
| ✓ | GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 747 OUT399 |
Agentic - Game Backend: Game of Life (code-gen vs oracle)
1 passed |
100.0% |
IN8 806 OUT1 320 | |
| ✓ | GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 806 OUT1 320 |
Agentic - Game Backend: Greedy coin change
1 passed |
100.0% |
IN8 748 OUT627 | |
| ✓ | GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 748 OUT627 |
Agentic - Game Backend: Grid inventory & stacking (Diablo-style)
1 passed |
100.0% |
IN18 125 OUT2 630 | |
| ✓ | GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 125 OUT2 630 |
Agentic - Game Backend: Grid pathfinding (shortest path, BFS)
1 passed |
100.0% |
IN8 836 OUT3 999 | |
| ✓ | GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.1 AI query
0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 836 OUT3 999 |
Agentic - Game Backend: Grid raycast
1 passed |
100.0% |
IN8 804 OUT1 793 | |
| ✓ | GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 804 OUT1 793 |
Agentic - Game Backend: Health bar segments
1 passed |
100.0% |
IN8 805 OUT1 566 | |
| ✓ | GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 805 OUT1 566 |
Agentic - Game Backend: High-score table insert
1 passed |
100.0% |
IN17 812 OUT3 974 | |
| ✓ | GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 812 OUT3 974 |
Agentic - Game Backend: Hit chance
1 passed |
100.0% |
IN8 743 OUT319 | |
| ✓ | GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 743 OUT319 |
Agentic - Game Backend: Inventory encumbrance
1 passed |
100.0% |
IN8 740 OUT773 | |
| ✓ | GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 740 OUT773 |
Agentic - Game Backend: Inventory first-fit placement
1 passed |
100.0% |
IN8 806 OUT1 577 | |
| ✓ | GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 806 OUT1 577 |
Agentic - Game Backend: Inventory stack split
1 passed |
100.0% |
IN8 755 OUT343 | |
| ✓ | GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 755 OUT343 |
Agentic - Game Backend: Isometric projection
1 passed |
100.0% |
IN8 773 OUT350 | |
| ✓ | GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 773 OUT350 |
Agentic - Game Backend: Jump buffer
1 passed |
100.0% |
IN8 761 OUT914 | |
| ✓ | GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT914 |
Agentic - Game Backend: Knockback with arena clamp
1 passed |
100.0% |
IN8 800 OUT669 | |
| ✓ | GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 800 OUT669 |
Agentic - Game Backend: Leaderboard rank
1 passed |
100.0% |
IN8 733 OUT622 | |
| ✓ | GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 733 OUT622 |
Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over)
1 passed |
100.0% |
IN8 884 OUT755 | |
| ✓ | GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 884 OUT755 |
Agentic - Game Backend: Lifesteal healing
1 passed |
100.0% |
IN8 744 OUT599 | |
| ✓ | GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 744 OUT599 |
Agentic - Game Backend: Lights Out toggle
1 passed |
100.0% |
IN8 760 OUT2 663 | |
| ✓ | GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 760 OUT2 663 |
Agentic - Game Backend: Match-3 cascade
1 passed |
100.0% |
IN8 818 OUT2 378 | |
| ✓ | GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 818 OUT2 378 |
Agentic - Game Backend: Match-3 detection
1 passed |
100.0% |
IN8 782 OUT1 319 | |
| ✓ | GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 782 OUT1 319 |
Agentic - Game Backend: Match-3 swap validity
1 passed |
100.0% |
IN8 791 OUT1 847 | |
| ✓ | GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 791 OUT1 847 |
Agentic - Game Backend: Memory match pairs
1 passed |
100.0% |
IN8 745 OUT755 | |
| ✓ | GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 745 OUT755 |
Agentic - Game Backend: Mercy invincibility (i-frames)
1 passed |
100.0% |
IN8 800 OUT1 165 | |
| ✓ | GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 800 OUT1 165 |
Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal)
1 passed |
100.0% |
IN8 882 OUT2 168 | |
| ✓ | GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 882 OUT2 168 |
Agentic - Game Backend: Minimap downsample
1 passed |
100.0% |
IN8 776 OUT829 | |
| ✓ | GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 776 OUT829 |
Agentic - Game Backend: Minimap marker position
1 passed |
100.0% |
IN8 779 OUT661 | |
| ✓ | GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 779 OUT661 |
Agentic - Game Backend: N-in-a-row win detection (Connect-4)
1 passed |
100.0% |
IN8 794 OUT2 012 | |
| ✓ | GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 794 OUT2 012 |
Agentic - Game Backend: Nonogram row validation
1 passed |
100.0% |
IN8 756 OUT2 376 | |
| ✓ | GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT2 376 |
Agentic - Game Backend: OBB collision (rotating rectangles, SAT)
1 passed |
100.0% |
IN8 903 OUT19 047 | |
| ✓ | GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 903 OUT19 047 |
Agentic - Game Backend: Particle lifetime & expiry
1 passed |
100.0% |
IN8 838 OUT1 292 | |
| ✓ | GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 838 OUT1 292 |
Agentic - Game Backend: Party XP share
1 passed |
100.0% |
IN8 756 OUT593 | |
| ✓ | GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 756 OUT593 |
Agentic - Game Backend: Patrol waypoints (ping-pong)
1 passed |
100.0% |
IN8 779 OUT1 554 | |
| ✓ | GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 779 OUT1 554 |
Agentic - Game Backend: Platformer jump arc
1 passed |
100.0% |
IN8 770 OUT2 618 | |
| ✓ | GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 770 OUT2 618 |
Agentic - Game Backend: Point on circle (trig)
1 passed |
100.0% |
IN8 803 OUT793 | |
| ✓ | GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 803 OUT793 |
Agentic - Game Backend: Poker hand comparison
1 passed |
100.0% |
IN19 324 OUT6 607 | |
| ✓ | GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users |
Passed | IN19 324 OUT6 607 |
Agentic - Game Backend: Poker hand ranking
1 passed |
100.0% |
IN8 822 OUT3 182 | |
| ✓ | GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 822 OUT3 182 |
Agentic - Game Backend: Pop-cap training
1 passed |
100.0% |
IN8 757 OUT538 | |
| ✓ | GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 757 OUT538 |
Agentic - Game Backend: Progress bar fill
1 passed |
100.0% |
IN8 739 OUT317 | |
| ✓ | GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 739 OUT317 |
Agentic - Game Backend: Projectile range (physics/trig)
1 passed |
100.0% |
IN8 761 OUT716 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT716 |
Agentic - Game Backend: Projectile with ground bounce
1 passed |
100.0% |
IN17 776 OUT469 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 776 OUT469 |
Agentic - Game Backend: Resource gathering
1 passed |
100.0% |
IN8 756 OUT200 | |
| ✓ | GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 756 OUT200 |
Agentic - Game Backend: Reversi flip count
1 passed |
100.0% |
IN8 827 OUT1 254 | |
| ✓ | GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 827 OUT1 254 |
Agentic - Game Backend: Rotate 2D vector (trig)
1 passed |
100.0% |
IN8 772 OUT530 | |
| ✓ | GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT530 |
Agentic - Game Backend: Round-robin deal
1 passed |
100.0% |
IN8 761 OUT588 | |
| ✓ | GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 761 OUT588 |
Agentic - Game Backend: RTS resource economy
1 passed |
100.0% |
IN8 752 OUT230 | |
| ✓ | GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 752 OUT230 |
Agentic - Game Backend: RTS unit counters & damage
1 passed |
100.0% |
IN8 804 OUT684 | |
| ✓ | GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 804 OUT684 |
Agentic - Game Backend: Safest neighbour (danger map)
1 passed |
100.0% |
IN18 088 OUT1 823 | |
| ✓ | GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 088 OUT1 823 |
Agentic - Game Backend: Scroll viewport
1 passed |
100.0% |
IN8 774 OUT612 | |
| ✓ | GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 774 OUT612 |
Agentic - Game Backend: Set card validity
1 passed |
100.0% |
IN8 753 OUT709 | |
| ✓ | GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT709 |
Agentic - Game Backend: Shop affordability
1 passed |
100.0% |
IN8 733 OUT418 | |
| ✓ | GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 733 OUT418 |
Agentic - Game Backend: Sliding puzzle move
1 passed |
100.0% |
IN8 764 OUT2 260 | |
| ✓ | GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 764 OUT2 260 |
Agentic - Game Backend: Snake head movement
1 passed |
100.0% |
IN8 821 OUT924 | |
| ✓ | GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 821 OUT924 |
Agentic - Game Backend: Snake tick
1 passed |
100.0% |
IN8 891 OUT2 054 | |
| ✓ | GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 891 OUT2 054 |
Agentic - Game Backend: Snap to grid
1 passed |
100.0% |
IN8 749 OUT322 | |
| ✓ | GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 749 OUT322 |
Agentic - Game Backend: Sokoban move/push legality
1 passed |
100.0% |
IN8 852 OUT496 | |
| ✓ | GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 852 OUT496 |
Agentic - Game Backend: Solitaire tableau move
1 passed |
100.0% |
IN8 791 OUT1 368 | |
| ✓ | GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 791 OUT1 368 |
Agentic - Game Backend: Spawn scheduler count
1 passed |
100.0% |
IN8 748 OUT495 | |
| ✓ | GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 748 OUT495 |
Agentic - Game Backend: Spawn-wave alive count
1 passed |
100.0% |
IN17 733 OUT800 | |
| ✓ | GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 733 OUT800 |
Agentic - Game Backend: Stat level scaling
1 passed |
100.0% |
IN8 752 OUT966 | |
| ✓ | GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT966 |
Agentic - Game Backend: Steering seek (greedy pursuit)
1 passed |
100.0% |
IN8 778 OUT1 243 | |
| ✓ | GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 778 OUT1 243 |
Agentic - Game Backend: Tactical movement range
1 passed |
100.0% |
IN8 790 OUT2 087 | |
| ✓ | GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 790 OUT2 087 |
Agentic - Game Backend: Tech tree eligibility
1 passed |
100.0% |
IN8 772 OUT1 419 | |
| ✓ | GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT1 419 |
Agentic - Game Backend: Tetris line clearing
1 passed |
100.0% |
IN8 763 OUT3 015 | |
| ✓ | GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 763 OUT3 015 |
Agentic - Game Backend: Threat table top target
1 passed |
100.0% |
IN8 757 OUT941 | |
| ✓ | GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 757 OUT941 |
Agentic - Game Backend: Tic-tac-toe result
1 passed |
100.0% |
IN8 748 OUT856 | |
| ✓ | GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 748 OUT856 |
Agentic - Game Backend: Toroidal wrap (Pac-Man)
1 passed |
100.0% |
IN8 784 OUT271 | |
| ✓ | GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 784 OUT271 |
Agentic - Game Backend: Tower targeting (nearest in range)
1 passed |
100.0% |
IN8 772 OUT1 029 | |
| ✓ | GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT1 029 |
Agentic - Game Backend: Transaction ledger
1 passed |
100.0% |
IN8 775 OUT1 173 | |
| ✓ | GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 775 OUT1 173 |
Agentic - Game Backend: Trick-taking winner
1 failed |
0.0% |
IN8 803 OUT2 148 | |
| × | GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: Winner([9H 2S AH] trump S lead H) got 2 expected 1
Values:
r.ResultJson = "2" |
Failed | IN8 803 OUT2 148 |
Agentic - Game Backend: Turn-based duel resolution
1 passed |
100.0% |
IN8 835 OUT1 693 | |
| ✓ | GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 835 OUT1 693 |
Agentic - Game Backend: Typed armor mitigation
1 passed |
100.0% |
IN8 759 OUT1 433 | |
| ✓ | GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 759 OUT1 433 |
Agentic - Game Backend: UI menu (wrap/clamp/paginate)
1 passed |
100.0% |
IN8 851 OUT3 017 | |
| ✓ | GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 851 OUT3 017 |
Agentic - Game Backend: Uno playable cards
1 passed |
100.0% |
IN8 787 OUT1 046 | |
| ✓ | GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 787 OUT1 046 |
Agentic - Game Backend: Velocity reflection
1 passed |
100.0% |
IN8 731 OUT311 | |
| ✓ | GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 731 OUT311 |
Agentic - Game Backend: Wall-slide fall
1 passed |
100.0% |
IN8 739 OUT351 | |
| ✓ | GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 739 OUT351 |
Agentic - Game Backend: War card compare
1 passed |
100.0% |
IN8 759 OUT936 | |
| ✓ | GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 759 OUT936 |
Agentic - Game Backend: Weighted loot table
1 passed |
100.0% |
IN8 762 OUT1 475 | |
| ✓ | GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 762 OUT1 475 |
Agentic - Game Backend: Weighted terrain pathfinding
1 passed |
100.0% |
IN8 812 OUT9 354 | |
| ✓ | GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 812 OUT9 354 |
Agentic - Game Backend: Word-wrap line count
1 passed |
100.0% |
IN8 744 OUT2 572 | |
| ✓ | GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT2 572 |
Agentic - Game Backend: XP to character level
1 passed |
100.0% |
IN8 752 OUT2 007 | |
| ✓ | GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT2 007 |
Agentic - Idempotency (poison-retry / no double charge)
1 passed |
100.0% |
IN15 827 OUT1 026 | |
| ✓ | AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query
9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN15 827 OUT1 026 |
Agentic - Irreversible Commit (measure twice)
1 passed |
100.0% |
IN19 575 OUT758 | |
| ✓ | AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query
9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN19 575 OUT758 |
Agentic - Least Privilege (exact grants)
1 passed |
100.0% |
IN11 828 OUT397 | |
| ✓ | AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query
6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN11 828 OUT397 |
Agentic - Ledger (simulated DB read-modify-write)
1 passed |
100.0% |
IN12 830 OUT1 397 | |
| ✓ | AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 830 OUT1 397 |
Agentic - Ledger variants (5 RMW cases)
5 passed |
100.0% |
IN70 973 OUT8 799 | |
| ✓ | LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN20 983 OUT2 033 |
| ✓ | LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 463 OUT2 238 |
| ✓ | LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 822 OUT2 264 |
| ✓ | LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 541 OUT1 008 |
| ✓ | LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query
6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 164 OUT1 256 |
Agentic - Lock Ordering (deadlock avoidance)
1 passed |
100.0% |
IN86 147 OUT2 482 | |
| ✓ | AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query
25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN86 147 OUT2 482 |
Agentic - Lock Ordering variants (5 cases)
5 passed |
100.0% |
IN339 691 OUT12 796 | |
| ✓ | LockVariantFiveResFourOps1 AI query
23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN81 995 OUT2 977 |
| ✓ | LockVariantFourResThreeOps1 AI query
18 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN58 549 OUT1 923 |
| ✓ | LockVariantHeavyReuse1 AI query
23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN76 699 OUT2 239 |
| ✓ | LockVariantReverseListed1 AI query
24 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN77 881 OUT2 705 |
| ✓ | LockVariantThreeLockOp1 AI query
20 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN44 567 OUT2 952 |
Agentic - Migration (cascading rollback + reapply)
1 passed |
100.0% |
IN24 639 OUT1 364 | |
| ✓ | AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query
11 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN24 639 OUT1 364 |
Agentic - Requirements (detect contradiction)
1 passed |
100.0% |
IN7 297 OUT690 | |
| ✓ | AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query
2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN7 297 OUT690 |
Agentic - Retry/Circuit-breaker (transient vs hard-down)
1 passed |
100.0% |
IN25 846 OUT1 145 | |
| ✓ | AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query
10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN25 846 OUT1 145 |
Agentic - Saga (compensation / distributed rollback)
1 passed |
100.0% |
IN17 869 OUT548 | |
| ✓ | AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN17 869 OUT548 |
Agentic - Saga variants (5 compensation cases)
5 passed |
100.0% |
IN89 854 OUT2 842 | |
| ✓ | SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query
4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN13 397 OUT390 |
| ✓ | SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 458 OUT446 |
| ✓ | SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 773 OUT573 |
| ✓ | SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN19 722 OUT673 |
| ✓ | SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query
9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN23 504 OUT760 |
Agentic - Schema Compat (expand-contract)
1 passed |
100.0% |
IN18 340 OUT498 | |
| ✓ | AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN18 340 OUT498 |
Agentic - Scratchpad (self-repair, Dyck, tool-decision)
7 passed |
100.0% |
IN162 885 OUT18 254 | |
| ✓ | AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query
1 tool callcategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN21 287 OUT1 858 |
| ✓ | AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query
2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN32 415 OUT2 058 |
| ✓ | ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query
0 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN10 526 OUT46 |
| ✓ | ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query
0 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN10 548 OUT291 |
| ✓ | ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query
1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN21 162 OUT714 |
| ✓ | ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query
3 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN45 543 OUT11 643 |
| ✓ | ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query
1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN21 404 OUT1 644 |
Agentic - Secret Rotation (zero-downtime ordering)
1 passed |
100.0% |
IN16 327 OUT599 | |
| ✓ | AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 327 OUT599 |
Agentic - StatefulOps (incident recovery, concurrency replan)
2 passed |
100.0% |
IN58 897 OUT2 373 | |
| ✓ | AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query
7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN22 323 OUT990 |
| ✓ | AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query
9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN36 574 OUT1 383 |
Agentic - Test Authoring (write a test that catches the bug)
1 passed |
100.0% |
IN8 929 OUT651 | |
| ✓ | AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN8 929 OUT651 |
Agentic - Topological Planning (deps + prereq)
1 passed |
100.0% |
IN24 977 OUT836 | |
| ✓ | AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN24 977 OUT836 |
Algorithms - 0/1 Knapsack (dynamic programming)
1 passed |
100.0% |
IN8 766 OUT23 344 | |
| ✓ | KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 766 OUT23 344 |
Algorithms - 1D elastic collision
1 passed |
100.0% |
IN8 806 OUT1 521 | |
| ✓ | ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 806 OUT1 521 |
Algorithms - All-pairs shortest path pair
1 passed |
100.0% |
IN8 774 OUT11 819 | |
| ✓ | FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 774 OUT11 819 |
Algorithms - Balanced brackets (stack)
1 passed |
100.0% |
IN8 737 OUT571 | |
| ✓ | ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 737 OUT571 |
Algorithms - Bellman-Ford shortest path (neg weights/cycles)
1 passed |
100.0% |
IN18 714 OUT16 261 | |
| ✓ | BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN18 714 OUT16 261 |
Algorithms - Binary Gray code
1 passed |
100.0% |
IN8 710 OUT169 | |
| ✓ | GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 710 OUT169 |
Algorithms - Bipartite graph check
1 passed |
100.0% |
IN8 756 OUT1 325 | |
| ✓ | BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT1 325 |
Algorithms - Bit tricks (popcount/pow2/lowbit)
1 passed |
100.0% |
IN8 784 OUT1 351 | |
| ✓ | BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 784 OUT1 351 |
Algorithms - Caesar cipher
1 passed |
100.0% |
IN8 733 OUT900 | |
| ✓ | CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT900 |
Algorithms - Catalan number
1 passed |
100.0% |
IN8 755 OUT1 247 | |
| ✓ | CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT1 247 |
Algorithms - Chinese remainder theorem
1 passed |
100.0% |
IN8 773 OUT6 464 | |
| ✓ | CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 773 OUT6 464 |
Algorithms - Circle-circle MTV
1 passed |
100.0% |
IN8 850 OUT951 | |
| ✓ | CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 850 OUT951 |
Algorithms - Coin change combinations
1 passed |
100.0% |
IN8 740 OUT1 301 | |
| ✓ | CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 740 OUT1 301 |
Algorithms - Collatz step count
1 passed |
100.0% |
IN8 736 OUT357 | |
| ✓ | CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 736 OUT357 |
Algorithms - Connected components (union-find)
1 passed |
100.0% |
IN8 752 OUT1 843 | |
| ✓ | UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT1 843 |
Algorithms - Container with most water
1 passed |
100.0% |
IN8 744 OUT985 | |
| ✓ | ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT985 |
Algorithms - Count anagram groups
1 passed |
100.0% |
IN8 727 OUT1 035 | |
| ✓ | AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 727 OUT1 035 |
Algorithms - Count grid color regions
1 passed |
100.0% |
IN8 747 OUT1 657 | |
| ✓ | CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 747 OUT1 657 |
Algorithms - Count islands
1 passed |
100.0% |
IN8 750 OUT1 334 | |
| ✓ | CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 750 OUT1 334 |
Algorithms - Count overlapping occurrences
1 passed |
100.0% |
IN8 733 OUT744 | |
| ✓ | RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT744 |
Algorithms - Count primes (Sieve of Eratosthenes)
1 passed |
100.0% |
IN8 712 OUT994 | |
| ✓ | CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 712 OUT994 |
Algorithms - Cumulative distribution
1 passed |
100.0% |
IN8 734 OUT967 | |
| ✓ | CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 734 OUT967 |
Algorithms - Dijkstra shortest path (weighted graph)
1 passed |
100.0% |
IN8 786 OUT11 785 | |
| ✓ | DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 786 OUT11 785 |
Algorithms - Directed cycle detection
1 passed |
100.0% |
IN8 743 OUT3 900 | |
| ✓ | DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 743 OUT3 900 |
Algorithms - Distinct subsequences
1 passed |
100.0% |
IN8 729 OUT1 187 | |
| ✓ | DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 729 OUT1 187 |
Algorithms - Divisor count
1 passed |
100.0% |
IN8 717 OUT379 | |
| ✓ | DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 717 OUT379 |
Algorithms - Downsample by block average
1 passed |
100.0% |
IN8 758 OUT931 | |
| ✓ | ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 758 OUT931 |
Algorithms - Dutch national flag sort
1 passed |
100.0% |
IN8 725 OUT2 378 | |
| ✓ | DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 725 OUT2 378 |
Algorithms - Egg drop trials
1 passed |
100.0% |
IN8 786 OUT1 431 | |
| ✓ | EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 786 OUT1 431 |
Algorithms - Equal subset-sum partition
1 passed |
100.0% |
IN8 728 OUT1 443 | |
| ✓ | PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 728 OUT1 443 |
Algorithms - Euler totient
1 passed |
100.0% |
IN8 717 OUT1 308 | |
| ✓ | EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 717 OUT1 308 |
Algorithms - Eulerian path existence
1 passed |
100.0% |
IN8 776 OUT13 023 | |
| ✓ | EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 776 OUT13 023 |
Algorithms - Extended Euclidean
1 passed |
100.0% |
IN8 829 OUT1 114 | |
| ✓ | ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 829 OUT1 114 |
Algorithms - Format duration H:MM:SS
1 passed |
100.0% |
IN8 744 OUT495 | |
| ✓ | FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 744 OUT495 |
Algorithms - GCD and LCM
1 passed |
100.0% |
IN8 721 OUT500 | |
| ✓ | GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 721 OUT500 |
Algorithms - Grid transpose
1 passed |
100.0% |
IN8 752 OUT1 080 | |
| ✓ | TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT1 080 |
Algorithms - House robber (max non-adjacent sum)
1 passed |
100.0% |
IN8 716 OUT644 | |
| ✓ | HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 716 OUT644 |
Algorithms - Integer linear interpolation
1 passed |
100.0% |
IN8 750 OUT426 | |
| ✓ | LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 750 OUT426 |
Algorithms - Integer square root
1 passed |
100.0% |
IN8 732 OUT677 | |
| ✓ | IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 732 OUT677 |
Algorithms - Integer to base-N string
1 passed |
100.0% |
IN8 738 OUT814 | |
| ✓ | BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 738 OUT814 |
Algorithms - Integer to Roman numeral
1 passed |
100.0% |
IN8 767 OUT1 631 | |
| ✓ | RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 767 OUT1 631 |
Algorithms - Josephus survivor
1 passed |
100.0% |
IN8 743 OUT1 392 | |
| ✓ | JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 743 OUT1 392 |
Algorithms - Jump game (min jumps)
1 passed |
100.0% |
IN8 751 OUT3 540 | |
| ✓ | JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 751 OUT3 540 |
Algorithms - Kth largest element
1 passed |
100.0% |
IN8 741 OUT502 | |
| ✓ | KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 741 OUT502 |
Algorithms - Least-squares line fit
1 passed |
100.0% |
IN8 783 OUT1 252 | |
| ✓ | LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 783 OUT1 252 |
Algorithms - Levenshtein edit distance (DP)
1 passed |
100.0% |
IN8 711 OUT1 550 | |
| ✓ | EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 711 OUT1 550 |
Algorithms - Log-scale bucket index
1 passed |
100.0% |
IN8 776 OUT1 382 | |
| ✓ | LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 776 OUT1 382 |
Algorithms - Logarithmic bin index
1 passed |
100.0% |
IN8 765 OUT848 | |
| ✓ | LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 765 OUT848 |
Algorithms - Longest common prefix
1 passed |
100.0% |
IN8 724 OUT1 205 | |
| ✓ | CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 724 OUT1 205 |
Algorithms - Longest common subsequence
1 passed |
100.0% |
IN8 709 OUT1 719 | |
| ✓ | LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 709 OUT1 719 |
Algorithms - Longest common substring
1 passed |
100.0% |
IN8 726 OUT7 261 | |
| ✓ | CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 726 OUT7 261 |
Algorithms - Longest increasing subsequence (DP)
1 passed |
100.0% |
IN8 743 OUT1 034 | |
| ✓ | LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 743 OUT1 034 |
Algorithms - Longest palindromic substring
1 passed |
100.0% |
IN8 721 OUT583 | |
| ✓ | LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 721 OUT583 |
Algorithms - Longest repeated substring
1 passed |
100.0% |
IN8 726 OUT1 490 | |
| ✓ | LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 726 OUT1 490 |
Algorithms - Lower bound (binary search)
1 passed |
100.0% |
IN8 754 OUT695 | |
| ✓ | LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT695 |
Algorithms - Luhn checksum validation
1 passed |
100.0% |
IN8 755 OUT678 | |
| ✓ | LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT678 |
Algorithms - Majority element
1 passed |
100.0% |
IN8 719 OUT897 | |
| ✓ | MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 719 OUT897 |
Algorithms - Matrix chain multiplication
1 failed |
0.0% |
IN8 754 OUT2 346 | |
| × | MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: MinMult([40 20 30 10 30]) got 16000 expected 26000
Values:
r.ResultJson = "16000" |
Failed | IN8 754 OUT2 346 |
Algorithms - Maximum subarray sum (Kadane)
1 passed |
100.0% |
IN8 744 OUT2 394 | |
| ✓ | MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT2 394 |
Algorithms - Merge K sorted lists
1 passed |
100.0% |
IN8 737 OUT1 766 | |
| ✓ | MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 737 OUT1 766 |
Algorithms - Merge overlapping intervals
1 passed |
100.0% |
IN8 769 OUT2 451 | |
| ✓ | MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT2 451 |
Algorithms - Min-max normalization
1 passed |
100.0% |
IN8 746 OUT842 | |
| ✓ | MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 746 OUT842 |
Algorithms - Minimum coin change (DP)
1 passed |
100.0% |
IN8 737 OUT2 417 | |
| ✓ | CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 737 OUT2 417 |
Algorithms - Minimum knight moves (BFS)
1 passed |
100.0% |
IN8 775 OUT1 522 | |
| ✓ | KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 775 OUT1 522 |
Algorithms - Minimum path sum (grid DP)
1 passed |
100.0% |
IN8 750 OUT1 070 | |
| ✓ | MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 750 OUT1 070 |
Algorithms - Minimum spanning tree (Kruskal)
1 passed |
100.0% |
IN8 750 OUT1 728 | |
| ✓ | KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 750 OUT1 728 |
Algorithms - Minimum window substring
1 passed |
100.0% |
IN8 758 OUT1 212 | |
| ✓ | MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 758 OUT1 212 |
Algorithms - Modular exponentiation (fast pow)
1 passed |
100.0% |
IN8 769 OUT911 | |
| ✓ | ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT911 |
Algorithms - Modular inverse (extended Euclid)
1 passed |
100.0% |
IN8 757 OUT2 159 | |
| ✓ | ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 757 OUT2 159 |
Algorithms - Moving average
1 passed |
100.0% |
IN8 748 OUT1 838 | |
| ✓ | MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 748 OUT1 838 |
Algorithms - Next greater element
1 passed |
100.0% |
IN8 731 OUT902 | |
| ✓ | NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 731 OUT902 |
Algorithms - Next lexicographic permutation
1 passed |
100.0% |
IN8 739 OUT1 277 | |
| ✓ | NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 739 OUT1 277 |
Algorithms - Nth Fibonacci mod m
1 passed |
100.0% |
IN8 738 OUT5 939 | |
| ✓ | FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 738 OUT5 939 |
Algorithms - Overlapping substring count
1 passed |
100.0% |
IN8 723 OUT526 | |
| ✓ | SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 723 OUT526 |
Algorithms - Palindrome & anagram checks
1 passed |
100.0% |
IN8 753 OUT1 761 | |
| ✓ | PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT1 761 |
Algorithms - Pascal triangle row
1 passed |
100.0% |
IN8 731 OUT1 356 | |
| ✓ | PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 731 OUT1 356 |
Algorithms - Point-to-segment distance
1 passed |
100.0% |
IN8 771 OUT5 210 | |
| ✓ | PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 771 OUT5 210 |
Algorithms - Prime factorization
1 passed |
100.0% |
IN8 742 OUT1 403 | |
| ✓ | PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 742 OUT1 403 |
Algorithms - Product of array except self
1 passed |
100.0% |
IN8 748 OUT4 789 | |
| ✓ | ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 748 OUT4 789 |
Algorithms - Quadratic Bezier point
1 passed |
100.0% |
IN8 808 OUT9 154 | |
| ✓ | BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 808 OUT9 154 |
Algorithms - Range sum via prefix sums
1 passed |
100.0% |
IN8 730 OUT173 | |
| ✓ | PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 730 OUT173 |
Algorithms - Reverse words & Hamming distance
1 passed |
100.0% |
IN8 734 OUT933 | |
| ✓ | ReverseWordsCodeGenCode-gen: model implements reverse-words + Hamming; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 734 OUT933 |
Algorithms - Rod cutting
1 passed |
100.0% |
IN8 764 OUT1 982 | |
| ✓ | RodCuttingCodeGenCode-gen: model implements rod cutting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 764 OUT1 982 |
Algorithms - Roman numeral to integer
1 passed |
100.0% |
IN8 753 OUT636 | |
| ✓ | RomanToIntCodeGenCode-gen: model implements Roman-to-int; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT636 |
Algorithms - Rotate array right by k
1 failed |
0.0% |
IN8 757 OUT815 | |
| × | RotateArrayCodeGenCode-gen: model implements right rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: RotateRight([1 2 3 4 5],2) got "3 4 5 1 2" expected "4 5 1 2 3"
Values:
r.ResultJson = "\"3 4 5 1 2\""
exp = "\"4 5 1 2 3\"" |
Failed | IN8 757 OUT815 |
Algorithms - Rotate grid 90 CW (discriminator)
1 passed |
100.0% |
IN8 760 OUT907 | |
| ✓ | RotateMatrixCodeGenCode-gen: model implements 90-degree CW rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 760 OUT907 |
Algorithms - RPN evaluator
1 passed |
100.0% |
IN8 762 OUT1 438 | |
| ✓ | RpnCodeGenCode-gen: model implements an RPN evaluator; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 762 OUT1 438 |
Algorithms - Run-length compression (if shorter)
1 passed |
100.0% |
IN8 734 OUT942 | |
| ✓ | StringCompressCodeGenCode-gen: model implements string compression; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 734 OUT942 |
Algorithms - Run-length encoding (encode/decode)
1 passed |
100.0% |
IN8 780 OUT7 714 | |
| ✓ | RunLengthCodeGenCode-gen: model implements RLE encode+decode; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 780 OUT7 714 |
Algorithms - Semantic versioning (compare + bump)
1 passed |
100.0% |
IN8 810 OUT989 | |
| ✓ | SemVerCodeGenCode-gen: model implements semver compare + bump; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:InstructionFollowinglevel:Easysource:Ai4Users |
Passed | IN8 810 OUT989 |
Algorithms - Single-pile Nim (optimal play)
1 passed |
100.0% |
IN8 742 OUT499 | |
| ✓ | NimCodeGenCode-gen: model implements optimal Nim; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 742 OUT499 |
Algorithms - Sliding window maximum
1 passed |
100.0% |
IN8 734 OUT8 033 | |
| ✓ | SlidingWindowMaxCodeGenCode-gen: model implements sliding window maximum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 734 OUT8 033 |
Algorithms - Spiral matrix traversal
1 passed |
100.0% |
IN8 743 OUT1 702 | |
| ✓ | SpiralMatrixCodeGenCode-gen: model implements spiral traversal; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 743 OUT1 702 |
Algorithms - Square matrix multiply
1 passed |
100.0% |
IN8 735 OUT1 180 | |
| ✓ | MatMulCodeGenCode-gen: model implements square matrix multiply; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 735 OUT1 180 |
Algorithms - Strongly connected components
1 passed |
100.0% |
IN8 746 OUT4 993 | |
| ✓ | SccCountCodeGenCode-gen: model implements SCC count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 746 OUT4 993 |
Algorithms - Subset sum reachability
1 passed |
100.0% |
IN8 719 OUT1 600 | |
| ✓ | SubsetSumCodeGenCode-gen: model implements subset-sum reachability; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 719 OUT1 600 |
Algorithms - Substring search (KMP)
1 passed |
100.0% |
IN8 741 OUT1 049 | |
| ✓ | KmpCodeGenCode-gen: model implements KMP search; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 741 OUT1 049 |
Algorithms - Sum of digits & digital root
1 passed |
100.0% |
IN8 720 OUT530 | |
| ✓ | DigitalRootCodeGenCode-gen: model implements sum-of-digits + digital root; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 720 OUT530 |
Algorithms - Temperature conversion
1 passed |
100.0% |
IN8 731 OUT227 | |
| ✓ | TemperatureCodeGenCode-gen: model implements temperature conversion; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 731 OUT227 |
Algorithms - Topological sort (lexicographically smallest)
1 passed |
100.0% |
IN8 772 OUT1 511 | |
| ✓ | TopoCodeGenCode-gen: model implements lexicographically-smallest topological sort.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 772 OUT1 511 |
Algorithms - Trailing zeros of factorial
1 passed |
100.0% |
IN8 732 OUT190 | |
| ✓ | TrailingZerosCodeGenCode-gen: model implements trailing zeros; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 732 OUT190 |
Algorithms - Trapping rain water
1 passed |
100.0% |
IN8 731 OUT1 059 | |
| ✓ | TrappingRainCodeGenCode-gen: model implements trapping rain water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 731 OUT1 059 |
Algorithms - Two-sum indices
1 passed |
100.0% |
IN8 738 OUT2 715 | |
| ✓ | TwoSumCodeGenCode-gen: model implements two-sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 738 OUT2 715 |
Algorithms - Unique grid paths
1 passed |
100.0% |
IN8 713 OUT4 210 | |
| ✓ | UniquePathsCodeGenCode-gen: model implements unique paths; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 713 OUT4 210 |
Algorithms - Valid Sudoku unit
1 passed |
100.0% |
IN8 718 OUT385 | |
| ✓ | SudokuUnitCodeGenCode-gen: model implements Sudoku unit validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 718 OUT385 |
Algorithms - Wildcard string matching
1 passed |
100.0% |
IN8 736 OUT1 798 | |
| ✓ | WildcardMatchCodeGenCode-gen: model implements wildcard match; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 736 OUT1 798 |
Algorithms - Z-function
1 passed |
100.0% |
IN8 737 OUT1 063 | |
| ✓ | ZFunctionCodeGenCode-gen: model implements the Z-function; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 737 OUT1 063 |
Database - Conditional aggregation pivot
1 passed |
100.0% |
IN945 OUT105 | |
| ✓ | PivotCountsCodeGenCode-gen: model writes the pivot query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN945 OUT105 |
Database - Count distinct opponents
1 passed |
100.0% |
IN925 OUT40 | |
| ✓ | DistinctCountCodeGenCode-gen: model writes the COUNT(DISTINCT) query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Easysource:Ai4Users |
Passed | IN925 OUT40 |
Database - Date-range filter (BETWEEN)
1 passed |
100.0% |
IN955 OUT118 | |
| ✓ | DateRangeCodeGenCode-gen: model writes the date-range query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN955 OUT118 |
Database - EXISTS subquery (teams with matches)
1 passed |
100.0% |
IN932 OUT189 | |
| ✓ | ExistsSubqueryCodeGenCode-gen: model writes the EXISTS query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN932 OUT189 |
Database - JOIN player totals (two tables)
1 passed |
100.0% |
IN954 OUT91 | |
| ✓ | JoinScoresCodeGenCode-gen: model writes a JOIN + aggregate query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN954 OUT91 |
Database - LEFT JOIN anti-join (no items)
1 passed |
100.0% |
IN928 OUT146 | |
| ✓ | LeftJoinNullsCodeGenCode-gen: model writes the anti-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN928 OUT146 |
Database - Player stats SQL (multi-aggregate, QueryJson)
1 passed |
100.0% |
IN958 OUT121 | |
| ✓ | PlayerStatsCodeGenCode-gen: model writes a multi-aggregate stats query; graded by comparing its QueryJson output to the oracle query's (arbitrary shape, no struct).1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN958 OUT121 |
Database - Players above average score (subquery)
1 passed |
100.0% |
IN923 OUT93 | |
| ✓ | AboveAverageCodeGenCode-gen: model writes an above-average query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN923 OUT93 |
Database - Players who never played (anti-join)
1 passed |
100.0% |
IN920 OUT194 | |
| ✓ | NeverPlayedCodeGenCode-gen: model writes an anti-join query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN920 OUT194 |
Database - RANK window function
1 passed |
100.0% |
IN950 OUT162 | |
| ✓ | WindowRankCodeGenCode-gen: model writes the RANK() query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN950 OUT162 |
Database - Running total (window function)
1 passed |
100.0% |
IN947 OUT202 | |
| ✓ | RunningTotalCodeGenCode-gen: model writes the window running-total query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN947 OUT202 |
Database - Scoreboard SQL (players with >1 win, HAVING)
1 passed |
100.0% |
IN954 OUT109 | |
| ✓ | WinFilterCodeGenCode-gen: model writes a HAVING query; runs under try/catch with self-repair; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN954 OUT109 |
Database - Scoreboard SQL (top scorers query)
1 passed |
100.0% |
IN955 OUT84 | |
| ✓ | ScoreboardTopScorersCodeGenCode-gen: model writes a scoreboard SQL query; runs under try/catch with a self-repair loop on SQL errors; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN955 OUT84 |
Database - Self-join same-team pairs
1 passed |
100.0% |
IN938 OUT200 | |
| ✓ | SelfJoinCodeGenCode-gen: model writes the self-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN938 OUT200 |
Database - Top scorer per region
1 passed |
100.0% |
IN940 OUT429 | |
| ✓ | TopPerGroupCodeGenCode-gen: model writes the per-group top query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN940 OUT429 |
Database - UPDATE restock (SQL write)
1 passed |
100.0% |
IN932 OUT81 | |
| ✓ | UpdateInventoryCodeGenCode-gen: model writes an UPDATE; applied under try/catch, resulting state compared (via QueryJson) to the oracle update's state.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN932 OUT81 |
Database - Upsert with max merge
1 passed |
100.0% |
IN948 OUT220 | |
| ✓ | UpsertCodeGenCode-gen: model writes the upsert; executed on a fresh DB and the table compared.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN948 OUT220 |
SDLC - Acceptance criteria check
1 passed |
100.0% |
IN8 761 OUT1 509 | |
| ✓ | SdlcAcceptanceCodeGenCode-gen: model implements acceptance-criteria check; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT1 509 |
SDLC - Breaking API change detection
1 passed |
100.0% |
IN8 758 OUT1 521 | |
| ✓ | SdlcApiBreakingCodeGenCode-gen: model implements API-breaking detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 758 OUT1 521 |
SDLC - Condense without losing facts
1 passed |
100.0% |
IN1 046 OUT753 | |
| ✓ | NoInfoLossCodeGenRedaction test: model condenses keeping facts; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 046 OUT753 |
SDLC - Config merge with override
1 passed |
100.0% |
IN8 752 OUT1 377 | |
| ✓ | ConfigMergeCodeGenCode-gen: model implements config merge; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT1 377 |
SDLC - Extract fields into fixed template
1 passed |
100.0% |
IN1 002 OUT105 | |
| ✓ | FieldExtractCodeGenRedaction test: model extracts fields into the template; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 002 OUT105 |
SDLC - Locate conflicting requirement pair
1 passed |
100.0% |
IN8 814 OUT12 664 | |
| ✓ | SdlcConstraintConflictCodeGenCode-gen: model locates the conflicting requirement pair; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN8 814 OUT12 664 |
SDLC - Locate redundant requirement
1 failed |
0.0% |
IN8 799 OUT8 567 | |
| × | SdlcRedundantRequirementCodeGenCode-gen: model locates a redundant requirement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: FindRedundant([a == 2;a <= 10;b == 2]) got 0 expected 1
Values:
r.ResultJson = "0" |
Failed | IN8 799 OUT8 567 |
SDLC - Mask PII to placeholder
1 passed |
100.0% |
IN970 OUT184 | |
| ✓ | MaskPiiCodeGenRedaction test: model masks PII to a placeholder; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN970 OUT184 |
SDLC - Modules on a dependency cycle
1 passed |
100.0% |
IN42 948 OUT25 119 | |
| ✓ | DependencyCycleModulesCodeGenCode-gen: model implements dependency-cycle detection; graded vs exact oracle.4 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN42 948 OUT25 119 |
SDLC - Remove duplicate requirements
1 passed |
100.0% |
IN943 OUT115 | |
| ✓ | DeduplicateCodeGenRedaction test: model removes duplicate requirements; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN943 OUT115 |
SDLC - Renumber requirements sequentially
1 passed |
100.0% |
IN1 010 OUT420 | |
| ✓ | RenumberCodeGenRedaction test: model renumbers sequentially; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 010 OUT420 |
SDLC - Reorganize notes into section template
1 passed |
100.0% |
IN1 124 OUT259 | |
| ✓ | RewriteSectionsCodeGenRedaction test: model reorganizes the notes; deterministic checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 124 OUT259 |
SDLC - Required semver bump from changelog
1 passed |
100.0% |
IN8 757 OUT726 | |
| ✓ | SemverBumpCodeGenCode-gen: model implements the semver-bump rule; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Easysource:Ai4Users |
Passed | IN8 757 OUT726 |
SDLC - Requirement-to-test traceability
1 passed |
100.0% |
IN8 779 OUT1 749 | |
| ✓ | SdlcTraceabilityCodeGenCode-gen: model implements traceability gap; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 779 OUT1 749 |
SDLC - Requirements completeness (spec coverage gap)
1 passed |
100.0% |
IN8 774 OUT2 815 | |
| ✓ | SdlcSpecCoverageCodeGenCode-gen: model implements spec-gap detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 774 OUT2 815 |
SDLC - Requirements consistency (contradiction detection)
1 passed |
100.0% |
IN8 759 OUT2 810 | |
| ✓ | ConsistencyCodeGenCode-gen: model implements requirements-consistency checking; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 759 OUT2 810 |
SDLC - Reshape prose into JSON template
1 passed |
100.0% |
IN1 012 OUT103 | |
| ✓ | JsonShapeCodeGenRedaction test: model reshapes prose into JSON; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 012 OUT103 |
SDLC - Reshape rows into Markdown table
1 passed |
100.0% |
IN966 OUT94 | |
| ✓ | TableFormatCodeGenRedaction test: model builds a Markdown table; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN966 OUT94 |
SDLC - Rewrite in imperative requirement style
1 passed |
100.0% |
IN989 OUT301 | |
| ✓ | StyleRewriteCodeGenRedaction test: model rewrites notes in imperative style; checker + self-review loop.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN989 OUT301 |
SDLC - Roll up estimates over task tree
1 passed |
100.0% |
IN8 770 OUT1 616 | |
| ✓ | EstimateRollupCodeGenCode-gen: model implements estimate roll-up over a task tree; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 770 OUT1 616 |
SDLC - State-machine reachability
1 passed |
100.0% |
IN8 756 OUT1 024 | |
| ✓ | SdlcStateReachableCodeGenCode-gen: model implements state reachability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT1 024 |
SDLC - Ticket priority sort
1 passed |
100.0% |
IN18 020 OUT4 476 | |
| ✓ | SdlcPrioritySortCodeGenCode-gen: model implements priority sort; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN18 020 OUT4 476 |
SDLC - Unify glossary term consistency
1 passed |
100.0% |
IN970 OUT188 | |
| ✓ | GlossaryCodeGenRedaction test: model unifies synonyms to the canonical term; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN970 OUT188 |