Benchmark summary
A custom Ai4Users benchmark suite for SC#, covering code generation, reasoning, and stateful agentic workflows where models use tools, observe results, adapt their actions, recover from failures, and verify the final state.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Agentic - Blue-Green Deploy (health-gated cutover)
1 passed |
100.0% |
IN17 810 OUT1 333 | |
| ✓ | AgenticBlueGreenCutoverHealth-gated blue-green cutover: green is deterministically unhealthy, so the correct outcome is to keep blue live (failed-safe) after observing the precheck; cutting over ships a broken env.1 AI query
5 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN17 810 OUT1 333 |
Agentic - Bugfix (root-cause vs regression)
1 passed |
100.0% |
IN63 032 OUT2 940 | |
| ✓ | AgenticBugfixRegressionGuardBugfix with regression guard: fix a defect so the reproduction passes without breaking regression. The naive over-narrow fix passes the repro but fails a regression case; the root-cause fix passes all.1 AI query
5 tool callscategory:AgenticToolUsecategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN63 032 OUT2 940 |
Agentic - Cache Invalidation (exact set)
1 passed |
100.0% |
IN14 313 OUT2 903 | |
| ✓ | AgenticCacheInvalidationCompletenessCache-invalidation completeness: after changing specific record fields, invalidate exactly the dependent caches (miss one = stale; evict an extra = unnecessary). Both fail the strict check.1 AI query
7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN14 313 OUT2 903 |
Agentic - Canary (metric-gated release)
1 passed |
100.0% |
IN29 918 OUT3 554 | |
| ✓ | AgenticCanaryMetricGatedReleaseMetric-driven canary: two candidates (one healthy, one regressed); the agent must read canary metrics and promote only the healthy one. Blind promotion ships the regression and fails.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN29 918 OUT3 554 |
Agentic - Canary variants (5 metric cases)
5 passed |
100.0% |
IN172 275 OUT18 035 | |
| ✓ | CanaryVariantBothRegressedTwo candidates, BOTH regressed -> promote none, keep the stable build in production.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN25 884 OUT3 105 |
| ✓ | CanaryVariantOnlyAHealthyTwo candidates, only A is healthy -> promote A.1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN42 347 OUT6 354 |
| ✓ | CanaryVariantOnlyBHealthyTwo candidates, only B is healthy -> promote B (do not assume the first listed is best).1 AI query
11 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN35 537 OUT3 057 |
| ✓ | CanaryVariantThreeOneHealthyThree candidates, exactly one healthy -> promote it, reject the other two.1 AI query
12 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN42 555 OUT3 203 |
| ✓ | CanaryVariantTieBreakLatencyTwo candidates, BOTH healthy -> tie-break on lowest p95 latency (promote the faster one).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN25 952 OUT2 316 |
Agentic - Code Review (find injected defect)
1 passed |
100.0% |
IN13 099 OUT1 204 | |
| ✓ | AgenticFindInjectedDefectCode review: identify the one hunk that breaks the contract (H3: off-by-one in remaining days) among behaviour-preserving red herrings.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN13 099 OUT1 204 |
Agentic - Config Drift (remediate exact)
1 passed |
100.0% |
IN11 823 OUT1 006 | |
| ✓ | AgenticConfigDriftRemediationConfig-drift remediation: fix exactly the non-allowlisted drifted keys to desired without touching the environment-specific key. Over-correcting or missing a drifted key fails.1 AI query
6 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN11 823 OUT1 006 |
Agentic - Constraints (multi-section satisfaction)
1 passed |
100.0% |
IN14 425 OUT3 714 | |
| ✓ | AgenticConstraintSatisfactionConstraint satisfaction across multiple runbook sections with a tempting-but-invalid default; the agent must intersect all constraints and pick a feasible pool_size.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN14 425 OUT3 714 |
Agentic - Constraints variants (5 cases)
5 passed |
100.0% |
IN48 471 OUT4 614 | |
| ✓ | ConstraintsVariantBatchSizebatch_size in [100,1000], multiple of 100, divides 10000 (feasible {100,200,500,1000}); tempting 300 does not divide 10000.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 854 OUT1 226 |
| ✓ | ConstraintsVariantCacheMbcache_mb in [128,512], power of two (feasible {128,256,512}); tempting "start at 100" is not a power of two.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 613 OUT735 |
| ✓ | ConstraintsVariantPoolSizepool_size in [40,56], multiple of 8 (feasible {40,48,56}); tempting default 24 violates the minimum.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 646 OUT785 |
| ✓ | ConstraintsVariantTimeoutMstimeout_ms in [200,1000], multiple of 50, and > 750; tempting "typical 500" violates the >750 rule.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 853 OUT1 087 |
| ✓ | ConstraintsVariantWorkerThreadsworker_threads in [12,64], multiple of 4, and NOT 32; tempting "start at 32" is forbidden.1 AI query
3 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN9 505 OUT781 |
Agentic - Data Reconciliation (per-field precedence)
1 passed |
100.0% |
IN13 898 OUT1 411 | |
| ✓ | AgenticTwoSourceReconciliationTwo-source reconciliation with per-field precedence (email->crm, plan->billing) and missing-record creation; blind whole-source copy or wrong precedence fails the per-field grade.1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN13 898 OUT1 411 |
Agentic - Diagnosis (root cause under budget)
1 passed |
100.0% |
IN12 411 OUT1 677 | |
| ✓ | AgenticRootCauseDiagnosisRoot-cause diagnosis with upward fault propagation: identify the deepest unhealthy component whose dependencies are all healthy (the true root), not a symptomatic component higher up.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN12 411 OUT1 677 |
Agentic - Diagnosis variants (5 cases)
5 passed |
100.0% |
IN87 909 OUT8 930 | |
| ✓ | DiagnosisVariantChainDiagnosis variant — linear chain a->b->c->d, fault at c (deepest unhealthy with a healthy dep).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN14 630 OUT2 023 |
| ✓ | DiagnosisVariantDeepLeafDiagnosis variant — shared leaf dependency (svcA->svcB->svcC, svcD->svcB), fault at the deep leaf svcC.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 243 OUT1 501 |
| ✓ | DiagnosisVariantDiamondDiagnosis variant — diamond (top->left,right; left->core; right->core), fault at the convergence node core. Two branches look unhealthy but both converge on the single root.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN14 137 OUT797 |
| ✓ | DiagnosisVariantTreeDiagnosis variant — tree (web->api,cache; api->auth,db), fault at db.1 AI query
10 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN15 042 OUT1 694 |
| ✓ | DiagnosisVariantWideGraphDiagnosis variant — larger 7-node graph, fault mid-graph at n4 (n4->n6 healthy).1 AI query
15 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN28 857 OUT2 915 |
Agentic - ETL Repair (root-cause in stage 2)
1 passed |
100.0% |
IN25 399 OUT2 201 | |
| ✓ | AgenticEtlPipelineRepairETL repair with propagating error: stage2 misconfiguration only surfaces at the stage4 report; the agent must trace back, fix stage2's scale, and reprocess downstream — not patch the symptom.1 AI query
14 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN25 399 OUT2 201 |
Agentic - Feature Flag (staged rollout + kill-switch)
1 passed |
100.0% |
IN23 148 OUT2 569 | |
| ✓ | AgenticFeatureFlagKillSwitchStaged feature-flag rollout with a kill-switch: the feature is healthy at 10% but spikes from 50%; the agent must observe the spike and roll back to a safe stage rather than ship to 100%.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN23 148 OUT2 569 |
Agentic - Game Backend: 1D bouncing ball
1 passed |
100.0% |
IN8 781 OUT2 309 | |
| ✓ | GameBackendBounceCodeGenCode-gen: model implements the bouncing-ball position; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 781 OUT2 309 |
Agentic - Game Backend: 1D swept anti-tunneling
1 passed |
100.0% |
IN8 792 OUT3 869 | |
| ✓ | GameBackendAntiTunnelCodeGenCode-gen: model implements swept 1D movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 792 OUT3 869 |
Agentic - Game Backend: 2048 left-move (slide + merge)
1 passed |
100.0% |
IN18 003 OUT7 417 | |
| ✓ | GameBackend2048CodeGenCode-gen: model implements a 2048 left-move; graded vs exact oracle.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 003 OUT7 417 |
Agentic - Game Backend: 8-direction from vector
1 failed |
0.0% |
IN8 808 OUT2 286 | |
| × | GameBackendDir8CodeGenCode-gen: model implements 8-direction classification; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Dir8(0,-1) got "W" expected "N"
Values:
r.ResultJson = "\"W\""
exp = "\"N\"" |
Failed | IN8 808 OUT2 286 |
Agentic - Game Backend: AABB collision resolution (MTV)
1 passed |
100.0% |
IN8 897 OUT2 630 | |
| ✓ | GameBackendAabbResolveCodeGenCode-gen: model implements AABB push-out; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 897 OUT2 630 |
Agentic - Game Backend: AABB containment
1 passed |
100.0% |
IN8 737 OUT659 | |
| ✓ | GameBackendAabbContainsCodeGenCode-gen: model implements AABB containment; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 737 OUT659 |
Agentic - Game Backend: Acceleration to max speed
1 passed |
100.0% |
IN8 739 OUT1 346 | |
| ✓ | GameBackendAccelCodeGenCode-gen: model implements acceleration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 739 OUT1 346 |
Agentic - Game Backend: Achievement thresholds
1 passed |
100.0% |
IN8 731 OUT2 116 | |
| ✓ | GameBackendAchievementsCodeGenCode-gen: model implements achievements; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 731 OUT2 116 |
Agentic - Game Backend: Active status effects
1 passed |
100.0% |
IN8 753 OUT1 009 | |
| ✓ | GameBackendStatusDurationCodeGenCode-gen: model implements status duration; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT1 009 |
Agentic - Game Backend: Animation frame at time
1 passed |
100.0% |
IN8 771 OUT1 386 | |
| ✓ | GameBackendFrameAtTimeCodeGenCode-gen: model implements frame-at-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 771 OUT1 386 |
Agentic - Game Backend: Army upkeep income
1 passed |
100.0% |
IN8 719 OUT3 755 | |
| ✓ | GameBackendUpkeepCodeGenCode-gen: model implements upkeep; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 719 OUT3 755 |
Agentic - Game Backend: Axis-aligned path follow
1 passed |
100.0% |
IN17 979 OUT6 019 | |
| ✓ | GameBackendPathFollowCodeGenCode-gen: model implements path following; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN17 979 OUT6 019 |
Agentic - Game Backend: Battleship shot resolution
1 failed |
0.0% |
IN8 821 OUT3 378 | |
| × | GameBackendBattleshipCodeGenCode-gen: model implements Battleship shot resolution; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Fire 3x1 [AA.] hits[0,0] (1,0) got "hit" expected "sunk"
Values:
r.ResultJson = "\"hit\""
exp = "\"sunk\"" |
Failed | IN8 821 OUT3 378 |
Agentic - Game Backend: Bounce restitution
1 passed |
100.0% |
IN8 758 OUT1 099 | |
| ✓ | GameBackendRestitutionCodeGenCode-gen: model implements restitution bounce; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 758 OUT1 099 |
Agentic - Game Backend: Bresenham line rasterization
1 passed |
100.0% |
IN17 974 OUT6 142 | |
| ✓ | GameBackendBresenhamCodeGenCode-gen: model implements Bresenham line; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 974 OUT6 142 |
Agentic - Game Backend: Buff/debuff stacking
1 passed |
100.0% |
IN8 784 OUT1 220 | |
| ✓ | GameBackendBuffsCodeGenCode-gen: model implements buff stacking; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 784 OUT1 220 |
Agentic - Game Backend: Build queue completions
1 passed |
100.0% |
IN8 752 OUT343 | |
| ✓ | GameBackendBuildQueueCodeGenCode-gen: model implements the build queue; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 752 OUT343 |
Agentic - Game Backend: Camera-relative movement
1 failed |
0.0% |
IN8 839 OUT922 | |
| × | GameBackendCameraMoveCodeGenCode-gen: model implements camera-relative movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: WorldMove(yaw=3,fx=1,fy=0) got "-0,1" expected "0,1"
Values:
r.ResultJson = "\"-0,1\""
exp = "\"0,1\"" |
Failed | IN8 839 OUT922 |
Agentic - Game Backend: Card hand sort
1 passed |
100.0% |
IN8 760 OUT2 701 | |
| ✓ | GameBackendHandSortCodeGenCode-gen: model implements hand sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 760 OUT2 701 |
Agentic - Game Backend: Card-game rules (blackjack/deck/compare)
1 passed |
100.0% |
IN8 935 OUT4 065 | |
| ✓ | GameBackendCardsCodeGenCode-gen: model implements the card-rule functions to spec; each is graded against an exact oracle over fixed cases + random hands drawn from a real deck.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 935 OUT4 065 |
Agentic - Game Backend: Chart scaling & histogram bins
1 passed |
100.0% |
IN8 824 OUT2 699 | |
| ✓ | GameBackendChartsCodeGenCode-gen: model implements chart scaling + binning; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 824 OUT2 699 |
Agentic - Game Backend: Checkers captures
1 passed |
100.0% |
IN8 794 OUT2 194 | |
| ✓ | GameBackendCheckersCodeGenCode-gen: model implements checkers captures; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 794 OUT2 194 |
Agentic - Game Backend: Collision primitives (circle/aabb/segment)
1 passed |
100.0% |
IN8 997 OUT4 307 | |
| ✓ | GameBackendCollisionCodeGenCode-gen: model implements the collision primitives to spec; each is graded against an exact integer oracle over fixed edge cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 997 OUT4 307 |
Agentic - Game Backend: Combat turn order (initiative)
1 passed |
100.0% |
IN8 733 OUT3 524 | |
| ✓ | GameBackendInitiativeCodeGenCode-gen: model implements combat turn order; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT3 524 |
Agentic - Game Backend: Combo scoring (rising multiplier)
1 passed |
100.0% |
IN8 756 OUT1 283 | |
| ✓ | GameBackendComboCodeGenCode-gen: model implements combo scoring; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 756 OUT1 283 |
Agentic - Game Backend: Control-point capture
1 passed |
100.0% |
IN8 769 OUT2 627 | |
| ✓ | GameBackendCapturePointCodeGenCode-gen: model implements capture point; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT2 627 |
Agentic - Game Backend: Controls & Inventory (spec-driven function battery)
1 passed |
100.0% |
IN8 910 OUT2 759 | |
| ✓ | GameBackendControlsCodeGenCode-gen: model implements TurnRight/TurnLeft/StepForward/ReloadMag to spec; each is run over an oracle-checked battery (fixed edge cases + random inputs).1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 910 OUT2 759 |
Agentic - Game Backend: Cooldowns & resource regen
1 passed |
100.0% |
IN8 827 OUT1 534 | |
| ✓ | GameBackendCooldownCodeGenCode-gen: model implements cooldown/regen helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 827 OUT1 534 |
Agentic - Game Backend: Coyote-time jump
1 passed |
100.0% |
IN8 748 OUT365 | |
| ✓ | GameBackendCoyoteCodeGenCode-gen: model implements coyote-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 748 OUT365 |
Agentic - Game Backend: Crafting capacity
1 passed |
100.0% |
IN18 178 OUT7 189 | |
| ✓ | GameBackendCraftingCodeGenCode-gen: model implements crafting count; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 178 OUT7 189 |
Agentic - Game Backend: Cribbage fifteens
1 passed |
100.0% |
IN8 749 OUT2 783 | |
| ✓ | GameBackendCribbageCodeGenCode-gen: model implements cribbage fifteens; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 749 OUT2 783 |
Agentic - Game Backend: Critical-hit damage
1 passed |
100.0% |
IN8 739 OUT513 | |
| ✓ | GameBackendCritCodeGenCode-gen: model implements crit damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 739 OUT513 |
Agentic - Game Backend: Damage over time
1 passed |
100.0% |
IN8 765 OUT2 680 | |
| ✓ | GameBackendDoTCodeGenCode-gen: model implements damage-over-time; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 765 OUT2 680 |
Agentic - Game Backend: Day/night cycle phase
1 passed |
100.0% |
IN8 751 OUT1 266 | |
| ✓ | GameBackendDayNightCodeGenCode-gen: model implements day/night phase; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 751 OUT1 266 |
Agentic - Game Backend: Dice notation max roll
1 passed |
100.0% |
IN8 766 OUT2 470 | |
| ✓ | GameBackendDiceCodeGenCode-gen: model implements dice-notation max roll; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 766 OUT2 470 |
Agentic - Game Backend: Enemy AI state machine
1 passed |
100.0% |
IN8 852 OUT2 709 | |
| ✓ | GameBackendEnemyFsmCodeGenCode-gen: model implements the enemy FSM; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 852 OUT2 709 |
Agentic - Game Backend: Equipment slot validation
1 passed |
100.0% |
IN8 766 OUT2 183 | |
| ✓ | GameBackendInventorySlotsCodeGenCode-gen: model implements slot validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 766 OUT2 183 |
Agentic - Game Backend: Flood-fill region size
1 passed |
100.0% |
IN8 754 OUT4 317 | |
| ✓ | GameBackendFloodFillCodeGenCode-gen: model implements flood-fill region size; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT4 317 |
Agentic - Game Backend: Fog of war visibility
1 passed |
100.0% |
IN17 871 OUT3 436 | |
| ✓ | GameBackendFogOfWarCodeGenCode-gen: model implements fog-of-war visibility; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN17 871 OUT3 436 |
Agentic - Game Backend: Fog reveal (Chebyshev radius)
1 passed |
100.0% |
IN17 884 OUT6 486 | |
| ✓ | GameBackendFogRadiusCodeGenCode-gen: model implements fog reveal; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 884 OUT6 486 |
Agentic - Game Backend: Friction stop frame
1 passed |
100.0% |
IN8 747 OUT1 149 | |
| ✓ | GameBackendFrictionCodeGenCode-gen: model implements friction stop; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 747 OUT1 149 |
Agentic - Game Backend: Game of Life (code-gen vs oracle)
1 failed |
0.0% |
IN8 806 OUT3 132 | |
| × | GameOfLifeCodeGenCode-gen: the model implements Tick(); we simulate its candidate on random boards for several steps and compare every generation to the reference oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users candidate did not compile: SC# compilation failed inside the script runner.
Compiler diagnostics:
7:32: Expected ')' to close the StringBuilder constructor. Use 'new StringBuilder()'., found 'board'. |
Failed | IN8 806 OUT3 132 |
Agentic - Game Backend: Greedy coin change
1 passed |
100.0% |
IN8 748 OUT2 120 | |
| ✓ | GameBackendMakeChangeCodeGenCode-gen: model implements greedy change; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 748 OUT2 120 |
Agentic - Game Backend: Grid inventory & stacking (Diablo-style)
1 passed |
100.0% |
IN8 881 OUT2 987 | |
| ✓ | GameBackendInventoryCodeGenCode-gen: model implements the grid-inventory functions to spec; each is graded against an exact oracle over fixed cases + random inputs.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 881 OUT2 987 |
Agentic - Game Backend: Grid pathfinding (shortest path, BFS)
1 passed |
100.0% |
IN8 836 OUT6 020 | |
| ✓ | GameBackendPathfindCodeGenCode-gen: model implements grid shortest-path length; graded against an exact BFS oracle over fixed cases + random grids.1 AI query
0 tool callscategory:AgenticCodingcategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 836 OUT6 020 |
Agentic - Game Backend: Grid raycast
1 passed |
100.0% |
IN17 915 OUT4 000 | |
| ✓ | GameBackendRayMarchCodeGenCode-gen: model implements raycast; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 915 OUT4 000 |
Agentic - Game Backend: Health bar segments
1 passed |
100.0% |
IN8 805 OUT2 804 | |
| ✓ | GameBackendHealthBarCodeGenCode-gen: model implements the health bar; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 805 OUT2 804 |
Agentic - Game Backend: High-score table insert
1 passed |
100.0% |
IN27 044 OUT9 541 | |
| ✓ | GameBackendHighScoreCodeGenCode-gen: model implements high-score insertion; graded vs exact oracle.3 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN27 044 OUT9 541 |
Agentic - Game Backend: Hit chance
1 passed |
100.0% |
IN8 743 OUT1 610 | |
| ✓ | GameBackendHitChanceCodeGenCode-gen: model implements hit chance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 743 OUT1 610 |
Agentic - Game Backend: Inventory encumbrance
1 passed |
100.0% |
IN8 740 OUT2 047 | |
| ✓ | GameBackendEncumbranceCodeGenCode-gen: model implements encumbrance; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 740 OUT2 047 |
Agentic - Game Backend: Inventory first-fit placement
1 passed |
100.0% |
IN17 901 OUT4 723 | |
| ✓ | GameBackendInventoryFitCodeGenCode-gen: model implements inventory first-fit; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 901 OUT4 723 |
Agentic - Game Backend: Inventory stack split
1 passed |
100.0% |
IN17 610 OUT1 840 | |
| ✓ | GameBackendStackSplitCodeGenCode-gen: model implements stack split; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN17 610 OUT1 840 |
Agentic - Game Backend: Isometric projection
1 passed |
100.0% |
IN8 773 OUT1 187 | |
| ✓ | GameBackendIsometricCodeGenCode-gen: model implements iso projection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 773 OUT1 187 |
Agentic - Game Backend: Jump buffer
1 passed |
100.0% |
IN8 761 OUT2 074 | |
| ✓ | GameBackendJumpBufferCodeGenCode-gen: model implements jump buffer; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT2 074 |
Agentic - Game Backend: Knockback with arena clamp
1 passed |
100.0% |
IN8 800 OUT2 383 | |
| ✓ | GameBackendKnockbackCodeGenCode-gen: model implements knockback; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 800 OUT2 383 |
Agentic - Game Backend: Leaderboard rank
1 passed |
100.0% |
IN8 733 OUT1 535 | |
| ✓ | GameBackendLeaderboardRankCodeGenCode-gen: model implements leaderboard rank; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 733 OUT1 535 |
Agentic - Game Backend: Lifecycle & counters (HP/death/respawn/game-over)
1 passed |
100.0% |
IN18 006 OUT4 423 | |
| ✓ | GameBackendLifecycleCodeGenCode-gen: model implements the lifecycle/counter functions to spec; each is graded against an exact oracle over fixed edge cases + random inputs.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 006 OUT4 423 |
Agentic - Game Backend: Lifesteal healing
1 passed |
100.0% |
IN8 744 OUT1 789 | |
| ✓ | GameBackendLifestealCodeGenCode-gen: model implements lifesteal; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 744 OUT1 789 |
Agentic - Game Backend: Lights Out toggle
1 passed |
100.0% |
IN8 760 OUT3 731 | |
| ✓ | GameBackendLightsOutCodeGenCode-gen: model implements Lights Out toggle; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 760 OUT3 731 |
Agentic - Game Backend: Match-3 cascade
1 failed |
0.0% |
IN8 818 OUT4 828 | |
| × | GameBackendMatch3CascadeCodeGenCode-gen: model implements match-3 cascade; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Resolve threw: The script exceeded the timeout of 30000 ms. |
Failed | IN8 818 OUT4 828 |
Agentic - Game Backend: Match-3 detection
1 passed |
100.0% |
IN8 782 OUT3 351 | |
| ✓ | GameBackendMatchThreeCodeGenCode-gen: model implements match-3 detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 782 OUT3 351 |
Agentic - Game Backend: Match-3 swap validity
1 passed |
100.0% |
IN18 293 OUT13 524 | |
| ✓ | GameBackendMatch3SwapCodeGenCode-gen: model implements match-3 swap validity; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 293 OUT13 524 |
Agentic - Game Backend: Memory match pairs
1 passed |
100.0% |
IN8 745 OUT1 865 | |
| ✓ | GameBackendMemoryMatchCodeGenCode-gen: model implements memory match; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 745 OUT1 865 |
Agentic - Game Backend: Mercy invincibility (i-frames)
1 passed |
100.0% |
IN8 800 OUT1 367 | |
| ✓ | GameBackendInvincibilityCodeGenCode-gen: model implements i-frames damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 800 OUT1 367 |
Agentic - Game Backend: Minesweeper (adjacency + flood-fill reveal)
1 passed |
100.0% |
IN8 882 OUT3 899 | |
| ✓ | GameBackendMinesweeperCodeGenCode-gen: model implements minesweeper adjacency + flood-fill reveal; graded against an exact oracle over fixed + random boards.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 882 OUT3 899 |
Agentic - Game Backend: Minimap downsample
1 passed |
100.0% |
IN8 776 OUT2 408 | |
| ✓ | GameBackendMinimapCodeGenCode-gen: model implements minimap downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 776 OUT2 408 |
Agentic - Game Backend: Minimap marker position
1 passed |
100.0% |
IN17 718 OUT2 484 | |
| ✓ | GameBackendMinimapMarkerCodeGenCode-gen: model implements minimap marker; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 718 OUT2 484 |
Agentic - Game Backend: N-in-a-row win detection (Connect-4)
1 passed |
100.0% |
IN18 218 OUT9 947 | |
| ✓ | GameBackendConnect4CodeGenCode-gen: model implements N-in-a-row winner detection; graded vs exact oracle.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN18 218 OUT9 947 |
Agentic - Game Backend: Nonogram row validation
1 passed |
100.0% |
IN8 756 OUT1 942 | |
| ✓ | GameBackendNonogramCodeGenCode-gen: model implements nonogram validation; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT1 942 |
Agentic - Game Backend: OBB collision (rotating rectangles, SAT)
1 passed |
100.0% |
IN18 627 OUT13 418 | |
| ✓ | GameBackendObbCodeGenCode-gen: model implements ObbOverlap for two oriented rectangles; graded against an exact integer SAT oracle over fixed + random boxes.2 AI queries
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 627 OUT13 418 |
Agentic - Game Backend: Particle lifetime & expiry
1 passed |
100.0% |
IN8 838 OUT1 956 | |
| ✓ | GameBackendParticlesCodeGenCode-gen: model implements particle lifetime functions; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 838 OUT1 956 |
Agentic - Game Backend: Party XP share
1 passed |
100.0% |
IN8 756 OUT1 521 | |
| ✓ | GameBackendXpShareCodeGenCode-gen: model implements XP share; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 756 OUT1 521 |
Agentic - Game Backend: Patrol waypoints (ping-pong)
1 passed |
100.0% |
IN8 779 OUT465 | |
| ✓ | GameBackendPatrolCodeGenCode-gen: model implements patrol positioning; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 779 OUT465 |
Agentic - Game Backend: Platformer jump arc
1 passed |
100.0% |
IN8 770 OUT1 240 | |
| ✓ | GameBackendJumpCodeGenCode-gen: model implements the jump arc; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 770 OUT1 240 |
Agentic - Game Backend: Point on circle (trig)
1 passed |
100.0% |
IN8 803 OUT1 678 | |
| ✓ | GameBackendCircleMotionCodeGenCode-gen: model implements point-on-circle; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 803 OUT1 678 |
Agentic - Game Backend: Poker hand comparison
1 failed |
0.0% |
IN8 879 OUT9 658 | |
| × | GameBackendPokerCompareCodeGenCode-gen: model compares poker hands; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: Compare(KS 8C 2D 4D 6S | 9C TS 9H 9S QD) got 1 expected -1
Values:
r.ResultJson = "1" |
Failed | IN8 879 OUT9 658 |
Agentic - Game Backend: Poker hand ranking
1 passed |
100.0% |
IN8 822 OUT5 726 | |
| ✓ | GameBackendPokerCodeGenCode-gen: model implements poker hand-category ranking; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 822 OUT5 726 |
Agentic - Game Backend: Pop-cap training
1 passed |
100.0% |
IN8 757 OUT1 291 | |
| ✓ | GameBackendPopCapCodeGenCode-gen: model implements pop-cap training; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 757 OUT1 291 |
Agentic - Game Backend: Progress bar fill
1 passed |
100.0% |
IN8 739 OUT1 390 | |
| ✓ | GameBackendProgressBarCodeGenCode-gen: model implements progress-bar fill; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 739 OUT1 390 |
Agentic - Game Backend: Projectile range (physics/trig)
1 passed |
100.0% |
IN8 761 OUT1 872 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements projectile range; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT1 872 |
Agentic - Game Backend: Projectile with ground bounce
1 passed |
100.0% |
IN8 795 OUT1 343 | |
| ✓ | GameBackendProjectileCodeGenCode-gen: model implements the projectile sim; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 795 OUT1 343 |
Agentic - Game Backend: Resource gathering
1 passed |
100.0% |
IN8 756 OUT697 | |
| ✓ | GameBackendGatherCodeGenCode-gen: model implements resource gathering; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 756 OUT697 |
Agentic - Game Backend: Reversi flip count
1 passed |
100.0% |
IN8 827 OUT4 192 | |
| ✓ | GameBackendReversiCodeGenCode-gen: model implements Reversi flip count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 827 OUT4 192 |
Agentic - Game Backend: Rotate 2D vector (trig)
1 passed |
100.0% |
IN8 772 OUT1 117 | |
| ✓ | GameBackendRotateVectorCodeGenCode-gen: model implements vector rotation; graded vs oracle with +/-1 tolerance.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT1 117 |
Agentic - Game Backend: Round-robin deal
1 passed |
100.0% |
IN8 761 OUT2 510 | |
| ✓ | GameBackendDeckDealCodeGenCode-gen: model implements dealing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 761 OUT2 510 |
Agentic - Game Backend: RTS resource economy
1 passed |
100.0% |
IN8 752 OUT519 | |
| ✓ | GameBackendRtsEconomyCodeGenCode-gen: model implements RTS economy; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 752 OUT519 |
Agentic - Game Backend: RTS unit counters & damage
1 passed |
100.0% |
IN8 804 OUT1 299 | |
| ✓ | GameBackendRtsCodeGenCode-gen: model implements RTS counters + damage; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 804 OUT1 299 |
Agentic - Game Backend: Safest neighbour (danger map)
1 passed |
100.0% |
IN8 792 OUT4 815 | |
| ✓ | GameBackendDangerMapCodeGenCode-gen: model implements danger-map safest neighbour; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 792 OUT4 815 |
Agentic - Game Backend: Scroll viewport
1 passed |
100.0% |
IN8 774 OUT1 757 | |
| ✓ | GameBackendScrollViewCodeGenCode-gen: model implements scroll viewport; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 774 OUT1 757 |
Agentic - Game Backend: Set card validity
1 passed |
100.0% |
IN8 753 OUT1 730 | |
| ✓ | GameBackendSetGameCodeGenCode-gen: model implements Set validity; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT1 730 |
Agentic - Game Backend: Shop affordability
1 passed |
100.0% |
IN8 733 OUT996 | |
| ✓ | GameBackendShopAffordCodeGenCode-gen: model implements shop affordability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 733 OUT996 |
Agentic - Game Backend: Sliding puzzle move
1 passed |
100.0% |
IN8 764 OUT3 421 | |
| ✓ | GameBackendSlidePuzzleCodeGenCode-gen: model implements sliding puzzle move; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 764 OUT3 421 |
Agentic - Game Backend: Snake head movement
1 passed |
100.0% |
IN8 821 OUT2 376 | |
| ✓ | GameBackendSnakeCodeGenCode-gen: model implements snake head movement; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 821 OUT2 376 |
Agentic - Game Backend: Snake tick
1 failed |
0.0% |
IN8 891 OUT1 038 | |
| × | GameBackendSnakeCodeGenCode-gen: model implements the Snake tick; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Tick [1,0;0,0] R food(9,9) got "2,0" expected "2,0;1,0"
Values:
r.ResultJson = "\"2,0\""
exp = "\"2,0;1,0\"" |
Failed | IN8 891 OUT1 038 |
Agentic - Game Backend: Snap to grid
1 passed |
100.0% |
IN8 749 OUT1 163 | |
| ✓ | GameBackendSnapGridCodeGenCode-gen: model implements snap-to-grid; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 749 OUT1 163 |
Agentic - Game Backend: Sokoban move/push legality
1 passed |
100.0% |
IN8 852 OUT1 319 | |
| ✓ | GameBackendSokobanCodeGenCode-gen: model implements Sokoban move legality; graded vs exact oracle.1 AI query
0 tool callscategory:AgenticCodingcategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 852 OUT1 319 |
Agentic - Game Backend: Solitaire tableau move
1 passed |
100.0% |
IN8 791 OUT3 079 | |
| ✓ | GameBackendSolitaireCodeGenCode-gen: model implements solitaire move legality; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 791 OUT3 079 |
Agentic - Game Backend: Spawn scheduler count
1 passed |
100.0% |
IN8 748 OUT1 879 | |
| ✓ | GameBackendSpawnSchedulerCodeGenCode-gen: model implements spawn scheduler; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 748 OUT1 879 |
Agentic - Game Backend: Spawn-wave alive count
1 passed |
100.0% |
IN8 760 OUT1 908 | |
| ✓ | GameBackendSpawnWaveCodeGenCode-gen: model implements spawn-wave alive count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 760 OUT1 908 |
Agentic - Game Backend: Stat level scaling
1 passed |
100.0% |
IN8 752 OUT1 497 | |
| ✓ | GameBackendLevelScalingCodeGenCode-gen: model implements level scaling; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT1 497 |
Agentic - Game Backend: Steering seek (greedy pursuit)
1 passed |
100.0% |
IN8 778 OUT2 417 | |
| ✓ | GameBackendSeekCodeGenCode-gen: model implements steering seek; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 778 OUT2 417 |
Agentic - Game Backend: Tactical movement range
1 passed |
100.0% |
IN8 790 OUT3 290 | |
| ✓ | GameBackendMoveRangeCodeGenCode-gen: model implements movement range; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 790 OUT3 290 |
Agentic - Game Backend: Tech tree eligibility
1 passed |
100.0% |
IN17 897 OUT7 269 | |
| ✓ | GameBackendTechTreeCodeGenCode-gen: model implements tech-tree eligibility; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 897 OUT7 269 |
Agentic - Game Backend: Tetris line clearing
1 passed |
100.0% |
IN8 763 OUT3 549 | |
| ✓ | GameBackendTetrisCodeGenCode-gen: model implements Tetris line clearing; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 763 OUT3 549 |
Agentic - Game Backend: Threat table top target
1 passed |
100.0% |
IN17 919 OUT6 268 | |
| ✓ | GameBackendThreatCodeGenCode-gen: model implements threat table; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 919 OUT6 268 |
Agentic - Game Backend: Tic-tac-toe result
1 passed |
100.0% |
IN8 748 OUT2 795 | |
| ✓ | GameBackendTicTacToeCodeGenCode-gen: model implements tic-tac-toe result; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 748 OUT2 795 |
Agentic - Game Backend: Toroidal wrap (Pac-Man)
1 passed |
100.0% |
IN17 685 OUT2 385 | |
| ✓ | GameBackendTorusWrapCodeGenCode-gen: model implements toroidal wrap; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 685 OUT2 385 |
Agentic - Game Backend: Tower targeting (nearest in range)
1 passed |
100.0% |
IN8 772 OUT3 142 | |
| ✓ | GameBackendTowerTargetCodeGenCode-gen: model implements tower targeting; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 772 OUT3 142 |
Agentic - Game Backend: Transaction ledger
1 passed |
100.0% |
IN8 775 OUT2 209 | |
| ✓ | GameBackendBankCodeGenCode-gen: model implements the ledger; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 775 OUT2 209 |
Agentic - Game Backend: Trick-taking winner
1 passed |
100.0% |
IN18 092 OUT8 305 | |
| ✓ | GameBackendTrickWinnerCodeGenCode-gen: model implements trick winner; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN18 092 OUT8 305 |
Agentic - Game Backend: Turn-based duel resolution
1 passed |
100.0% |
IN8 835 OUT2 627 | |
| ✓ | GameBackendCombatDuelCodeGenCode-gen: model implements the duel resolver; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 835 OUT2 627 |
Agentic - Game Backend: Typed armor mitigation
1 passed |
100.0% |
IN17 873 OUT5 465 | |
| ✓ | GameBackendArmorCodeGenCode-gen: model implements armor mitigation; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN17 873 OUT5 465 |
Agentic - Game Backend: UI menu (wrap/clamp/paginate)
1 passed |
100.0% |
IN8 851 OUT2 956 | |
| ✓ | GameBackendUiMenuCodeGenCode-gen: model implements the UI/menu helpers; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 851 OUT2 956 |
Agentic - Game Backend: Uno playable cards
1 passed |
100.0% |
IN8 787 OUT3 011 | |
| ✓ | GameBackendUnoCodeGenCode-gen: model implements Uno playability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 787 OUT3 011 |
Agentic - Game Backend: Velocity reflection
1 passed |
100.0% |
IN8 731 OUT1 082 | |
| ✓ | GameBackendReflectCodeGenCode-gen: model implements reflection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 731 OUT1 082 |
Agentic - Game Backend: Wall-slide fall
1 passed |
100.0% |
IN8 739 OUT503 | |
| ✓ | GameBackendWallSlideCodeGenCode-gen: model implements wall slide; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 739 OUT503 |
Agentic - Game Backend: War card compare
1 passed |
100.0% |
IN8 759 OUT2 821 | |
| ✓ | GameBackendWarCodeGenCode-gen: model implements War comparison; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 759 OUT2 821 |
Agentic - Game Backend: Weighted loot table
1 passed |
100.0% |
IN8 762 OUT2 518 | |
| ✓ | GameBackendLootTableCodeGenCode-gen: model implements weighted loot pick; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Easysource:Ai4Users |
Passed | IN8 762 OUT2 518 |
Agentic - Game Backend: Weighted terrain pathfinding
1 passed |
100.0% |
IN8 812 OUT3 263 | |
| ✓ | GameBackendWeightedPathCodeGenCode-gen: model implements weighted grid pathfinding; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 812 OUT3 263 |
Agentic - Game Backend: Word-wrap line count
1 passed |
100.0% |
IN8 744 OUT2 104 | |
| ✓ | GameBackendTooltipWrapCodeGenCode-gen: model implements word-wrap line count; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT2 104 |
Agentic - Game Backend: XP to character level
1 passed |
100.0% |
IN8 752 OUT2 691 | |
| ✓ | GameBackendXpLevelCodeGenCode-gen: model implements XP-to-level; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT2 691 |
Agentic - Idempotency (poison-retry / no double charge)
1 passed |
100.0% |
IN14 683 OUT929 | |
| ✓ | AgenticIdempotentReconciliationPoison-retry hazard: a charge returns a transient failure AFTER the effect applied; blind retry double-charges. Correct agent reconciles via status before retrying.1 AI query
8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN14 683 OUT929 |
Agentic - Irreversible Commit (measure twice)
1 passed |
100.0% |
IN22 677 OUT1 413 | |
| ✓ | AgenticIrreversibleCommitSafetyIrreversible commit under prerequisites: the agent must consult all sources, resolve the dispute and apply the pending credit, THEN finalize. Finalizing early commits a permanent bad state.1 AI query
9 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN22 677 OUT1 413 |
Agentic - Least Privilege (exact grants)
1 passed |
100.0% |
IN11 501 OUT494 | |
| ✓ | AgenticLeastPrivilegeGrantLeast-privilege grant: provision exactly the documented permission set. Missing one fails the job; granting any extra fails the least-privilege audit. Both are caught by the strict check.1 AI query
5 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN11 501 OUT494 |
Agentic - Ledger (simulated DB read-modify-write)
1 passed |
100.0% |
IN12 352 OUT3 614 | |
| ✓ | AgenticLedgerReadModifyWriteSimulated-DB read-modify-write over an account store: apply chained transfers in order, skipping overdrafts, preserving money conservation. Stale reads or reordering fail the grade.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 352 OUT3 614 |
Agentic - Ledger variants (5 RMW cases)
5 passed |
100.0% |
IN61 423 OUT14 601 | |
| ✓ | LedgerVariantEarlyChainLedger RMW variant v2 (4 accounts, early chain then a skip).1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 159 OUT3 504 |
| ✓ | LedgerVariantFiveAccountsLedger RMW variant v4 (5 accounts, two chains + a skip).1 AI query
8 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 533 OUT3 068 |
| ✓ | LedgerVariantLongChainLedger RMW variant v5 (6 accounts, long dependent chain).1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 906 OUT3 660 |
| ✓ | LedgerVariantSmallChainLedger RMW variant v1 (4 accounts, chained batch, one overdraft-skip).1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN12 101 OUT2 514 |
| ✓ | LedgerVariantTightMarginsLedger RMW variant v3 (3 accounts, tight overdraft margins).1 AI query
6 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN11 724 OUT1 855 |
Agentic - Lock Ordering (deadlock avoidance)
1 passed |
100.0% |
IN24 618 OUT2 752 | |
| ✓ | AgenticLockOrderingDeadlockAvoidanceDeadlock avoidance: acquire each operation's locks in canonical order; the listed order is a trap. Any out-of-order acquire, missed operation, or leftover held lock fails.1 AI query
25 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN24 618 OUT2 752 |
Agentic - Lock Ordering variants (5 cases)
1 passed2 failed2 broken |
20.0% |
IN147 750 OUT40 532 | |
| ! | LockVariantFiveResFourOps1 AI query
10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: response.Completed
Context: Agent response did not complete.
Values:
response.Completed = false |
Broken | IN23 780 OUT4 853 |
| × | LockVariantFourResThreeOps1 AI query
36 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: !Globals.Get<bool>("lock.violation", false)
Context: A lock-ordering violation (deadlock risk) occurred: locks acquired out of canonical order. |
Failed | IN60 730 OUT11 112 |
| ✓ | LockVariantHeavyReuse1 AI query
23 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN35 161 OUT2 893 |
| × | LockVariantReverseListed1 AI query
361 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users External function 'AskAi' threw InvalidOperationException: Connection refused (127.0.0.1:8000) |
Failed | IN18 807 OUT17 395 |
| ! | LockVariantThreeLockOp1 AI query
7 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: response.Completed
Context: Agent response did not complete.
Values:
response.Completed = false |
Broken | IN9 272 OUT4 279 |
Agentic - Migration (cascading rollback + reapply)
1 passed |
100.0% |
IN24 902 OUT1 383 | |
| ✓ | AgenticCascadingMigrationRollbackMulti-stage agentic migration with a happy-path trap: naive in-order apply passes each step but leaves an inconsistent backfill; recovery needs dependency-aware rollback + reapply.1 AI query
11 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN24 902 OUT1 383 |
Agentic - Requirements (detect contradiction)
1 passed |
100.0% |
IN7 149 OUT732 | |
| ✓ | AgenticDetectConflictingRequirementsRequirements-analysis: find the one contradictory requirement pair (deterministic; pinned by stated latency numbers) among consistent red herrings.1 AI query
2 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN7 149 OUT732 |
Agentic - Retry/Circuit-breaker (transient vs hard-down)
1 passed |
100.0% |
IN25 846 OUT1 883 | |
| ✓ | AgenticRetryCircuitBreakerRetry/circuit-breaker discipline: persist through a transient failure (svc_a succeeds on attempt 3) but stop and open the circuit on a hard-down service (svc_b), without exceeding the retry budget.1 AI query
10 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN25 846 OUT1 883 |
Agentic - Saga (compensation / distributed rollback)
1 passed |
100.0% |
IN17 869 OUT895 | |
| ✓ | AgenticSagaCompensationDistributed-transaction saga where the final step fails permanently; the only consistent outcome is full compensation of earlier steps in reverse order.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN17 869 OUT895 |
Agentic - Saga variants (5 compensation cases)
5 passed |
100.0% |
IN80 624 OUT3 120 | |
| ✓ | SagaVariantCommitAllSucceed3-step saga where every step succeeds -> COMMITTED; the agent must NOT over-compensate a success.1 AI query
4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN13 397 OUT429 |
| ✓ | SagaVariantFailStep2Of33-step saga where step 2 fails permanently -> compensate only step 1 (aborted); never reaches step 3.1 AI query
4 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN13 447 OUT763 |
| ✓ | SagaVariantFailStep3Of33-step saga where the final step (3) fails permanently -> compensate steps 1-2 (aborted).1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 773 OUT556 |
| ✓ | SagaVariantFailStep3Of44-step saga where step 3 fails permanently -> compensate steps 1-2 (aborted); step 4 never runs.1 AI query
6 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 773 OUT608 |
| ✓ | SagaVariantFailStep4Of44-step saga where the final step (4) fails permanently -> compensate steps 1-3 (aborted).1 AI query
8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN20 234 OUT764 |
Agentic - Schema Compat (expand-contract)
1 passed |
100.0% |
IN17 669 OUT882 | |
| ✓ | AgenticBackwardCompatibleMigrationBackward-compatible (expand-contract) schema migration: add nullable -> backfill -> mark NOT NULL. Any other order permanently breaks compatibility; graded on the whole path, not just the end state.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN17 669 OUT882 |
Agentic - Scratchpad (self-repair, Dyck, tool-decision)
4 passed3 failed |
57.1% |
IN234 616 OUT44 673 | |
| ✓ | AgenticCongruentWindowsRepairAgentic coding: write, execute, inspect feedback, repair, retry.1 AI query
2 tool callscategory:AgenticToolUsecategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN32 516 OUT5 593 |
| × | AgenticDyckExecutableScratchpadDyck completion using model-written executable SC# scratchpad.1 AI query
4 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: got != ""
Context: Model did not provide an ANSWER marker.
Values:
got = "" |
Failed | IN54 682 OUT11 601 |
| ✓ | ToolDecisionEasyArithmeticTool-decision calibration: trivial arithmetic.1 AI query
0 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN10 526 OUT51 |
| ✓ | ToolDecisionExclusiveMultiplesTool-decision calibration: moderate exact counting.1 AI query
2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN32 012 OUT2 065 |
| ✓ | ToolDecisionFibonacciModuloTool-decision calibration: exact larger arithmetic.1 AI query
1 tool callcategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN21 185 OUT1 520 |
| × | ToolDecisionLongDyckTool-decision calibration: long Dyck symbolic tracking.1 AI query
3 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:VeryHardsource:Ai4Users Assertion expression evaluated to false: answer != ""
Context: ToolDecisionLongDyck: missing ANSWER marker.
Values:
answer = "" |
Failed | IN45 231 OUT10 051 |
| × | ToolDecisionSubsetCountTool-decision calibration: exact subset counting.1 AI query
2 tool callscategory:AgenticReasoningcategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: answer != ""
Context: ToolDecisionSubsetCount: missing ANSWER marker.
Values:
answer = "" |
Failed | IN38 464 OUT13 792 |
Agentic - Secret Rotation (zero-downtime ordering)
1 passed |
100.0% |
IN16 327 OUT1 427 | |
| ✓ | AgenticZeroDowntimeSecretRotationZero-downtime secret rotation: safe order is add-new -> switch-consumers -> revoke-old. Any other order opens a downtime window (permanently flagged). Graded on the whole path, not just end state.1 AI query
5 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN16 327 OUT1 427 |
Agentic - StatefulOps (incident recovery, concurrency replan)
2 passed |
100.0% |
IN53 736 OUT2 767 | |
| ✓ | AgenticCheckoutIncidentRecoveryStateful agentic incident recovery: inspect, plan, modify, restart, verify and repair.1 AI query
7 tool callscategory:AgenticToolUselevel:Hardsource:Ai4Users |
Passed | IN22 243 OUT1 255 |
| ✓ | AgenticConcurrentDeploymentRecoveryStateful agentic recovery with optimistic concurrency, hidden concurrent mutation, deterministic verification and up to 10 conversational recovery attempts.1 AI query
8 tool callscategory:AgenticToolUselevel:VeryHardsource:Ai4Users |
Passed | IN31 493 OUT1 512 |
Agentic - Test Authoring (write a test that catches the bug)
1 passed |
100.0% |
IN11 654 OUT1 360 | |
| ✓ | AgenticWriteBugCatchingTestTest authoring: submit a single test case that is correct per the contract AND catches the bug in the current (upper-bound-missing) Clamp implementation.1 AI query
7 tool callscategory:AgenticToolUsecategory:Reasoninglevel:Hardsource:Ai4Users |
Passed | IN11 654 OUT1 360 |
Agentic - Topological Planning (deps + prereq)
1 passed |
100.0% |
IN31 104 OUT2 282 | |
| ✓ | AgenticTopologicalPipelineTopological execution of a task DAG with one unmet external prerequisite; naive listed-order runs fail and the missing credential must be provisioned before its dependent task.1 AI query
9 tool callscategory:AgenticToolUsecategory:Reasoninglevel:VeryHardsource:Ai4Users |
Passed | IN31 104 OUT2 282 |
Algorithms - 0/1 Knapsack (dynamic programming)
1 passed |
100.0% |
IN8 766 OUT3 977 | |
| ✓ | KnapsackCodeGenCode-gen: model implements 0/1 knapsack; graded against an exact DP oracle over fixed + random item sets.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 766 OUT3 977 |
Algorithms - 1D elastic collision
1 passed |
100.0% |
IN8 806 OUT2 569 | |
| ✓ | ElasticCollision1DCodeGenCode-gen: model implements 1D elastic collision; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 806 OUT2 569 |
Algorithms - All-pairs shortest path pair
1 failed |
0.0% |
IN28 323 OUT16 056 | |
| × | FloydWarshallPairCodeGenCode-gen: model implements all-pairs shortest path query; graded vs exact oracle.4 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users External function 'AskAi' threw InvalidOperationException: Response status code does not indicate success: 500 (Internal Server Error). |
Failed | IN28 323 OUT16 056 |
Algorithms - Balanced brackets (stack)
1 passed |
100.0% |
IN8 737 OUT3 178 | |
| ✓ | ValidBracketsCodeGenCode-gen: model implements bracket validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 737 OUT3 178 |
Algorithms - Bellman-Ford shortest path (neg weights/cycles)
1 passed |
100.0% |
IN18 139 OUT12 027 | |
| ✓ | BellmanCodeGenCode-gen: model implements Bellman-Ford; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN18 139 OUT12 027 |
Algorithms - Binary Gray code
1 passed |
100.0% |
IN8 710 OUT521 | |
| ✓ | GrayCodeCodeGenCode-gen: model implements Gray code; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 710 OUT521 |
Algorithms - Bipartite graph check
1 passed |
100.0% |
IN8 756 OUT5 129 | |
| ✓ | BipartiteCodeGenCode-gen: model implements bipartite check; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT5 129 |
Algorithms - Bit tricks (popcount/pow2/lowbit)
1 passed |
100.0% |
IN8 784 OUT1 682 | |
| ✓ | BitTricksCodeGenCode-gen: model implements bit tricks; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 784 OUT1 682 |
Algorithms - Caesar cipher
1 passed |
100.0% |
IN8 733 OUT2 635 | |
| ✓ | CaesarCodeGenCode-gen: model implements the Caesar cipher; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT2 635 |
Algorithms - Catalan number
1 passed |
100.0% |
IN8 755 OUT1 707 | |
| ✓ | CatalanCodeGenCode-gen: model implements Catalan number; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT1 707 |
Algorithms - Chinese remainder theorem
1 failed |
0.0% |
IN8 773 OUT764 | |
| × | CrtCodeGenCode-gen: model implements CRT; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: Solve(2,3,3,5) got 2 expected 8
Values:
r.ResultJson = "2" |
Failed | IN8 773 OUT764 |
Algorithms - Circle-circle MTV
1 passed |
100.0% |
IN8 850 OUT2 735 | |
| ✓ | CircleMtvCodeGenCode-gen: model implements circle-circle MTV; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 850 OUT2 735 |
Algorithms - Coin change combinations
1 passed |
100.0% |
IN8 740 OUT2 581 | |
| ✓ | CoinCombosCodeGenCode-gen: model implements coin combinations; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 740 OUT2 581 |
Algorithms - Collatz step count
1 passed |
100.0% |
IN8 736 OUT989 | |
| ✓ | CollatzCodeGenCode-gen: model implements Collatz step count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 736 OUT989 |
Algorithms - Connected components (union-find)
1 passed |
100.0% |
IN8 752 OUT4 176 | |
| ✓ | UnionFindCodeGenCode-gen: model implements connected-component counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT4 176 |
Algorithms - Container with most water
1 passed |
100.0% |
IN8 744 OUT2 208 | |
| ✓ | ContainerWaterCodeGenCode-gen: model implements container with most water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT2 208 |
Algorithms - Count anagram groups
1 passed |
100.0% |
IN8 727 OUT2 263 | |
| ✓ | AnagramGroupsCodeGenCode-gen: model implements anagram grouping; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 727 OUT2 263 |
Algorithms - Count grid color regions
1 failed |
0.0% |
IN8 747 OUT3 189 | |
| × | CountComponentsGridCodeGenCode-gen: model implements region counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: res.ResultJson == exp
Context: Regions(1,2,aa) got 2 expected 1
Values:
res.ResultJson = "2"
exp = "1" |
Failed | IN8 747 OUT3 189 |
Algorithms - Count islands
1 passed |
100.0% |
IN8 750 OUT4 446 | |
| ✓ | CountIslandsCodeGenCode-gen: model implements island counting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 750 OUT4 446 |
Algorithms - Count overlapping occurrences
1 passed |
100.0% |
IN8 733 OUT1 464 | |
| ✓ | RabinKarpCountCodeGenCode-gen: model implements overlapping occurrence count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 733 OUT1 464 |
Algorithms - Count primes (Sieve of Eratosthenes)
1 passed |
100.0% |
IN8 712 OUT3 583 | |
| ✓ | CountPrimesCodeGenCode-gen: model implements prime counting; graded vs exact sieve oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 712 OUT3 583 |
Algorithms - Cumulative distribution
1 passed |
100.0% |
IN8 734 OUT3 306 | |
| ✓ | CdfCodeGenCode-gen: model implements CDF; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 734 OUT3 306 |
Algorithms - Dijkstra shortest path (weighted graph)
1 passed |
100.0% |
IN8 786 OUT5 271 | |
| ✓ | DijkstraCodeGenCode-gen: model implements Dijkstra shortest distance; graded against an exact oracle over fixed + random weighted graphs.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 786 OUT5 271 |
Algorithms - Directed cycle detection
1 passed |
100.0% |
IN17 952 OUT8 812 | |
| ✓ | DirectedCycleCodeGenCode-gen: model implements cycle detection; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN17 952 OUT8 812 |
Algorithms - Distinct subsequences
1 passed |
100.0% |
IN8 729 OUT2 943 | |
| ✓ | DistinctSubseqCodeGenCode-gen: model implements distinct subsequences; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 729 OUT2 943 |
Algorithms - Divisor count
1 passed |
100.0% |
IN8 717 OUT2 145 | |
| ✓ | DivisorCountCodeGenCode-gen: model implements divisor count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 717 OUT2 145 |
Algorithms - Downsample by block average
1 passed |
100.0% |
IN8 758 OUT3 589 | |
| ✓ | ResampleCodeGenCode-gen: model implements block-average downsampling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 758 OUT3 589 |
Algorithms - Dutch national flag sort
1 passed |
100.0% |
IN8 725 OUT3 555 | |
| ✓ | DutchFlagSortCodeGenCode-gen: model implements Dutch flag sort; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 725 OUT3 555 |
Algorithms - Egg drop trials
1 passed |
100.0% |
IN8 786 OUT3 153 | |
| ✓ | EggDropCodeGenCode-gen: model implements egg drop; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 786 OUT3 153 |
Algorithms - Equal subset-sum partition
1 passed |
100.0% |
IN8 728 OUT2 400 | |
| ✓ | PartitionEqualCodeGenCode-gen: model implements equal partition; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 728 OUT2 400 |
Algorithms - Euler totient
1 passed |
100.0% |
IN8 717 OUT1 589 | |
| ✓ | EulerPhiCodeGenCode-gen: model implements Euler's totient; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 717 OUT1 589 |
Algorithms - Eulerian path existence
1 passed |
100.0% |
IN8 776 OUT3 813 | |
| ✓ | EulerianPathExistsCodeGenCode-gen: model implements Eulerian-path existence; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 776 OUT3 813 |
Algorithms - Extended Euclidean
1 passed |
100.0% |
IN8 829 OUT2 484 | |
| ✓ | ExtendedGcdCodeGenCode-gen: model implements extended gcd; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 829 OUT2 484 |
Algorithms - Format duration H:MM:SS
1 passed |
100.0% |
IN8 744 OUT3 107 | |
| ✓ | FormatTimeCodeGenCode-gen: model implements duration formatting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 744 OUT3 107 |
Algorithms - GCD and LCM
1 passed |
100.0% |
IN8 721 OUT1 363 | |
| ✓ | GcdLcmCodeGenCode-gen: model implements GCD + LCM; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 721 OUT1 363 |
Algorithms - Grid transpose
1 passed |
100.0% |
IN8 752 OUT2 657 | |
| ✓ | TransposeCodeGenCode-gen: model implements grid transpose; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT2 657 |
Algorithms - House robber (max non-adjacent sum)
1 passed |
100.0% |
IN8 716 OUT1 732 | |
| ✓ | HouseRobberCodeGenCode-gen: model implements house robber; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 716 OUT1 732 |
Algorithms - Integer linear interpolation
1 passed |
100.0% |
IN8 750 OUT783 | |
| ✓ | LerpIntCodeGenCode-gen: model implements integer lerp; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 750 OUT783 |
Algorithms - Integer square root
1 passed |
100.0% |
IN8 732 OUT2 267 | |
| ✓ | IntSqrtCodeGenCode-gen: model implements integer sqrt; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 732 OUT2 267 |
Algorithms - Integer to base-N string
1 passed |
100.0% |
IN8 738 OUT2 447 | |
| ✓ | BaseConversionCodeGenCode-gen: model implements int->base string; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 738 OUT2 447 |
Algorithms - Integer to Roman numeral
1 passed |
100.0% |
IN8 767 OUT2 086 | |
| ✓ | RomanCodeGenCode-gen: model implements int->Roman; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 767 OUT2 086 |
Algorithms - Josephus survivor
1 passed |
100.0% |
IN8 743 OUT2 848 | |
| ✓ | JosephusCodeGenCode-gen: model implements Josephus survivor; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 743 OUT2 848 |
Algorithms - Jump game (min jumps)
1 passed |
100.0% |
IN8 751 OUT3 751 | |
| ✓ | JumpGameCodeGenCode-gen: model implements jump game; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 751 OUT3 751 |
Algorithms - Kth largest element
1 passed |
100.0% |
IN8 741 OUT1 529 | |
| ✓ | KthLargestCodeGenCode-gen: model implements kth largest; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 741 OUT1 529 |
Algorithms - Least-squares line fit
1 passed |
100.0% |
IN8 783 OUT3 079 | |
| ✓ | LinearRegressionCodeGenCode-gen: model implements OLS line fit; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 783 OUT3 079 |
Algorithms - Levenshtein edit distance (DP)
1 passed |
100.0% |
IN8 711 OUT3 832 | |
| ✓ | EditDistanceCodeGenCode-gen: model implements Levenshtein edit distance; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 711 OUT3 832 |
Algorithms - Log-scale bucket index
1 passed |
100.0% |
IN8 776 OUT2 861 | |
| ✓ | LogScaleBinsCodeGenCode-gen: model implements log-scale binning; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 776 OUT2 861 |
Algorithms - Logarithmic bin index
1 passed |
100.0% |
IN8 765 OUT1 966 | |
| ✓ | LogBinsCodeGenCode-gen: model implements log bins; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 765 OUT1 966 |
Algorithms - Longest common prefix
1 passed |
100.0% |
IN8 724 OUT3 031 | |
| ✓ | CommonPrefixCodeGenCode-gen: model implements longest common prefix; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 724 OUT3 031 |
Algorithms - Longest common subsequence
1 passed |
100.0% |
IN8 709 OUT3 197 | |
| ✓ | LcsCodeGenCode-gen: model implements LCS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 709 OUT3 197 |
Algorithms - Longest common substring
1 passed |
100.0% |
IN8 726 OUT2 531 | |
| ✓ | CommonSubstringCodeGenCode-gen: model implements longest common substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 726 OUT2 531 |
Algorithms - Longest increasing subsequence (DP)
1 passed |
100.0% |
IN8 743 OUT3 866 | |
| ✓ | LISCodeGenCode-gen: model implements LIS length; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 743 OUT3 866 |
Algorithms - Longest palindromic substring
1 passed |
100.0% |
IN8 721 OUT1 983 | |
| ✓ | LongestPalindromeCodeGenCode-gen: model implements longest palindromic substring length; graded vs oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 721 OUT1 983 |
Algorithms - Longest repeated substring
1 passed |
100.0% |
IN8 726 OUT2 588 | |
| ✓ | LongestRepeatedCodeGenCode-gen: model implements longest repeated substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 726 OUT2 588 |
Algorithms - Lower bound (binary search)
1 passed |
100.0% |
IN8 754 OUT3 194 | |
| ✓ | LowerBoundCodeGenCode-gen: model implements lower bound; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 754 OUT3 194 |
Algorithms - Luhn checksum validation
1 passed |
100.0% |
IN8 755 OUT2 719 | |
| ✓ | LuhnCodeGenCode-gen: model implements Luhn validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 755 OUT2 719 |
Algorithms - Majority element
1 passed |
100.0% |
IN8 719 OUT3 261 | |
| ✓ | MajorityElementCodeGenCode-gen: model implements majority element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 719 OUT3 261 |
Algorithms - Matrix chain multiplication
1 passed |
100.0% |
IN17 872 OUT8 461 | |
| ✓ | MatrixChainCodeGenCode-gen: model implements matrix-chain order; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN17 872 OUT8 461 |
Algorithms - Maximum subarray sum (Kadane)
1 passed |
100.0% |
IN8 744 OUT2 677 | |
| ✓ | MaxSubarrayCodeGenCode-gen: model implements maximum subarray sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 744 OUT2 677 |
Algorithms - Merge K sorted lists
1 passed |
100.0% |
IN8 737 OUT2 316 | |
| ✓ | MergeKSortedCodeGenCode-gen: model implements merge K sorted; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 737 OUT2 316 |
Algorithms - Merge overlapping intervals
1 passed |
100.0% |
IN8 769 OUT3 885 | |
| ✓ | MergeIntervalsCodeGenCode-gen: model implements interval merging; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT3 885 |
Algorithms - Min-max normalization
1 passed |
100.0% |
IN8 746 OUT2 563 | |
| ✓ | MinMaxScaleCodeGenCode-gen: model implements min-max scaling; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 746 OUT2 563 |
Algorithms - Minimum coin change (DP)
1 passed |
100.0% |
IN8 737 OUT1 129 | |
| ✓ | CoinChangeCodeGenCode-gen: model implements min-coin change; graded vs exact DP oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 737 OUT1 129 |
Algorithms - Minimum knight moves (BFS)
1 failed |
0.0% |
IN17 987 OUT8 117 | |
| × | KnightMovesCodeGenCode-gen: model implements min knight moves; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users MinMoves threw: Cannot cast value to 'int' because it is outside the target type range. |
Failed | IN17 987 OUT8 117 |
Algorithms - Minimum path sum (grid DP)
1 passed |
100.0% |
IN8 750 OUT1 838 | |
| ✓ | MinPathSumCodeGenCode-gen: model implements min path sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 750 OUT1 838 |
Algorithms - Minimum spanning tree (Kruskal)
1 failed |
0.0% |
IN27 912 OUT12 816 | |
| × | KruskalCodeGenCode-gen: model implements Kruskal MST; graded vs exact oracle.4 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users External function 'AskAi' threw InvalidOperationException: Connection refused (127.0.0.1:8000) |
Failed | IN27 912 OUT12 816 |
Algorithms - Minimum window substring
1 passed |
100.0% |
IN8 758 OUT4 931 | |
| ✓ | MinWindowContainsCodeGenCode-gen: model implements minimum window substring; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 758 OUT4 931 |
Algorithms - Modular exponentiation (fast pow)
1 passed |
100.0% |
IN8 769 OUT2 276 | |
| ✓ | ModPowCodeGenCode-gen: model implements modular exponentiation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 769 OUT2 276 |
Algorithms - Modular inverse (extended Euclid)
1 passed |
100.0% |
IN8 757 OUT3 552 | |
| ✓ | ModInverseCodeGenCode-gen: model implements modular inverse; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 757 OUT3 552 |
Algorithms - Moving average
1 passed |
100.0% |
IN8 748 OUT2 646 | |
| ✓ | MovingAverageCodeGenCode-gen: model implements moving average; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 748 OUT2 646 |
Algorithms - Next greater element
1 passed |
100.0% |
IN8 731 OUT2 587 | |
| ✓ | NextGreaterCodeGenCode-gen: model implements next greater element; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 731 OUT2 587 |
Algorithms - Next lexicographic permutation
1 passed |
100.0% |
IN8 739 OUT2 902 | |
| ✓ | NextPermutationCodeGenCode-gen: model implements next permutation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 739 OUT2 902 |
Algorithms - Nth Fibonacci mod m
1 passed |
100.0% |
IN8 738 OUT2 190 | |
| ✓ | FibModCodeGenCode-gen: model implements nth Fibonacci mod; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 738 OUT2 190 |
Algorithms - Overlapping substring count
1 passed |
100.0% |
IN8 723 OUT967 | |
| ✓ | SubstrCountCodeGenCode-gen: model implements substring count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 723 OUT967 |
Algorithms - Palindrome & anagram checks
1 passed |
100.0% |
IN8 753 OUT3 776 | |
| ✓ | PalindromeCodeGenCode-gen: model implements palindrome + anagram; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT3 776 |
Algorithms - Pascal triangle row
1 passed |
100.0% |
IN8 731 OUT2 359 | |
| ✓ | PascalRowCodeGenCode-gen: model implements Pascal's row; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 731 OUT2 359 |
Algorithms - Point-to-segment distance
1 passed |
100.0% |
IN8 771 OUT2 168 | |
| ✓ | PointSegmentDistanceCodeGenCode-gen: model implements point-to-segment distance; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 771 OUT2 168 |
Algorithms - Prime factorization
1 passed |
100.0% |
IN8 742 OUT2 821 | |
| ✓ | PrimeFactorizeCodeGenCode-gen: model implements prime factorization; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 742 OUT2 821 |
Algorithms - Product of array except self
1 failed |
0.0% |
IN8 748 OUT3 521 | |
| × | ProductExceptSelfCodeGenCode-gen: model implements product except self; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Products([1 2 3 4]) got "6 6 6 6" expected "24 12 8 6"
Values:
r.ResultJson = "\"6 6 6 6\""
exp = "\"24 12 8 6\"" |
Failed | IN8 748 OUT3 521 |
Algorithms - Quadratic Bezier point
1 passed |
100.0% |
IN8 808 OUT2 335 | |
| ✓ | BezierPointCodeGenCode-gen: model evaluates a quadratic Bézier; graded vs oracle within tolerance.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 808 OUT2 335 |
Algorithms - Range sum via prefix sums
1 passed |
100.0% |
IN8 730 OUT1 063 | |
| ✓ | PrefixSumRangeCodeGenCode-gen: model implements range sum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 730 OUT1 063 |
Algorithms - Reverse words & Hamming distance
1 passed |
100.0% |
IN8 734 OUT1 931 | |
| ✓ | ReverseWordsCodeGenCode-gen: model implements reverse-words + Hamming; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 734 OUT1 931 |
Algorithms - Rod cutting
1 passed |
100.0% |
IN8 764 OUT4 612 | |
| ✓ | RodCuttingCodeGenCode-gen: model implements rod cutting; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 764 OUT4 612 |
Algorithms - Roman numeral to integer
1 passed |
100.0% |
IN8 753 OUT2 872 | |
| ✓ | RomanToIntCodeGenCode-gen: model implements Roman-to-int; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 753 OUT2 872 |
Algorithms - Rotate array right by k
1 passed |
100.0% |
IN8 757 OUT2 802 | |
| ✓ | RotateArrayCodeGenCode-gen: model implements right rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 757 OUT2 802 |
Algorithms - Rotate grid 90 CW (discriminator)
1 passed |
100.0% |
IN8 760 OUT5 298 | |
| ✓ | RotateMatrixCodeGenCode-gen: model implements 90-degree CW rotation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:GameDevlevel:Hardsource:Ai4Users |
Passed | IN8 760 OUT5 298 |
Algorithms - RPN evaluator
1 passed |
100.0% |
IN8 762 OUT2 279 | |
| ✓ | RpnCodeGenCode-gen: model implements an RPN evaluator; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 762 OUT2 279 |
Algorithms - Run-length compression (if shorter)
1 passed |
100.0% |
IN17 699 OUT3 913 | |
| ✓ | StringCompressCodeGenCode-gen: model implements string compression; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN17 699 OUT3 913 |
Algorithms - Run-length encoding (encode/decode)
1 passed |
100.0% |
IN8 780 OUT2 286 | |
| ✓ | RunLengthCodeGenCode-gen: model implements RLE encode+decode; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 780 OUT2 286 |
Algorithms - Semantic versioning (compare + bump)
1 passed |
100.0% |
IN8 810 OUT2 182 | |
| ✓ | SemVerCodeGenCode-gen: model implements semver compare + bump; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:InstructionFollowinglevel:Easysource:Ai4Users |
Passed | IN8 810 OUT2 182 |
Algorithms - Single-pile Nim (optimal play)
1 passed |
100.0% |
IN8 742 OUT1 187 | |
| ✓ | NimCodeGenCode-gen: model implements optimal Nim; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 742 OUT1 187 |
Algorithms - Sliding window maximum
1 failed |
0.0% |
IN8 734 OUT4 500 | |
| × | SlidingWindowMaxCodeGenCode-gen: model implements sliding window maximum; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users WindowMax threw: List index -1 is out of range. The list has 0 element(s). |
Failed | IN8 734 OUT4 500 |
Algorithms - Spiral matrix traversal
1 failed |
0.0% |
IN8 743 OUT5 449 | |
| × | SpiralMatrixCodeGenCode-gen: model implements spiral traversal; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp
Context: Spiral 3x3 [abcdefghi] got "abcfihgda" expected "abcfihgde"
Values:
r.ResultJson = "\"abcfihgda\""
exp = "\"abcfihgde\"" |
Failed | IN8 743 OUT5 449 |
Algorithms - Square matrix multiply
1 passed |
100.0% |
IN8 735 OUT3 139 | |
| ✓ | MatMulCodeGenCode-gen: model implements square matrix multiply; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN8 735 OUT3 139 |
Algorithms - Strongly connected components
1 passed |
100.0% |
IN8 746 OUT5 068 | |
| ✓ | SccCountCodeGenCode-gen: model implements SCC count; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:VeryHardsource:Ai4Users |
Passed | IN8 746 OUT5 068 |
Algorithms - Subset sum reachability
1 passed |
100.0% |
IN17 722 OUT5 412 | |
| ✓ | SubsetSumCodeGenCode-gen: model implements subset-sum reachability; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Mediumsource:Ai4Users |
Passed | IN17 722 OUT5 412 |
Algorithms - Substring search (KMP)
1 passed |
100.0% |
IN8 741 OUT3 613 | |
| ✓ | KmpCodeGenCode-gen: model implements KMP search; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 741 OUT3 613 |
Algorithms - Sum of digits & digital root
1 passed |
100.0% |
IN8 720 OUT968 | |
| ✓ | DigitalRootCodeGenCode-gen: model implements sum-of-digits + digital root; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 720 OUT968 |
Algorithms - Temperature conversion
1 passed |
100.0% |
IN8 731 OUT554 | |
| ✓ | TemperatureCodeGenCode-gen: model implements temperature conversion; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 731 OUT554 |
Algorithms - Topological sort (lexicographically smallest)
1 passed |
100.0% |
IN8 772 OUT4 455 | |
| ✓ | TopoCodeGenCode-gen: model implements lexicographically-smallest topological sort.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 772 OUT4 455 |
Algorithms - Trailing zeros of factorial
1 passed |
100.0% |
IN8 732 OUT1 530 | |
| ✓ | TrailingZerosCodeGenCode-gen: model implements trailing zeros; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 732 OUT1 530 |
Algorithms - Trapping rain water
1 failed |
0.0% |
IN8 731 OUT2 589 | |
| × | TrappingRainCodeGenCode-gen: model implements trapping rain water; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users Assertion expression evaluated to false: r.ResultJson == exp.ToString()
Context: Trapped([0 1 0 2 1 0 1 3 2 1 2 1]) got 11 expected 6
Values:
r.ResultJson = "11" |
Failed | IN8 731 OUT2 589 |
Algorithms - Two-sum indices
1 passed |
100.0% |
IN17 723 OUT7 370 | |
| ✓ | TwoSumCodeGenCode-gen: model implements two-sum; graded vs exact oracle.2 AI queries
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN17 723 OUT7 370 |
Algorithms - Unique grid paths
1 passed |
100.0% |
IN8 713 OUT1 558 | |
| ✓ | UniquePathsCodeGenCode-gen: model implements unique paths; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 713 OUT1 558 |
Algorithms - Valid Sudoku unit
1 passed |
100.0% |
IN8 718 OUT2 436 | |
| ✓ | SudokuUnitCodeGenCode-gen: model implements Sudoku unit validation; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Easysource:Ai4Users |
Passed | IN8 718 OUT2 436 |
Algorithms - Wildcard string matching
1 passed |
100.0% |
IN8 736 OUT2 947 | |
| ✓ | WildcardMatchCodeGenCode-gen: model implements wildcard match; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users |
Passed | IN8 736 OUT2 947 |
Algorithms - Z-function
1 failed |
0.0% |
IN8 737 OUT3 563 | |
| × | ZFunctionCodeGenCode-gen: model implements the Z-function; graded vs exact oracle.1 AI query
0 tool callscategory:Algorithmscategory:CodeGenerationlevel:Hardsource:Ai4Users External function 'RunScript' threw InvalidOperationException: ScSharp execution failed unexpectedly. This is a bug in the ScSharp library. |
Failed | IN8 737 OUT3 563 |
Database - Conditional aggregation pivot
1 passed |
100.0% |
IN945 OUT527 | |
| ✓ | PivotCountsCodeGenCode-gen: model writes the pivot query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN945 OUT527 |
Database - Count distinct opponents
1 passed |
100.0% |
IN925 OUT369 | |
| ✓ | DistinctCountCodeGenCode-gen: model writes the COUNT(DISTINCT) query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Easysource:Ai4Users |
Passed | IN925 OUT369 |
Database - Date-range filter (BETWEEN)
1 passed |
100.0% |
IN955 OUT443 | |
| ✓ | DateRangeCodeGenCode-gen: model writes the date-range query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN955 OUT443 |
Database - EXISTS subquery (teams with matches)
1 passed |
100.0% |
IN932 OUT757 | |
| ✓ | ExistsSubqueryCodeGenCode-gen: model writes the EXISTS query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN932 OUT757 |
Database - JOIN player totals (two tables)
1 passed |
100.0% |
IN954 OUT685 | |
| ✓ | JoinScoresCodeGenCode-gen: model writes a JOIN + aggregate query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN954 OUT685 |
Database - LEFT JOIN anti-join (no items)
1 passed |
100.0% |
IN928 OUT446 | |
| ✓ | LeftJoinNullsCodeGenCode-gen: model writes the anti-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN928 OUT446 |
Database - Player stats SQL (multi-aggregate, QueryJson)
1 passed |
100.0% |
IN958 OUT664 | |
| ✓ | PlayerStatsCodeGenCode-gen: model writes a multi-aggregate stats query; graded by comparing its QueryJson output to the oracle query's (arbitrary shape, no struct).1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN958 OUT664 |
Database - Players above average score (subquery)
1 passed |
100.0% |
IN923 OUT332 | |
| ✓ | AboveAverageCodeGenCode-gen: model writes an above-average query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN923 OUT332 |
Database - Players who never played (anti-join)
1 passed |
100.0% |
IN920 OUT201 | |
| ✓ | NeverPlayedCodeGenCode-gen: model writes an anti-join query; graded via QueryJson vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN920 OUT201 |
Database - RANK window function
1 passed |
100.0% |
IN950 OUT546 | |
| ✓ | WindowRankCodeGenCode-gen: model writes the RANK() query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN950 OUT546 |
Database - Running total (window function)
1 passed |
100.0% |
IN947 OUT369 | |
| ✓ | RunningTotalCodeGenCode-gen: model writes the window running-total query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN947 OUT369 |
Database - Scoreboard SQL (players with >1 win, HAVING)
1 passed |
100.0% |
IN954 OUT334 | |
| ✓ | WinFilterCodeGenCode-gen: model writes a HAVING query; runs under try/catch with self-repair; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN954 OUT334 |
Database - Scoreboard SQL (top scorers query)
1 passed |
100.0% |
IN955 OUT328 | |
| ✓ | ScoreboardTopScorersCodeGenCode-gen: model writes a scoreboard SQL query; runs under try/catch with a self-repair loop on SQL errors; result compared to the oracle query.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN955 OUT328 |
Database - Self-join same-team pairs
1 passed |
100.0% |
IN938 OUT469 | |
| ✓ | SelfJoinCodeGenCode-gen: model writes the self-join query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Mediumsource:Ai4Users |
Passed | IN938 OUT469 |
Database - Top scorer per region
1 passed |
100.0% |
IN940 OUT437 | |
| ✓ | TopPerGroupCodeGenCode-gen: model writes the per-group top query; compared to the oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN940 OUT437 |
Database - UPDATE restock (SQL write)
1 passed |
100.0% |
IN932 OUT192 | |
| ✓ | UpdateInventoryCodeGenCode-gen: model writes an UPDATE; applied under try/catch, resulting state compared (via QueryJson) to the oracle update's state.1 AI query
0 tool callscategory:CodeGenerationcategory:Databasecategory:GameDevlevel:Mediumsource:Ai4Users |
Passed | IN932 OUT192 |
Database - Upsert with max merge
1 passed |
100.0% |
IN948 OUT700 | |
| ✓ | UpsertCodeGenCode-gen: model writes the upsert; executed on a fresh DB and the table compared.1 AI query
0 tool callscategory:CodeGenerationcategory:Databaselevel:Hardsource:Ai4Users |
Passed | IN948 OUT700 |
SDLC - Acceptance criteria check
1 passed |
100.0% |
IN8 761 OUT2 988 | |
| ✓ | SdlcAcceptanceCodeGenCode-gen: model implements acceptance-criteria check; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 761 OUT2 988 |
SDLC - Breaking API change detection
1 passed |
100.0% |
IN8 758 OUT3 441 | |
| ✓ | SdlcApiBreakingCodeGenCode-gen: model implements API-breaking detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 758 OUT3 441 |
SDLC - Condense without losing facts
1 passed |
100.0% |
IN1 046 OUT1 228 | |
| ✓ | NoInfoLossCodeGenRedaction test: model condenses keeping facts; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 046 OUT1 228 |
SDLC - Config merge with override
1 passed |
100.0% |
IN8 752 OUT3 947 | |
| ✓ | ConfigMergeCodeGenCode-gen: model implements config merge; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 752 OUT3 947 |
SDLC - Extract fields into fixed template
1 passed |
100.0% |
IN1 002 OUT335 | |
| ✓ | FieldExtractCodeGenRedaction test: model extracts fields into the template; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 002 OUT335 |
SDLC - Locate conflicting requirement pair
1 failed |
0.0% |
IN18 223 OUT9 896 | |
| × | SdlcConstraintConflictCodeGenCode-gen: model locates the conflicting requirement pair; graded vs exact oracle.3 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users External function 'AskAi' threw InvalidOperationException: Response status code does not indicate success: 500 (Internal Server Error). |
Failed | IN18 223 OUT9 896 |
SDLC - Locate redundant requirement
1 failed |
0.0% |
IN18 224 OUT9 873 | |
| × | SdlcRedundantRequirementCodeGenCode-gen: model locates a redundant requirement; graded vs exact oracle.3 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users External function 'AskAi' threw InvalidOperationException: Connection refused (127.0.0.1:8000) |
Failed | IN18 224 OUT9 873 |
SDLC - Mask PII to placeholder
1 passed |
100.0% |
IN970 OUT290 | |
| ✓ | MaskPiiCodeGenRedaction test: model masks PII to a placeholder; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN970 OUT290 |
SDLC - Modules on a dependency cycle
1 failed |
0.0% |
IN18 081 OUT9 830 | |
| × | DependencyCycleModulesCodeGenCode-gen: model implements dependency-cycle detection; graded vs exact oracle.3 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Hardsource:Ai4Users External function 'AskAi' threw InvalidOperationException: Connection refused (127.0.0.1:8000) |
Failed | IN18 081 OUT9 830 |
SDLC - Remove duplicate requirements
1 passed |
100.0% |
IN943 OUT280 | |
| ✓ | DeduplicateCodeGenRedaction test: model removes duplicate requirements; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN943 OUT280 |
SDLC - Renumber requirements sequentially
1 passed |
100.0% |
IN1 010 OUT1 568 | |
| ✓ | RenumberCodeGenRedaction test: model renumbers sequentially; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 010 OUT1 568 |
SDLC - Reorganize notes into section template
1 passed |
100.0% |
IN1 124 OUT444 | |
| ✓ | RewriteSectionsCodeGenRedaction test: model reorganizes the notes; deterministic checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 124 OUT444 |
SDLC - Required semver bump from changelog
1 passed |
100.0% |
IN8 757 OUT2 276 | |
| ✓ | SemverBumpCodeGenCode-gen: model implements the semver-bump rule; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Easysource:Ai4Users |
Passed | IN8 757 OUT2 276 |
SDLC - Requirement-to-test traceability
1 passed |
100.0% |
IN8 779 OUT2 672 | |
| ✓ | SdlcTraceabilityCodeGenCode-gen: model implements traceability gap; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 779 OUT2 672 |
SDLC - Requirements completeness (spec coverage gap)
1 passed |
100.0% |
IN8 774 OUT3 934 | |
| ✓ | SdlcSpecCoverageCodeGenCode-gen: model implements spec-gap detection; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 774 OUT3 934 |
SDLC - Requirements consistency (contradiction detection)
1 passed |
100.0% |
IN8 759 OUT4 864 | |
| ✓ | ConsistencyCodeGenCode-gen: model implements requirements-consistency checking; graded vs oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 759 OUT4 864 |
SDLC - Reshape prose into JSON template
1 passed |
100.0% |
IN1 012 OUT358 | |
| ✓ | JsonShapeCodeGenRedaction test: model reshapes prose into JSON; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN1 012 OUT358 |
SDLC - Reshape rows into Markdown table
1 passed |
100.0% |
IN966 OUT669 | |
| ✓ | TableFormatCodeGenRedaction test: model builds a Markdown table; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN966 OUT669 |
SDLC - Rewrite in imperative requirement style
1 passed |
100.0% |
IN989 OUT456 | |
| ✓ | StyleRewriteCodeGenRedaction test: model rewrites notes in imperative style; checker + self-review loop.1 AI query
0 tool callscategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN989 OUT456 |
SDLC - Roll up estimates over task tree
1 passed |
100.0% |
IN17 847 OUT6 802 | |
| ✓ | EstimateRollupCodeGenCode-gen: model implements estimate roll-up over a task tree; graded vs exact oracle.2 AI queries
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN17 847 OUT6 802 |
SDLC - State-machine reachability
1 passed |
100.0% |
IN8 756 OUT5 674 | |
| ✓ | SdlcStateReachableCodeGenCode-gen: model implements state reachability; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 756 OUT5 674 |
SDLC - Ticket priority sort
1 passed |
100.0% |
IN8 749 OUT2 852 | |
| ✓ | SdlcPrioritySortCodeGenCode-gen: model implements priority sort; graded vs exact oracle.1 AI query
0 tool callscategory:CodeGenerationcategory:Reasoninglevel:Mediumsource:Ai4Users |
Passed | IN8 749 OUT2 852 |
SDLC - Unify glossary term consistency
1 passed |
100.0% |
IN970 OUT494 | |
| ✓ | GlossaryCodeGenRedaction test: model unifies synonyms to the canonical term; checker + self-review loop.1 AI query
0 tool callscategory:ContextProcessingcategory:InstructionFollowinglevel:Mediumsource:Ai4Users |
Passed | IN970 OUT494 |