Benchmark summary
The model builds a Mario-like scrolling platformer in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Building Games in SC#: Mario-like platformer
14 failed1 broken |
0.0% |
IN20 997 OUT62 144 | |
| ! | PlatformerThe model builds a scrolling platform game from the task as an agent; this item passes when the app compiles and can be exported, and each hidden test is its own item.1 AI query
0 tool calls Assertion expression evaluated to false: app.Built
Context: AI: AI output was marked broken by AiRunner. The response text is partial and must not be treated as a complete answer.
App: not submitted; the last file version is graded (end: AI error: AI output was marked broken by AiRunner. The response text is partial and must not be treated as a complete answer.)
Work: 1 AI calls (1 model rounds with tool calls), 0 whole-file writes, 0 edits (0 lines), 0 CheckApp calls, 0 refused rewrites, 0 refused repeated edits, 0 rejected submissions, 0 reminders to hand in
Compile errors: (1,1) Unknown type 'SumResult'. Supported types: bool, byte, int, decimal, double, char, string, Date, DateTime, TimeSpan, List<...>, Dictionary<...>, HashSet<...> and declared structs. (8,12) Unknown struct 'SumResult'.
Values:
app.Built = false |
Broken | IN20 997 OUT62 144 |
| × | Hidden_BricksA big player breaks a brick from below for 50 points; a small player only bumps it.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_CameraThe camera follows the player's middle minus 160, never moves back, stops at 640, and the player cannot walk left out of the view.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_CoinBlockJumping into a question block from below uses it: a coin block gives a coin and 200 points.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_CoinsTouching a coin collects it for 200 points.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_EnemiesWalkEnemies walk at a constant speed of 0.5 to 1 px per frame, turn at walls and at the edge of a pit, and never fall or enter solid tiles.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_EnemyHitsTouching an enemy from the side costs a small player a life and makes a big player small for 60 safe frames.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_FlagTouching the flag wins: the game stops, and Enter starts a new game.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_InitialStateAfter AppInit the level is the given map, the coins and enemies come from it, and the player and counters have their starting values.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_JumpingA jump starts with -7, gravity adds 0.5 per frame; holding Up gives the full jump, releasing it early a lower one.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_MushroomA mushroom block lets a mushroom out on top; the mushroom moves at a constant speed and falls to the ground without entering solid tiles; eating it makes the player big.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_PitsAndGameOverFalling into a pit costs a life; with the last life the game is over, and Enter starts a new game.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_ScreenThe screen shows the tiles, the enemies, the player and the coins at their positions relative to the camera.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_StompFalling onto an enemy stomps it: it is removed, 100 points, and the player bounces up with -4.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_WalkingLeft and Right move the player by 2 px per frame; walls stop the player at their edge.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |