Benchmark summary
The model builds a Mario-like scrolling platformer in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Building Games in SC#: Mario-like platformer
7 passed8 failed |
46.7% |
IN3 324 006 OUT33 155 | |
| ✓ | PlatformerThe model builds a scrolling platform game from the task as an agent; this item passes when the app compiles and can be exported, and each hidden test is its own item.1 AI query
55 tool calls |
Passed | IN3 324 006 OUT33 155 |
| × | Hidden_BricksA big player breaks a brick from below for 50 points; a small player only bumps it.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Hidden_At(10, 9) == 2 && score == 0 && playerY == 192
Context: a small player does not break the brick
Values:
score = 0
playerY = 200 (line 1137) |
Failed | IN0 OUT0 |
| ✓ | Hidden_CameraThe camera follows the player's middle minus 160, never moves back, stops at 640, and the player cannot walk left out of the view.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| × | Hidden_CoinBlockJumping into a question block from below uses it: a coin block gives a coin and 200 points.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: playerY == 160 && playerVY == 0
Context: the head stops at the block: y 157, vy 0
Values:
playerY = 157
playerVY = 0 (line 1071) |
Failed | IN0 OUT0 |
| ✓ | Hidden_CoinsTouching a coin collects it for 200 points.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| ✓ | Hidden_EnemiesWalkEnemies walk at a constant speed of 0.5 to 1 px per frame, turn at walls and at the edge of a pit, and never fall or enter solid tiles.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| × | Hidden_EnemyHitsTouching an enemy from the side costs a small player a life and makes a big player small for 60 safe frames.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: !playerBig && playerY == 192 && hurtFrames == 59 && lives == 3
Context: big false, y 195.5, hurtFrames 59, lives 3
Values:
playerY = 195.5
playerBig = false
hurtFrames = 59
lives = 3 (line 1214) |
Failed | IN0 OUT0 |
| ✓ | Hidden_FlagTouching the flag wins: the game stops, and Enter starts a new game.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| × | Hidden_InitialStateAfter AppInit the level is the given map, the coins and enemies come from it, and the player and counters have their starting values.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: level[ty * 60 + tx] == v
Context: tile (11, 9) is 2, expected 0
Values:
v = 0 (line 994) |
Failed | IN0 OUT0 |
| × | Hidden_JumpingA jump starts with -7, gravity adds 0.5 per frame; holding Up gives the full jump, releasing it early a lower one.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Hidden_Near(top, 178) && playerY == 192 && onGround && playerVY == 0
Context: the low jump reached 178 and landed at 209.5
Values:
playerY = 209.5
onGround = true
playerVY = 0 (line 1051) |
Failed | IN0 OUT0 |
| × | Hidden_MushroomA mushroom block lets a mushroom out on top; the mushroom moves at a constant speed and falls to the ground without entering solid tiles; eating it makes the player big.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: !mushroomActive && playerBig && playerY == 176 && score == 1000
Context: eaten: big true, y 176.5, score 1000
Values:
playerY = 176.5
mushroomActive = false
playerBig = true
score = 1000 (line 1122) |
Failed | IN0 OUT0 |
| ✓ | Hidden_PitsAndGameOverFalling into a pit costs a life; with the last life the game is over, and Enter starts a new game.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| × | Hidden_ScreenThe screen shows the tiles, the enemies, the player and the coins at their positions relative to the camera.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: pipe.R == 0x00 && pipe.G == 0xA8 && pipe.B == 0x00
Context: the pipe at world x 480 on screen at 180
Values:
pipe.R = 92
pipe.G = 148
pipe.B = 252 (line 1327) |
Failed | IN0 OUT0 |
| ✓ | Hidden_StompFalling onto an enemy stomps it: it is removed, 100 points, and the player bounces up with -4.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| × | Hidden_WalkingLeft and Right move the player by 2 px per frame; walls stop the player at their edge.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: playerX == 52 && playerY == 192 && onGround
Context: walked to 52, y 197
Values:
playerX = 52
playerY = 197
onGround = true (line 1021) |
Failed | IN0 OUT0 |