Benchmark summary
The model builds a bubble shooter game in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Building Games in SC#: Bubble shooter
18 broken |
0.0% |
IN20 728 OUT13 710 | |
| ! | PuzzleBobbleThe model builds a Puzzle Bobble game from the task as an agent; this item passes when the app compiles and can be exported, and each hidden test is its own item.1 AI query
0 tool calls Assertion expression evaluated to false: app.Built
Context: AI: AI output was marked broken by AiRunner. The response text is partial and must not be treated as a complete answer.
App: none handed in (end: AI error: AI output was marked broken by AiRunner. The response text is partial and must not be treated as a complete answer.)
Work: 1 AI calls (1 model rounds with tool calls), 0 whole-file writes, 0 edits (0 lines), 0 CheckApp calls, 0 refused rewrites, 0 refused repeated edits, 0 rejected submissions, 0 reminders to hand in
Values:
app.Built = false |
Broken | IN20 728 OUT13 710 |
| ! | Hidden_AimingLeft and Right turn the launcher by 2 degrees per frame while held, within 15-165 degrees.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_AngledShotsShots at an angle reach the ceiling in the cell nearest to where they stop.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_CeilingAndTiesA bubble that reaches the ceiling goes into the nearest empty cell; on a tie the smaller row, then the smaller column wins.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_CeilingDropAfter every 8th shot that does not clear the field the ceiling drops by one row; the field moves down 14 pixels and the shots reach the new ceiling sooner.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_FallingAfter a pop, bubbles that no longer hang from row 0 fall for 20 points each; bubbles still connected to row 0 stay, whatever their color.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_GameOverWhen a bubble's center reaches y 192 the game is over; then only Enter works, and it starts a new game in a frame that does nothing else.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_GameOverByBubbleA bubble that attaches in a row with center y 192 or more ends the game.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_HitBubblesA bubble that hits grid bubbles goes into the nearest empty cell.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_InitialStateAfter AppInit, level 1 is loaded from its map, the launcher points up and its two colors are on the field.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_LevelClearWhen the field is empty after a shot, the level is clear and the shot counter stays; Enter starts the next level and keeps the score.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_NextColorThe next bubble always gets a color that is on the field when the shot starts.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_OddRowNeighborsNeighbors on odd rows are shifted: (1, 3) touches (2, 3) and (2, 4), and (3, 3) touches (2, 3) and (2, 4).0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_PopThree or more connected bubbles of the shot's color pop for 10 points each; two stay.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_ScreenThe screen shows the field, the lowered ceiling, the deadline, the bubbles at their cells, the launcher, the next bubble, the aim and the texts.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_ShotSpace starts a shot from (160, 220) that moves 8 pixels per frame from the next frame on; the launcher colors move on, and Space does nothing while a bubble flies.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_StopDistanceA bubble stops next to a grid bubble only when it comes closer than 14 pixels; passing at 14-16 pixels does not stop it.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_WallBounceThe side walls reflect the bubble; the bubble stays between x = 104 and x = 216.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |