Benchmark summary
The model builds an 8-ball pool game, seen from the top in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Building Games in SC#: Billiards
17 broken |
0.0% |
IN21 212 OUT4 495 | |
| ! | BilliardsThe model builds a billiards game from the task as an agent; this item passes when the app compiles and can be exported, and each hidden test is its own item.1 AI query
0 tool calls Assertion expression evaluated to false: app.Built
Context: AI: AI output was marked broken by AiRunner. The response text is partial and must not be treated as a complete answer.
App: none handed in (end: AI error: AI output was marked broken by AiRunner. The response text is partial and must not be treated as a complete answer.)
Work: 1 AI calls (1 model rounds with tool calls), 0 whole-file writes, 0 edits (0 lines), 0 CheckApp calls, 0 refused rewrites, 0 refused repeated edits, 0 rejected submissions, 0 reminders to hand in
Values:
app.Built = false |
Broken | IN21 212 OUT4 495 |
| ! | Hidden_AimAndPowerLeft and Right turn the aim by 2 degrees per frame and wrap around; Up and Down change the power from 1 to 10.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_AngledCollisionIn an angled hit both balls move apart along the line of their centers at the contact, as the task's collision rule says.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_BreakThe break with full power ends after a while, with every ball on the table at rest on the cloth.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_CushionBounceA ball bounces off a cushion: the position is mirrored and the speed across the cushion drops to 0.8 of it.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_EightBallLossPocketing the 8-ball too early, without a group, or together with the cue ball loses the game.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_EightBallWinPocketing the 8-ball after the own group is cleared wins, also together with the last ball of the group.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_HeadOnCollisionA head-on hit stops the cue ball and sends the other ball on with the cue ball's speed.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_InitialStateAfter AppInit the balls form the rack, the cue ball is on its spot and nobody has a group.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_NewGameEnter after the game is over starts a new game, and the frame of Enter does nothing else.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_PocketAndGroupsThe first ball pocketed by a player without a group gives that player its group; a ball of the own group lets the player shoot again.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_PocketBeforeCushionA ball that passes the cushion line close to a pocket falls in, because pockets are checked before cushions.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_RollAndStopA lone ball rolls straight, slows down by friction and stops; the shot ends in the frame the ball stops, and the turn passes.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_ScratchWhen the cue ball falls into a pocket, it comes back at (90, 120), or 12 further left for every time that spot is taken, and the turn passes.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_ScreenThe screen shows the cloth, the 6 pockets and the balls in their colors at their places, with the spots of the stripes.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_ShotStartSpace gives the cue ball the speed power in the aim direction; the ball first moves in the next frame, and the keys do nothing until the balls stop.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |
| ! | Hidden_TurnRulesWith groups given, the turn passes after a miss or after pocketing only the other player's ball, and stays after pocketing an own ball.0 AI queries
0 tool callsHiddenTest Not run: the model's answer was cut off before an app existed (AiRunner marked it broken: a loop or the output token limit). |
Broken | IN0 OUT0 |