Benchmark summary
The model builds an 8-ball pool game, seen from the top in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Building Games in SC#: Billiards
16 failed1 broken |
0.0% |
IN161 086 OUT66 559 | |
| ! | BilliardsThe model builds a billiards game from the task as an agent; this item passes when the app compiles and can be exported, and each hidden test is its own item.1 AI query
5 tool calls Assertion expression evaluated to false: app.Built
Context: AI: AI output was marked broken by AiRunner. The response text is partial and must not be treated as a complete answer.
App: not submitted; the last file version is graded (end: AI error: AI output was marked broken by AiRunner. The response text is partial and must not be treated as a complete answer.)
Work: 1 AI calls (6 model rounds with tool calls), 1 whole-file writes, 2 edits (5 lines), 1 CheckApp calls, 0 refused rewrites, 0 refused repeated edits, 0 rejected submissions, 0 reminders to hand in
Compile errors: (101,37) Operator '+' cannot be applied to 'string' and 'int'. It supports numbers and string/char concatenation. (102,31) Operator '+' cannot be applied to 'string' and 'int'. It supports numbers and string/char concatenation. (111,38) Operator '+' cannot be applied to 'string' and 'int'. It supports numbers and string/char concatenation.
Values:
app.Built = false |
Broken | IN161 086 OUT66 559 |
| × | Hidden_AimAndPowerLeft and Right turn the aim by 2 degrees per frame and wrap around; Up and Down change the power from 1 to 10.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_AngledCollisionIn an angled hit both balls move apart along the line of their centers at the contact, as the task's collision rule says.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_BreakThe break with full power ends after a while, with every ball on the table at rest on the cloth.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_CushionBounceA ball bounces off a cushion: the position is mirrored and the speed across the cushion drops to 0.8 of it.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_EightBallLossPocketing the 8-ball too early, without a group, or together with the cue ball loses the game.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_EightBallWinPocketing the 8-ball after the own group is cleared wins, also together with the last ball of the group.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_HeadOnCollisionA head-on hit stops the cue ball and sends the other ball on with the cue ball's speed.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_InitialStateAfter AppInit the balls form the rack, the cue ball is on its spot and nobody has a group.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_NewGameEnter after the game is over starts a new game, and the frame of Enter does nothing else.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_PocketAndGroupsThe first ball pocketed by a player without a group gives that player its group; a ball of the own group lets the player shoot again.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_PocketBeforeCushionA ball that passes the cushion line close to a pocket falls in, because pockets are checked before cushions.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_RollAndStopA lone ball rolls straight, slows down by friction and stops; the shot ends in the frame the ball stops, and the turn passes.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_ScratchWhen the cue ball falls into a pocket, it comes back at (90, 120), or 12 further left for every time that spot is taken, and the turn passes.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_ScreenThe screen shows the cloth, the 6 pockets and the balls in their colors at their places, with the spots of the stripes.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_ShotStartSpace gives the cue ball the speed power in the aim direction; the ball first moves in the next frame, and the keys do nothing until the balls stop.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_TurnRulesWith groups given, the turn passes after a miss or after pocketing only the other player's ball, and stays after pocketing an own ball.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |