Benchmark summary
The model builds an 8-ball pool game, seen from the top in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Building Games in SC#: Billiards
11 passed6 failed |
64.7% |
IN698 210 OUT9 044 | |
| ✓ | BilliardsThe model builds a billiards game from the task as an agent; this item passes when the app compiles and can be exported, and each hidden test is its own item.1 AI query
19 tool calls |
Passed | IN698 210 OUT9 044 |
| ✓ | Hidden_AimAndPowerLeft and Right turn the aim by 2 degrees per frame and wrap around; Up and Down change the power from 1 to 10.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| ✓ | Hidden_AngledCollisionIn an angled hit both balls move apart along the line of their centers at the contact, as the task's collision rule says.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| ✓ | Hidden_BreakThe break with full power ends after a while, with every ball on the table at rest on the cloth.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| ✓ | Hidden_CushionBounceA ball bounces off a cushion: the position is mirrored and the speed across the cushion drops to 0.8 of it.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| ✓ | Hidden_EightBallLossPocketing the 8-ball too early, without a group, or together with the cue ball loses the game.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| ✓ | Hidden_EightBallWinPocketing the 8-ball after the own group is cleared wins, also together with the last ball of the group.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| ✓ | Hidden_HeadOnCollisionA head-on hit stops the cue ball and sends the other ball on with the cue ball's speed.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| ✓ | Hidden_InitialStateAfter AppInit the balls form the rack, the cue ball is on its spot and nobody has a group.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| ✓ | Hidden_NewGameEnter after the game is over starts a new game, and the frame of Enter does nothing else.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| × | Hidden_PocketAndGroupsThe first ball pocketed by a player without a group gives that player its group; a ball of the own group lets the player shoot again.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: playerGroup[0] == 1 && playerGroup[1] == 2
Context: player 1 gets the solids: groups [0, 0]
Values:
playerGroup[0] = 0
playerGroup[1] = 0 (line 924) |
Failed | IN0 OUT0 |
| × | Hidden_PocketBeforeCushionA ball that passes the cushion line close to a pocket falls in, because pockets are checked before cushions.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: ballOnTable[0] && Hidden_Near(ballX[0], 90) && Hidden_Near(ballY[0], 120) && currentPlayer == 1
Context: the cue ball fell and came back: (168, 34.360190244274484), player 0
Values:
ballOnTable[0] = false
currentPlayer = 0 (line 1002) |
Failed | IN0 OUT0 |
| × | Hidden_RollAndStopA lone ball rolls straight, slows down by friction and stops; the shot ends in the frame the ball stops, and the turn passes.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: currentPlayer == 1
Context: nothing was pocketed, so the turn passes
Values:
currentPlayer = 0 (line 858) |
Failed | IN0 OUT0 |
| × | Hidden_ScratchWhen the cue ball falls into a pocket, it comes back at (90, 120), or 12 further left for every time that spot is taken, and the turn passes.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: ballOnTable[0] && ballVX[0] == 0 && ballVY[0] == 0
Context: the cue ball is back on the table at rest
Values:
ballOnTable[0] = false
ballVX[0] = 0
ballVY[0] = 0 (line 972) |
Failed | IN0 OUT0 |
| × | Hidden_ScreenThe screen shows the cloth, the 6 pockets and the balls in their colors at their places, with the spots of the stripes.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: hiddenRed.Found && hiddenRed.X == 244 && hiddenRed.Y == 74 && hiddenRed.Width == 12 && hiddenRed.Height == 12
Context: ball 3: (21, 17) 235 x 69
Values:
hiddenRed.X = 21
hiddenRed.Found = true
hiddenRed.Y = 17
hiddenRed.Width = 235
hiddenRed.Height = 69 (line 1123) |
Failed | IN0 OUT0 |
| ✓ | Hidden_ShotStartSpace gives the cue ball the speed power in the aim direction; the ball first moves in the next frame, and the keys do nothing until the balls stop.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| × | Hidden_TurnRulesWith groups given, the turn passes after a miss or after pocketing only the other player's ball, and stays after pocketing an own ball.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: currentPlayer == 1
Context: pocketing only the opponent's ball passes the turn
Values:
currentPlayer = 0 (line 946) |
Failed | IN0 OUT0 |