Benchmark summary
The model builds an 8-ball pool game, seen from the top in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Building Games in SC#: Billiards
4 passed13 failed |
23.5% |
IN1 709 158 OUT80 516 | |
| ✓ | BilliardsThe model builds a billiards game from the task as an agent; this item passes when the app compiles and can be exported, and each hidden test is its own item.4 AI queries
26 tool calls |
Passed | IN1 709 158 OUT80 516 |
| ✓ | Hidden_AimAndPowerLeft and Right turn the aim by 2 degrees per frame and wrap around; Up and Down change the power from 1 to 10.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| × | Hidden_AngledCollisionIn an angled hit both balls move apart along the line of their centers at the contact, as the task's collision rule says.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: ballVY[0] < 0 && ballVY[5] > 0 && ballVX[5] > 0
Context: after a hit below the middle the cue ball goes up and ball 5 down: cue (5.10, 0.00), ball 5 (0.00, 0.00)
Values:
ballVY[0] = 0
ballVY[5] = 0
ballVX[5] = 0 (line 719) |
Failed | IN0 OUT0 |
| ✓ | Hidden_BreakThe break with full power ends after a while, with every ball on the table at rest on the cloth.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| × | Hidden_CushionBounceA ball bounces off a cushion: the position is mirrored and the speed across the cushion drops to 0.8 of it.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Math.Abs(ballX[hiddenI] - hiddenX[hiddenI]) < 0.01 && Math.Abs(ballY[hiddenI] - hiddenY[hiddenI]) < 0.01
Context: straight into the left cushion, frame 1: ball 0 at (58.750, 120.000), expected (55.000, 120.000) (line 565) |
Failed | IN0 OUT0 |
| × | Hidden_EightBallLossPocketing the 8-ball too early, without a group, or together with the cue ball loses the game.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Math.Abs(ballX[hiddenI] - hiddenX[hiddenI]) < 0.01 && Math.Abs(ballY[hiddenI] - hiddenY[hiddenI]) < 0.01
Context: the 8-ball while ball 3 is on the table, frame 1: ball 0 at (160.000, 118.000), expected (160.000, 112.000) (line 565) |
Failed | IN0 OUT0 |
| × | Hidden_EightBallWinPocketing the 8-ball after the own group is cleared wins, also together with the last ball of the group.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Math.Abs(ballX[hiddenI] - hiddenX[hiddenI]) < 0.01 && Math.Abs(ballY[hiddenI] - hiddenY[hiddenI]) < 0.01
Context: the 8-ball into the top middle pocket, frame 1: ball 0 at (160.000, 118.000), expected (160.000, 112.000) (line 565) |
Failed | IN0 OUT0 |
| × | Hidden_HeadOnCollisionA head-on hit stops the cue ball and sends the other ball on with the cue ball's speed.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: ballVX[0] == 0 && ballVY[0] == 0
Context: the cue ball stops after a head-on hit: (5.315054285183999, 0)
Values:
ballVX[0] = 5.315054285183999
ballVY[0] = 0 (line 704) |
Failed | IN0 OUT0 |
| ✓ | Hidden_InitialStateAfter AppInit the balls form the rack, the cue ball is on its spot and nobody has a group.0 AI queries
0 tool callsHiddenTest |
Passed | IN0 OUT0 |
| × | Hidden_NewGameEnter after the game is over starts a new game, and the frame of Enter does nothing else.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Math.Abs(ballX[hiddenI] - hiddenX[hiddenI]) < 0.01 && Math.Abs(ballY[hiddenI] - hiddenY[hiddenI]) < 0.01
Context: the 8-ball, frame 1: ball 0 at (160.000, 118.000), expected (160.000, 112.000) (line 565) |
Failed | IN0 OUT0 |
| × | Hidden_PocketAndGroupsThe first ball pocketed by a player without a group gives that player its group; a ball of the own group lets the player shoot again.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Math.Abs(ballX[hiddenI] - hiddenX[hiddenI]) < 0.01 && Math.Abs(ballY[hiddenI] - hiddenY[hiddenI]) < 0.01
Context: ball 1 into the top middle pocket, frame 1: ball 0 at (160.000, 118.000), expected (160.000, 112.000) (line 565) |
Failed | IN0 OUT0 |
| × | Hidden_PocketBeforeCushionA ball that passes the cushion line close to a pocket falls in, because pockets are checked before cushions.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Math.Abs(ballX[hiddenI] - hiddenX[hiddenI]) < 0.01 && Math.Abs(ballY[hiddenI] - hiddenY[hiddenI]) < 0.01
Context: the cue ball at the edge of the top middle pocket, frame 1: ball 0 at (168.000, 137.500), expected (168.000, 130.000) (line 565) |
Failed | IN0 OUT0 |
| × | Hidden_RollAndStopA lone ball rolls straight, slows down by friction and stops; the shot ends in the frame the ball stops, and the turn passes.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Math.Abs(ballX[hiddenI] - hiddenX[hiddenI]) < 0.01 && Math.Abs(ballY[hiddenI] - hiddenY[hiddenI]) < 0.01
Context: a roll to the right, frame 1: ball 0 at (60.750, 120.000), expected (63.000, 120.000) (line 565) |
Failed | IN0 OUT0 |
| × | Hidden_ScratchWhen the cue ball falls into a pocket, it comes back at (90, 120), or 12 further left for every time that spot is taken, and the turn passes.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Math.Abs(ballX[hiddenI] - hiddenX[hiddenI]) < 0.01 && Math.Abs(ballY[hiddenI] - hiddenY[hiddenI]) < 0.01
Context: the cue ball into the top middle pocket, frame 1: ball 0 at (160.000, 58.750), expected (160.000, 55.000) (line 565) |
Failed | IN0 OUT0 |
| × | Hidden_ScreenThe screen shows the cloth, the 6 pockets and the balls in their colors at their places, with the spots of the stripes.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: hiddenRed.Found && hiddenRed.X == 244 && hiddenRed.Y == 74 && hiddenRed.Width == 12 && hiddenRed.Height == 12
Context: ball 3: (14, 56) 242 x 30
Values:
hiddenRed.X = 14
hiddenRed.Found = true
hiddenRed.Y = 56
hiddenRed.Width = 242
hiddenRed.Height = 30 (line 940) |
Failed | IN0 OUT0 |
| × | Hidden_ShotStartSpace gives the cue ball the speed power in the aim direction; the ball first moves in the next frame, and the keys do nothing until the balls stop.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Hidden_Near(ballX[0], 100 + hiddenVX) && Hidden_Near(ballY[0], 120 + hiddenVY)
Context: the first physics frame moves the cue ball by its velocity: (101.29903810567666, 120.75) (line 657) |
Failed | IN0 OUT0 |
| × | Hidden_TurnRulesWith groups given, the turn passes after a miss or after pocketing only the other player's ball, and stays after pocketing an own ball.0 AI queries
0 tool callsHiddenTest Assertion expression evaluated to false: Math.Abs(ballX[hiddenI] - hiddenX[hiddenI]) < 0.01 && Math.Abs(ballY[hiddenI] - hiddenY[hiddenI]) < 0.01
Context: an opponent's ball into the pocket, frame 1: ball 0 at (160.000, 118.000), expected (160.000, 112.000) (line 565) |
Failed | IN0 OUT0 |