Benchmark summary
The model builds Mahjong solitaire (72 tiles in 3 layers) played with the mouse with a polished look in SC# as an agent: it writes the app, compiles it, runs its own tests and fixes the code. Hidden tests check the rules from the task; each hidden test is its own result.
Tests with multiple category tags are counted in each matching category, so category totals may exceed the report total.
| Benchmark items | |||
|---|---|---|---|
| Script / test | Status | Tokens | |
Building Games in SC#: Mahjong solitaire
13 failed |
0.0% |
IN228 136 OUT17 090 | |
| × | MahjongThe model builds a Mahjong solitaire game played with the mouse from the task as an agent; this item passes when the app compiles and can be exported, and each hidden test is its own item.4 AI queries
1 tool call Assertion expression evaluated to false: app.Built
Context: AI: completed
App: not submitted; the last file version is graded (end: stopped: no tool call in 3 answers in a row)
Work: 4 AI calls (5 model rounds with tool calls), 1 whole-file writes, 0 edits (0 lines), 0 CheckApp calls, 0 refused rewrites, 0 refused repeated edits, 0 rejected submissions, 3 reminders to hand in, 0 broken answers
Compile errors: (221,32) Operator '+' cannot be applied to 'string' and 'int'. It supports numbers and string/char concatenation. (222,32) Operator '+' cannot be applied to 'string' and 'int'. It supports numbers and string/char concatenation.
Values:
app.Built = false |
Failed | IN228 136 OUT17 090 |
| × | Hidden_AvailablePairsavailablePairs counts the faces with two free tiles; with none left the game shows NO MOVES until a pair opens up.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_FreeTilesOnly free tiles can be selected: tiles with a tile on them or with both sides closed ignore clicks.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_NewGameAfter AppInit the layout lists match the task, every face is on exactly two tiles, nothing is removed or selected, and the start has a pair to remove.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_NewGameWithEnterEnter starts a new game at any time; a click in the same frame does nothing.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_PickinghoverTile is the tile under the mouse from the highest layer, with the rectangle edges exactly as in the task.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_RemoveAPairClicking a second free tile with the same face removes the pair and records it in history.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_ScreenFacesDifferent faces look different in the tiles' middle areas, and the same face looks the same.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_ScreenRemovedTilesA removed pair disappears from the screen: what is left there does not depend on the removed tiles' face, and undo draws them again.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_ScreenSelectionAndHoverThe selected tile and a free tile under the mouse look different; a tile that is not free does not react to the mouse.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_SelectionA click on a free tile with another face moves the selection; clicks on blocked tiles or outside the pile keep it.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_UndoU puts back the last removed pair and clears the selection; with an empty history it only clears the selection.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |
| × | Hidden_WinningRemoving the last pair wins; U after the win puts that pair back.0 AI queries
0 tool callsHiddenTest Not run: the app does not compile. |
Failed | IN0 OUT0 |