Introduction

We test local models by asking them to build small apps as agents. The model gets a task, writes the app in SC#, our C#-like scripting language, tests it, fixes it and submits it. Hidden tests, which the model never sees, then check whether the app follows the rules of the task. In the end, the app is translated for a web browser, so anyone can try it.

We want to know what local models can and cannot do in real work. Popular benchmarks give one number per model, mostly for short, separate tasks. They do not show how a model copes when it has to plan a whole program, write it, test it and fix its own mistakes. So we built an environment where we can create such larger tasks easily and run them on any local model.

Why apps? A small app is a complete program: input, state, timing, rules and output on the screen. It needs planning, and its behavior is easy to check. Unlike a single function with a few unit tests, an app shows whether a model can keep many rules consistent in one program and check its own work.

What an SC# app is

An SC# app is a normal SC# script with two special functions. The host calls them; the script never runs its own endless loop.

void AppInit()   // runs once at the start: opens the screen and sets the initial state
bool AppLoop()   // runs once per frame; returns false to quit

• One call of AppLoop is one frame. The app reads the input, updates its state and draws the whole screen again. Everything drawn during the call is shown together.
• The frame rate is fixed, for example 30 frames per second, so movement and timers are counted in frames, not in seconds.
• The state lives in script variables. They keep their values between frames, and tests can read and set them.
• The keyboard and the mouse are the input. Input.IsDown(key) is true while a key is held, and Input.WasPressed(key) is true in the first frame after the key went down. Mouse buttons work like keys, and the app reads the position and the movement of the mouse in screen pixels.
• Drawing has no antialiasing. A pixel belongs to a shape when its center is inside the shape, so the same code gives the same pixels everywhere. Text is drawn by the browser with its fonts, so tests check text as text, not as pixels.
• Apps can paint their own images. An app paints an image once, pixel by pixel or with shapes, and then draws it as a picture in 2D or puts it as a texture on 3D shapes.
• Apps can be 2D or 3D. A 3D app describes a scene of boxes, spheres and other shapes and a camera. The host draws the scene, and tests can ask which object is visible at a point of the screen.

How a run works

The process

1. Task. The model gets the task, a reference of the SC# language, guides to 2D and 3D SC# apps with examples, and four tools.
2. Code and own tests. The model writes the app and its own tests into one file.
3. Check and fix. The model checks the file, reads the errors and the test results, fixes the code and checks again.
4. Submission. When the app is ready, the model submits it. An accepted submission ends the work.
5. Hidden tests. Our benchmark service runs the hidden tests on the submitted app. Each hidden test is a separate result.
6. App in the browser. The service translates the app from SC# to JavaScript for a web page. The model's own tests stay out.

The model never runs or watches its app. It knows only what the compiler and its own tests tell it. This is on purpose: the test shows whether a model can find its own mistakes by testing, not by looking.

The tools

• Create the file: writes the whole app file.
• Change lines: changes selected lines of the file.
• Check: compiles the file, runs the model's own tests, checks that the app can be translated for the browser, and returns the compile errors, one line per test and the compiler warnings.
• Submit: accepts the app only when it compiles, can be translated, has its own tests and all of them pass. Otherwise it is rejected with the list of problems.

Rules of the work

• Small steps. The task asks the model to start with a small version that compiles and to grow it in steps. Once the file exists, the model must change it in small edits instead of rewriting it, and edits that cannot work are refused.
• Reminders. If the model stops before an accepted submission, it gets the current problems of the file and is asked to go on.
• Limits. The work ends after a fixed number of model answers in total, or when the model answers three times in a row without a tool call. The limit is 50 answers by default. A bigger task sets a higher one, for example 100 for a 3D block world.
• The graded file. We grade the submitted app. If there is no accepted submission, we grade the newest version of the file that compiles. If no version compiles, we grade the newest one.

Every setup runs once

Every setup runs the task once, and that run counts. We do not rerun failed runs and do not pick the best of several runs. The aim is to show what a user can expect from one attempt. This also means that a single result is a hint, not a proof: with one run, a model can be lucky or unlucky.

Each article names the models, the mode (Instruct or Thinking), the quantization and the runtime with its version. All models in one article run with matching settings.

The task

Every task has two parts.

• Part A has the rules that the hidden tests check. It fixes what the tests need to read the app: the screen size and frame rate, the names and meaning of the variables that hold the state, the order of the steps in one frame, the numbers, the colors of the objects and the texts on the screen.
• Part B lists features without exact rules, for example help texts on the screen, short visual effects or a pause key. The hidden tests do not check part B. You see it when you use the app.

Part B must not change what the hidden tests count. It must not use the colors of part A, and it may draw over the area of part A only for a short effect. A screen test then waits until such an effect is over.

In many tasks, the model designs part of the behavior. The task then describes what must happen, not how. For example, a moving object must reach a target within a number of steps, but the model chooses how it finds the way. The hidden tests then check properties instead of exact values: they try many starting points and check the rule in each of them.

The order of steps in a frame is part of the rules. For example, the task says whether a new object moves already in the frame in which it appears, or whether a frame with a restart key does anything else. Without such rules, two careful people could build two different apps, and a test could not decide which one is right.

Hidden tests

How a test works

A hidden test is an SC# function that drives the app frame by frame. It starts the app, sets up a situation through the state variables, presses keys, moves and clicks the mouse, and checks the state and the screen. A short example for a cursor on a grid of 8 columns with 12-pixel cells, which starts at x = 64:

/// The cursor moves one cell per key press and stops at the right edge.
void Hidden_CursorStopsAtTheEdge()
{
    AppInit();
    cursorX = 6;
    Hidden_Tap("Right");
    Assert(cursorX == 7, $"cursor at {cursorX}, expected 7");
    Hidden_Tap("Right");
    Assert(cursorX == 7, "the cursor stays in the last column");
    var box = Frame.FindColor(Color.Hex(0xFFD54F));
    Assert(box.Found && box.X == 64 + 7 * 12, "the cursor is drawn in column 7");
}

void Hidden_Tap(string key)
{
    // a press held for one frame, then a frame without it
    Input.Hold(key);
    AppLoop();
    Input.Release(key);
    AppLoop();
}

The model's own tests use the same tools, so it can test its app in the same way.

What they check

• The state: positions, speeds, counters, the contents of lists and grids.
• The screen: where each object is drawn and in which color (Frame.PixelAt, Frame.FindColor, Frame.CountPixels), and which texts are shown (Frame.HasText). In 3D apps, tests also check which object is visible at a point of the screen (Frame.TagAt).
• The timing: in which frame something happens, how many frames a timer lasts, and that a frame does nothing more than the task says.

The task says that the variables of part A are the whole state, so a test can set them directly. This keeps tests short and independent of random events.

Fair keys, numbers and time

• A key press lasts one frame. When a test presses the same key twice, there is a frame with no key between the presses. An app that reacts to the moment a key goes down and an app that checks whether a key is held both see every press. Mouse clicks follow the same rule.
• Random events are not graded by luck. Where an app uses random numbers, the test sets the state directly or checks a property over many runs, for example that every random value comes from the allowed set.
• Decimal numbers get a small tolerance where the order of operations may differ. Two correct formulas can differ in the last digits. A tolerance covers that, but not a different rule or a different order of steps.
• Each test has a time limit. While the model works, each of its own tests may run for 20 seconds. In the final check, each hidden test may run for 60 seconds, so a busy machine does not fail a correct app. When a hidden test still fails only because of the time limit, we check the same app again, without the model.

How we know the tests are fair

A task is correct if a careful person, given only the task, can work out one answer without asking anyone. We check this in three ways:

1. Two reference apps. We write two apps of our own from the task text alone, with different designs, for example a different way to store the data, to read the keys or to solve the parts the model designs. Both must pass all hidden tests.
2. Broken copies. We make many broken copies of a reference app, each with one typical mistake, for example a wrong number, a missing edge case, a step in the wrong order or a key read in the wrong way. Every broken copy must fail at least one hidden test. If one passes, a test is too weak, and we make it stronger.
3. Test runs. Before the final runs, we run each task with a model and read every failed test in the model's code. If the task text allows two readings, we make the text clear. If a test assumes something the task does not say, we fix the test.

We never make a task easier or a test looser because models fail it. A failure that follows from a clear rule in the task stays a failure, even if every model makes the same mistake.

How we grade

The score of a run is the share of passed results: the build of the app and each hidden test. The build passes when the model produced an app that compiles and can be translated for the browser. If there is no app, or the code does not compile, all hidden tests fail too.

Besides the score, the articles report how the model worked:

• Own tests: how many tests the model wrote for its app.
• Checks: how often the model compiled the app and ran its tests.
• Edits: how many line edits the model made, and how many lines they changed.
• Model requests: our requests to the model, that is the first one and every reminder.
• Model rounds: answers of the model with tool calls, within all requests. The limit counts all answers.
• Unwanted tool calls: calls in a wrong format, refused edits, rejected submissions.
• Tokens read and written: every tool call sends the whole conversation to the model again, so long runs read far more tokens.

Fewer steps are not better by themselves. What counts is the result of the hidden tests. But the numbers show how differently models work: some write the whole app at once, others build it in many small steps.

The results of every run, with each hidden test, are on the Benchmarks page.

The app in the browser

The submitted app is translated from SC# to JavaScript and runs on a web page. On the results page of a published run, the link "Play in browser" opens it. The translation follows the same rules as the SC# runtime: we compare the results of both on a large set of test programs. By default, the page scales the app in whole pixels, so the picture is sharp and the same as in the tests. A switch on the page turns this off and scales the app to the full window.

In the browser you also see what the hidden tests do not check: help texts, visual effects and the general look and feel.

Changes to the method

The method grows with the tasks. We published this article on 10 October 2026. Since then we added the mouse, images and textures, 3D apps and larger tasks, and on 14 October 2026 we changed two rules of the work. The table shows what changed.

The new rule for the graded file matters only when a model runs out of answers in the middle of a change. Before, such a run scored zero for all hidden tests if the last version did not compile, even when an earlier version worked. Now it is graded on the last version that worked, as a person would hand in the last working state.

Visual improvements

Apps built after these changes can look much closer to real games. Models can now paint their own graphics, build 3D worlds and mark what the player points at. The look is still not graded: the hidden tests check only the rules of part A. The visual features let a model meet part B of a task, and you can compare the results with the link "Play in browser" on the Benchmarks page.

Limits of the method

Because of these limits, we treat each result as one practical signal about a model, not as a final verdict. We compare it with the results of other tasks before we draw conclusions.