An AI coding agent says it fixed the bug. Expectrail plays the game and watches to find out.
"The tests pass" proves only what someone wrote a test for, and the agent's own word proves less. Expectrail replays the changed game, has Meta's vision models find what each plain-English check names in the frames, and measures that against the engine's own record. The change is accepted only if every check passes, and the agent can't reach the checks.
In your browser, with no keys: pick any of twelve cases from the benchmark and see its captured clip, its checks and the verdict.
Two fixes that both stop the fall
Muse Code's fix
AcceptedWritten from a plain bug report, in one line: the robot's body collides with the ground again. Every check passes, on the public route and the private ones.
A fix that only looks right
RejectedThe same line, plus one that draws the robot lower than its body. The fall stops, and the robot is drawn inside the ground.
Real frames: frame 5 of each replay, from the recorded reviews of runs 41223748922 (review) and 41123758768 (review). The right-hand fix was written by us for the judge benchmark. The yellow line is drawn here at the floor's edge; the verdicts are the runs' own.
Every correct fix accepted. No fake fix accepted.
Fifteen fixes with known answers: seven that only look right, eight correct. Expectrail accepted all eight correct ones and none of the seven, in every run. Claude and GPT judged those fixes from the same screenshots Muse Code saw. They accepted none of the seven either, but turned down correct fixes.
| Judge | Fake fixes accepted | Correct fixes turned down |
|---|---|---|
| Expectrail, with SAM 3.1 | 0 of 210, 0 and 0 of 7, in three runs | 0 of 240, 0 and 0 of 8 |
| Claude Fable 5.1, from screenshots | 0 of 210, 0 and 0 of 7, in three passes | 5 of 242, 2 and 1 of 8: it read both sinking fixes as a robot floating |
| GPT-6 Astra, from screenshots | 0 of 210, 0 and 0 of 7, in three passes | 1 of 240, 1 and 0 of 8 |
| Muse Code, from screenshots | 6 of 212, 2 and 2 of 7, in three runs | 5 of 242, 1 and 2 of 8 |
Expectrail and Muse Code: three runs of the judge benchmark. Claude and GPT: three passes in one run, given Muse's prompt and frames byte for byte. We wrote the fake fixes, and each judge decides from its own evidence: Expectrail from the engine's record and SAM's boxes on public and private routes, the others from ten screenshots and the bug report. On fifteen cases, the gap from GPT is one judgment.
One of the seven, up close
Three judges saw the fix that sinks the robot. Two of them said it worked.
| Expectrail with vision off, on the engine's position log | Accepted | The engine says the robot stopped on the floor, which is true. Where it is drawn is not in the log. |
|---|---|---|
| Muse Code, shown before-and-after screenshots | Accepted | It wrote that the player stays in view in all framesand that the fall was fixed. |
| Expectrail, with SAM 3.1 | Rejected | SAM found the robot's feet below the floor's edge, on the public route and the private ones. |
The same decisions in all three runs of the judge benchmark.
Why a second opinion
Coding agents now change code faster than anyone can review it. Existing checks still matter, but they cannot cover failures nobody thought to look for.
"The tests pass" is not proof
A test checks what someone already thought to write a test for. The bugs that matter most are the ones nobody anticipated, and a game's worst ones show only on screen: a robot through the floor, coins that are not drawn, a character hidden behind trees. The demo game we use ships with no tests at all.
The agent grades its own homework
Ask a fixer to judge its own patch, and it may approve a change that looks convincing but is still wrong. Shown before-and-after screenshots of a fix that draws the robot inside the ground, Muse Code said the fall was fixed.
So the judge has to be independent
Expectrail verifies by observation, not by trust. Evidence comes from running the changed game and watching it. Models observe; plain code decides. And the agent that wrote the change cannot reach the checks, the held-out routes or the key that signs a release.
We built and measured it on a game first because a game shows the gap plainly, then pointed the same verifier at a web checkout page to see whether the engineering carries. It adds a verification step alongside CI, tests and QA; it does not replace them.
How it works
People write the checks that decide: a person approves, in one sentence, what the player should see. Code, not a model, decides whether each check passes. Advisory discovery adds checks nobody wrote, read from the game's own code; for now they report, they don't reject.
Play
A harness replays a fixed route through the changed game and captures every frame, the same way every time.
See
SAM 3.1, or the open-weight Muse Glimmer, finds what the check names. The engine logs where things really are.
Decide
A deterministic evaluator compares the two. A missing or uncertain observation is "needs review", never a pass.
Fix
Muse Code gets a plain bug report and patches the game in a sandbox. If an attempt fails, the next one gets Expectrail's evidence; the two sinking bugs needed it.
Hand over
Every check passes on public and private routes, or nothing ships. A signed release candidate waits for a person.
What the fixer can reach
- The game's editable files, in its own worktree
- The bug report: the failing check's sentence, screenshots, the route and its log
- After a failed attempt, Expectrail's evidence: the failing check's approved spec (read-only), the verdict and the first failing frame
In the live runs Muse Code works inside bubblewrap with only its worktree mounted; without bubblewrap it falls back to a plain worktree. Since Oct 30 its only network is a key gateway to Meta's API, which holds the key; it shares the host's kernel, so this is a boundary by absence, not a VM.
Kept out of its reach
- The held-out bugs, the private routes, and why a guard failed
- The evaluator's code, and write access to any check
- The key that signs a release
If the vision model doesn't answer, the same candidate is judged again (unless the key was refused or is out of credit), then by Glimmer checked against the engine. It is never sent back to the fixer. When Glimmer decides, the report says so, because the ground is then judged on the engine's log alone.
Measured, with the run behind every number
The three headline results come from CI output recorded in the repository, with the commands needed to rerun each one.
Held-out bugs caught
Eight bugs kept from the fixer, across four families, judged with SAM 3.1. No false alarm on the clean game. Run 41003749744
Fixes that only look right, accepted
Seven bad fixes, three runs, the same answer every time. The position log alone let 3 through; Muse Code judging from screenshots let 2 through. Judge benchmark
Correct fixes accepted, every run
No correct fix turned down. Claude Fable 5.1 and GPT-6 Astra, judging the same screenshots, turned down correct fixes 5 and 1 times in 24. Outside models
All results, with the run behind every number
Held-out bugs caught
Eight bugs kept from the fixer, across four families: falling through the floor, missing textures, hidden behind scenery, sinking.
Fixes that only look right, accepted
Seven bad fixes, judged three times; fewer is better. Expectrail also accepted all eight correct fixes, each an exact revert of a seeded bug, in every run.
Claude Fable 5.1 and GPT-6 Astra, shown the same screenshots, also accepted none of the seven, but turned down correct fixes: 5 and 1 times in 24. Outside models
A mutation bench, registered first
The sample, the oracle (any difference from the clean game, in positions or in frames) and its thresholds were committed before any case in the study was judged, and only public checks decide. Bugs nobody wrote are where Expectrail is weakest (what it can't do yet). The bench and its limits
The whole loop, live
Checks nobody wrote: advisory discovery
Twenty-two generic probes replayed on each build, judged by rules read from the game's code, with no rule written for any one bug. A finding has to repeat three times out of three. It doesn't gate yet. The blind rows are fresh defects written after its rules were frozen; the reported bugs are a development result, because its rules were chosen after reading those reports.
Beyond games: a web checkout page
The same evaluator, contracts and fixer, with Chromium in place of the game engine and the page's layout in place of the position log. The checks were tuned on the development bugs and changes below; the four private bugs are the test they were not tuned on.
What we have not shown yet. We gave Muse Code four bugs that other people reported in Godot games, in the reporters' own words, five times each. Expectrail accepted all 20, each bug's own check passed, and no guard fired; Muse from screenshots rejected one that we have not yet judged ourselves. Given 10 generated bugs the checks had already caught, Muse Code fixed all 10 correctly, so there was nothing to catch there either. There is no blind catch on a real-world bug yet. One of the eight held-out catches came from a check built after that bug was first missed. Discovery's blind tests are sixteen defects in two sets, a small sample, written by AI agents from a prompt we wrote, and its Muse Spark planner has one blind run. All results and their limits
Seven Meta models, and what each does
The models look, write and fix. None of them decides whether a change passes.
- SAM 3.1Finds what a check names in the frames of a replay: the robot, the coins, the checkout button.
- Muse Glimmer 30B open weightsJudges the vision checks with no SAM key, its boxes checked against the engine. Its boxes are too loose for the ground check, which then uses the engine's log. Also the fallback when SAM doesn't answer.
- Muse SparkDrafts a test spec from a one-sentence check, for a person to tighten and approve, and sorts playtest remarks into draft checks. For discovery it proposes probes and rules in a closed language, each tested against the clean game first; blind, it added one defect to the templates' three on a set of eight.
- Muse Spark Open open weightsThe same two jobs on open weights, served on Baseten.
- Muse CodeThe fixer. It patches the game in a sandbox, and can neither change the checks nor see the held-out ones.
- Muse WonderTriage: turns failure evidence into a bug report and a repair request.
- Muse Voice TranscribeTurns a recorded playtest into who said what. Raw speech never becomes a check, and a request to switch one off is refused and queued for a person. Built; not yet run live.
Why open weights matter here: an unreleased game build is confidential. With Glimmer, the vision checks could run on a studio's own hardware; so far we have run it only on a Fireworks deployment. How we use them
What it can't do yet
- It checks a change; it doesn't explore. It replays fixed routes and 22 fixed discovery probes, so a bug where none of them goes is invisible to it. Of 77 generated single edits that differ from the clean game, its checks caught 10, and none that move, resize or re-layer scenery; weighted by how common each kind of edit is, that is 0.9%. Bugs nobody wrote are where it is weakest, and we measured it on purpose (the mutation bench).
- Only checks people wrote can reject a change. Three approved checks guard the game. Discovered checks only report, reach just the four mechanics their profile describes, and flagged 7 of 16 fresh defects blind.
- One game and one web page. Mostly with bugs we seeded. A teammate wrote the game's held-out bugs; the page's private bugs came from an AI agent of the same model family that tuned its checks.
- The sandbox isn't a hard boundary. The fixer reaches Meta's API through the key gateway and shares the host's kernel.