Expectrail: the interactive demo

Case Accepted Captured with the harness: Godot Movie Maker, smoke route, 60 fps

The replay, with the clean game's floor drawn over it

Before: the seeded bug

trace

This case's recorded review

Verifier decision ACCEPT

1. regressionPASS

2. gameplayPASS

3. clean controlPASS

4. held-outPASS

all 4 checks passed; candidate verified under clean level reference.

candidate.patch: Muse Code's recorded fix

from bench/live-muse-clean-v1/fix/candidate.patch

--- a/player/player.tscn

+++ b/player/player.tscn

-collision_mask = 14

+collision_mask = 30

The eight development bugs: caught, then fixed

Each bug replayed on the smoke route and judged with and without SAM 3.1, then handed to Muse Code twice: once with Expectrail's evidence, once with screenshots alone. Both briefings fixed all eight; what differs is who can judge the result.

Sources: development fix matrix (bench/fix-matrix); detection bench (bench/detection, run 40841977642); held-out runs 1 and 2 (run 41099054641, run 41199992529); the three-briefings run (run 41223754703); mutation bench (wrong-fix run 41268021760).

Development bugs fixed with Expectrail's evidence: 8 of 8

Held-out bugs fixed, two runs: 8 of 8 with the evidence, 7 of 8 from screenshots

False alarms on the clean game: 0

Case	What the bug does	Caught without SAM	Caught with SAM 3.1	Fixed, with the evidence	Fixed, from screenshots
physics-fall-through / v1	The robot's collision mask drops the ground (14)	Yes (position y)	Yes	Fixed, 12 min	Fixed, 10 min
physics-fall-through / v2	The level's tileset drops its collision layer (0)	Yes (position y)	Yes	Fixed, 13 min	Fixed, 15 min
missing-texture / v1	The coin texture points at a missing file	No (coordinates valid)	Yes (no coins seen)	Fixed, 17 min	Fixed, 9 min
missing-texture / v2	The coin sprite's texture property is cleared	No (coordinates valid)	Yes (no coins seen)	Fixed, 20 min	Fixed, 9 min
layering / v1	The robot's z-index drops it behind the level (−3)	No (coordinates valid)	Yes (robot seen in 17 of 31 frames)	Fixed, 16 min	Fixed, 18 min
layering / v2	The robot's sprite is given z-index −6	No (coordinates valid)	Yes (occluded)	Fixed, 10 min	Fixed, 23 min
sinking / v1	The collision shape sits 26 px too high	Yes (ground line)	Yes	Fixed, 36 min	Fixed, 22 min
sinking / v2	The sprite is drawn 26 px too low; physics intact	No (physics intact)	Yes (SAM against the engine)	Fixed, 23 min	Fixed, 23 min

Held-out bugs fixed

8 of 8

Run 41199992529 (commit e216fb7): with Expectrail's evidence Muse Code fixed 8 of 8; from screenshots, 7 of 8 (sinking/v4 timed out three times). The first held-out run, 41099054641, ended the same: 8 of 8 and 7 of 8.

Three briefings, October 31

8 · 7 · 6

Run 41223754703, development bugs: the evidence first, 8 of 8, each on the first attempt; the report first with the evidence after a failed attempt, 7 of 8; the report only, 6 of 8. Every miss was a sinking bug where the report-only attempts timed out (report-first.md).

Mutation bench: wrong fixes

31 of 39

Run 41268021760: 47 candidates from six operators the team wrote, registered before judging, public checks only. The tiered judge caught 31 of 39 bad fixes (79.5%), 24 of them only with SAM, and accepted the 8 good ones. The 8 misses are one operator that changed nothing in the running game.

Fixes that only look right

0 of 7 accepted

Judge benchmark, 7 bad fixes, 3 runs: Expectrail with SAM accepted 0 of 7; without SAM, 3 of 7; Muse Code judging from screenshots, 2 of 7. Good fixes: Expectrail accepted 8 of 8 every run. The four rejected fixes in the Cases tab are four of the seven.

The whole loop, live

10 of 10

Run 40990485265: the loop end to end, 10 independent runs, each accepted on the first attempt; 12 to 22 minutes a run (median 14.5). After the report-first change, run 41223748922: 3 of 3 accepted, each on the first attempt from the plain report.

Held-out bugs caught

8 of 8 with SAM

Run 41003749744, SAM 3.1 on: 8 of 8 held-out bugs caught, no false alarm on the clean game. With no vision model (engine and gameplay checks only), 4 of 8.

Why the vision model is not optional

Without it, 5 of the 8 development bugs pass every engine and gameplay check: missing textures, a robot drawn behind the tiles, and a sprite drawn off its body all leave the physics coordinates valid. SAM 3.1, or the open-weight Muse Glimmer checked against the engine (also 8 of 8 on the held-out bugs), sees what the player sees. The full results, with every limit, are in the README.

Twelve cases, with the evidence beside each

The eight development bugs: caught, then fixed

Held-out bugs fixed

Three briefings, October 31

Mutation bench: wrong fixes

Fixes that only look right

The whole loop, live

Held-out bugs caught

Why the vision model is not optional

What the fixer can reach, and what it cannot

The morning report: the hand-over to a person

Expectrail morning report

Checks

Suite provenance

Gameplay

Verified candidate diff

Signed release candidate (Ed25519)

Unresolved for a person