A prompt experiment can look rigorous while quietly changing everything that made the old result comparable. Mastra’s new observational-memory simulator takes that problem seriously. Its public memory commit lists Mastra Code (openai/gpt-5.6-sol) as a co-author, then adds 2,789 lines of local extract, replay, diff, and test machinery. The interesting builder move is not “use an agent to write a benchmark.” It is deciding what the benchmark must refuse to change.
The simulator starts from persisted observational-memory records, not a fresh reread of raw conversation messages. Its public README explains why: prompt changes can alter observation-cycle boundaries, which would make an apparent A/B difference a comparison of two different inputs. The tool extracts a bounded source set read-only, reconstructs the recorded cycles, and replays capture and curation into a local input database and separate local arm databases.
The repo evidence: make the comparison narrow enough to believe
Arm A and Arm B may differ only in the instructions appended to the capture or curation prompt. The A/B driver rejects a run when models, cadence, scopes, or selected cycles differ, recreates a fresh database for each arm, and then runs a third control arm with Arm A’s own configuration. That A/A result is the model-noise floor: if the A-versus-B difference is no larger than the same-prompt difference, the source says the prompt change has not shown a detectable effect. A colocated isolation test checks the state boundary underneath that claim.
The transferable building skill: promote the control to a first-class feature
Builders can reuse this practice far beyond memory systems. Before comparing two agent prompts, pin the source data, reconstruction rule, model pair, cadence, scope, and storage boundary. Change one instruction layer, then run the original configuration again as an A/A control. Treat the control’s drift as a constraint on what the A/B result can mean, not an inconvenience to omit. This makes the experiment smaller, but it makes a positive result harder to mistake for a change in state, sampling, or hidden setup.
The limit
This is repository-local tooling, not a shipped memory feature or a production result. The commit reports its own test totals; this reporting run inspected the source and tests but did not execute the simulator. The README also requires local PostgreSQL with pgvector, built workspace packages, and model credentials, and it deliberately does not replay the full observer or reflector lifecycle. That restraint is part of the lesson: a simulator is useful when its excluded behavior is visible enough to keep its result honest.