A refund-handling judge has little to assess in a conversation about the weather. Give that conversation zero and the refund average falls. Give it full marks and the average rises. Ask a model to judge it anyway and you may spend money producing a number that should never have existed.
Mastra’s September 19 addition of notScorable() lets a scorer say that a run is outside its remit. The pull request states that it was made with Cursor, and the merged commit credits Cursor as a co-author. Those are public receipts of coding-agent assistance; they do not tell us which lines the human or agent wrote.
The little return value with a long journey
The helper is small. The engineering is in every place that would otherwise expect a number. A function step can return notScorable(reason); the scorer then stops the remaining pipeline, preserves results from earlier steps and returns a tagged outcome without a score property. Put the decision in preprocessing and the later judge call never happens. Put it after an expensive step and that earlier cost has already been incurred.
The scorer implementation carries this distinction in its result type and runtime control flow. The accumulator counts these outcomes separately and leaves them out of averages. Live scoring avoids a score row; experiment results expose the not-scorable outcome alongside null score and error values. This is a change across the evaluation system, not just an early return in a callback.
The pipeline tests check both halves: an irrelevant input never invokes the judge or score generator, while a relevant input still takes the full path. They also check that a later skip preserves earlier step results and emits no observability score.
The empty verdict is the revealing case
The first tempting description is “cheaper, cleaner evaluations.” The stronger test is what happens when nothing qualifies. A careless implementation could interpret no failures as success.
Mastra’s gate scenarios explicitly expect no verdict when every top-level gate or threshold is not scorable. In a mixed evaluation, a verdict can still pass on the assertions that did produce numbers, while the skipped gate remains absent from the gate results and visible in the skip count. A passing verdict therefore needs to be read with its coverage.
The public review records a reviewer checking consistency between top-level and per-turn all-skipped behavior. That review changes the meaning of the feature: its contract includes how missing judgment survives aggregation, not merely whether a model call was saved.
A reusable skill: test where absence changes meaning
When introducing a “not applicable” result, trace it through each consumer before celebrating the new API. Keep three fixtures beside one another: one item that must score, one that must skip, and a batch where everything skips. Assert the shape as well as the value: no score property, a retained reason, a separate count and no fabricated verdict. Then add a mixed batch to show exactly what a passing result covers.
The application still owns eligibility. If the customer needed a refund but the agent failed to call the refund tool, “tool absent” could hide the very failure you wanted to measure. Use a separate check for whether the required action occurred; reserve a tool-execution scorer for judging executions that actually happened. The helper can preserve a distinction faithfully even when a builder chose the wrong distinction.
We inspected the merged code, tests, documentation and public review. We did not rerun the upstream suite or measure judge-cost savings, and the Cursor attribution is not a productivity measurement. The practical next check is a dashboard or release gate that displays the skipped population beside its average. Otherwise, the pipeline can preserve the truth and the presentation can still throw it away.