One reviewer checks several citations. It finishes the first, starts writing the second, then hits its time limit. Treat the batch as a single success or failure and you face a bad choice: discard useful work, or trust an incomplete answer.
The citation checker in Shepherd 0.3.1, released September 9, gives each reference its own outcome. A batch may end partially complete while the unfinished citation remains visible. Its release commit credits Codex as a co-author, public evidence of coding-agent involvement. That credit does not identify which lines the agent wrote or establish that the work was autonomous.
The host keeps watch
In checkpoints.py, a host-side observer watches the reviewer’s output directory while the confined process runs. It looks for a separate JSON record for each expected citation. It copies only readable, complete records whose identifier matches the expected reference, with size and path checks. A half-written file is not promoted just because it exists.
The observer checks periodically and takes a final pass when execution exits. It also hashes protected inputs. Before captured results can be finalized, an integrity record must show those inputs remained unchanged. The results then pass through identity checks and report validation. Capturing a model’s draft and accepting its conclusion are separate operations.
There is an easily missed repair inside that loop: if a previously valid output becomes malformed, the corresponding saved capture is removed. Otherwise an old good-looking result could quietly stand in for the reviewer’s newer broken write. The system preserves useful work without pretending that the last trustworthy state is automatically the current one.
A failure you can inspect
The checkpoint tests model the awkward cases directly: an alarm with one completed neighbor, a changed protected input, a malformed neighboring result, and a worker that misses its hard deadline. Their assertions distinguish completed, failed validation and unfinished references.
A small offline probe of the published capture functions reproduced four boundaries: the complete record was copied; partial JSON was excluded; changing a protected input failed the integrity seal; and replacing a valid record with broken JSON removed its stale capture. We did not run a live model review or the full package test suite.
The transferable skill: shrink the unit of recovery
Batching can make an agent economical while making failure expensive. Shepherd’s useful pattern is to keep the batch as the scheduling unit and make one citation the recovery unit. A builder processing invoices, migration findings or test failures can use the same separation: assign stable item IDs, persist independently complete records, verify their source inputs, then validate each result before combining the report.
Crucially, retain the missing outcomes too. A report with five accepted items and one unfinished item should expose all six. Otherwise partial recovery merely hides a smaller failure inside a more reassuring status.
This checker verifies a work’s identity and bibliographic metadata; it does not determine whether the paper supports the argument citing it. Its recorded development evaluation scores 116 of 140, with abstentions and unfinished work counted as errors, after excluding ten ambiguous labels. The selection was screened during development and is not held out. Preserving results is an engineering achievement; it supplies no independent accuracy gain.
The next test for a builder borrowing this design is simple to state: interrupt a batch halfway through, then ask the report to explain every item. Good recovery should leave less work to repeat and no uncertainty concealed.