A green check can mean a workflow passed. It can also mean a job was skipped so cleanly that the required check still looked healthy. That distinction gets expensive when the skipped job is supposed to send a coding agent into a pull request, review a security boundary, or create an artifact a team believes it is receiving.
Pydantic AI has made that failure mode unusually legible. Its public workflow guide describes a set of gh-aw agentic workflows: people edit Markdown sources, those sources compile into generated .lock.yml files, and GitHub Actions runs the locks. That last fact changes what deserves testing. The readable prompt is not the complete program; the compiled workflow graph is.
In a fresh public change, the project added a static policy guard and a substantial test suite around that graph. The work is not a generic “lint your YAML” gesture. It is a catalogue of configurations in which a coding-agent system could look active from the outside while its intended work was never reachable.
The failure begins with an empty value that looks harmless
The new guard calls out one especially sharp case. In a GitHub Actions condition, a job can refer to needs.detect.outputs.touched without declaring detect as a dependency. The expression resolves empty. The guarded job skips. A skipped job can still report success.
For ordinary CI, that is already a problem. For an agentic workflow, it is worse: the team may think a review, triage, or investigation agent received the assignment when the execution path never reached it. The public source lists other versions of the same trap: a generated lock left stale after its Markdown source changed; a prompt pointing the agent’s file tool outside its readable workspace; output bounds that discard results; and a job timeout that disagrees with the timeout the agent runtime actually receives.
Each one lives in configuration, but configuration is execution. A convincing prompt cannot repair a graph that never schedules the agent or a lock that never incorporated the prompt change.
The engineering receipt is the test’s memory
The stronger move is not simply that Pydantic AI wrote a guard. Its regression tests reconstruct the public pre-fix shapes the guard is meant to catch. One fixture recreates the missing dependency: the activation job checks needs.detect, but its declared dependencies omit detect. The test asserts that the guard reports the dangling reference, then supplies the repaired graph and expects no violation.
That is a durable kind of engineering receipt. A policy name such as “workflow reliability” is too vague to defend. A fixture that preserves the exact path by which a security review skipped while the check stayed green gives the next maintainer a concrete thing not to reintroduce. It also makes the policy tool auditable: the reader can see the bug shape, the rule, and the expected repair.
The same pattern runs through the guard. It examines expressions in conditions, outputs, environments, action inputs, and interpolated run steps; checks explicit output maxima; flags prompt paths that an agent’s file tools cannot read; requires matching timeout declarations; detects mixed compiler versions among generated locks; and requires a regenerated lock alongside a changed workflow source. These are small rules with a shared purpose: turn silent no-ops into visible failures before they consume model budget or create a false sense of coverage.
Make the system prove it reached the work
The transferable skill is to give every invisible automation defect a home at the layer where it occurred. Do not respond to a skipped agent run by only rewriting the prompt or adding a dashboard counter. Preserve the broken compiled graph as a test fixture. Write the smallest check that rejects it. Keep a corresponding clean graph so the check does not become a blunt ban. Run the guard against the artifact the platform actually executes.
- When an automated agent fails quietly, save the configuration shape—not merely the incident summary.
- Identify whether the failure was in source authoring, compilation, dependency wiring, runtime budgets, tool access, or result handling.
- Recreate that shape in a focused regression test and assert a specific, actionable violation.
- Test the repaired version too, so a valid workflow is not rejected by accident.
- Place the guard where it can inspect generated or compiled artifacts, not only the friendly source file.
This does not make coding agents correct, safe, or useful by itself. It does something more basic and often more neglected: it prevents a team from mistaking the absence of a failure signal for evidence that the agent had a chance to work at all.
Pydantic AI’s source code establishes that the project is hardening a documented agentic-workflow system, not that every hosted run or release has these protections. The next public signal worth watching is whether the project adds end-to-end fixtures that prove a real agent job starts, receives the intended context, and emits a bounded result after this static gate passes. Until then, the lesson is still valuable: a green check is an observation about a workflow state, not a witness statement from the agent.