start

The verification gap

How do we know agent work is correct?

In one minute

Passing tests is not a merge signal. In a recent maintainer study, roughly half of test-passing AI pull requests would not have been merged.
Writing code stopped being the hard part. Proving a change is right is the bottleneck.
The state of practice fits in 36 checkpoints across eight phases, from intake to organization-level controls.
Stageway is built against those checkpoints and publishes its own scorecard, gaps included.

Passing tests is not a merge signal

In March 2026, METR asked four active maintainers from three SWE-bench Verified repositories to review 296 AI-generated pull requests. Roughly half of the pull requests that passed the benchmark's tests would not have been merged into main, even after adjusting for noise in the maintainers' own decisions. On average, maintainer merge decisions ran about 24 percentage points below the automated grader. The reasons were ordinary review reasons: the core functionality did not really work, the patch broke other code, or the code quality was not acceptable. (METR, "Many SWE-bench-Passing PRs Would Not Be Merged into Main")

METR is careful about what this does and does not show. The agents never had the chance to iterate on feedback the way a person would, so it is not a ceiling on capability. It is a warning about reading a green result as a finished result.

A second study looked at real agent pull requests in open source. The AIDev analysis found that 46.41% of fixes proposed by Copilot, Devin, Cursor, and Claude were rejected. Rejected fixes were incorrect or incomplete, failed CI, never got implemented at all, or solved something nobody had prioritized. Every one of them still took human review time. (AIDev, "Understanding the Rejection of Fixes Generated by Agentic Pull Requests")

Both point the same way: the scarce work is no longer generating a diff. It is knowing, with evidence, that the diff is right. That is the problem Stageway is built around.

The 36 checkpoints

A checkpoint is a place in the lifecycle where a claim about the work must be backed by something observable. The list below is the rubric Stageway is scored against, grouped into eight phases. Each item links to its row on the scorecard, where you can see how Stageway addresses it and what remains.

Intake

IN-1 Spec has a start locus and a completion artifact Implemented
IN-2 Ambiguity surfaced before the first write Implemented
IN-3 Task prioritized before assignment External
IN-4 Task size bounded Partial

Context

CTX-1 Agent instruction file at the repository root, in version control Implemented
CTX-2 Instruction file pruned, not just grown Partial
CTX-3 Environment declarative and snapshotted Partial
CTX-4 Docs and architecture layers machine-checked Partial
CTX-5 Agent discoveries captured as skills and scripts Implemented

Planning

PLN-1 Read-only explore phase before execution Implemented
PLN-2 Plan reviewed by a non-author Implemented
PLN-3 Planning and implementation contexts separated Implemented
PLN-4 Vertical slices, each independently testable Partial

Execution

EXE-1 Unattended runs inside an isolation boundary Partial
EXE-2 Network egress allowlisted Gap
EXE-3 Untrusted content sanitized before the model Partial
EXE-4 Risk tiers gate what runs automatically Partial
EXE-5 Parallel agents in isolated checkouts Policy choice

Verification

VER-1 Agent-runnable check per task Implemented
VER-2 Completion gated deterministically Implemented
VER-3 Fresh-context reviewer on the diff Conditional
VER-4 Behavioral verification for user-facing changes Conditional
VER-5 Agent-written tests reviewed adversarially Implemented
VER-6 SAST, secret, and dependency scans before review Partial

Review and merge

REV-1 Named human approver, no self-merge External
REV-2 PR carries evidence, not assertions Partial
REV-3 Review capacity tracked as a constraint External
REV-4 Auto-fix loops bounded Implemented

Post-merge

PST-1 Agent authorship labeled Policy choice
PST-2 Sessions instrumented end to end Partial
PST-3 Failures feed back into instructions Partial

Organization

ORG-1 Stability metrics segmented by provenance External
ORG-2 Spend monitored, not incentivized Partial
ORG-3 Agent credentials scoped and audited Gap
ORG-4 Production access gated in multiple layers External
ORG-5 Skills inventoried and evaluated Partial

What changed in the last 12 months

Shift	Stageway's response
Verification became the named bottleneck, not code generation.	Evidence gates at every stage. The runner executes the checks itself.
Asynchronous runs overtook interactive sessions. If you weren't watching, the run must come back with proof.	Runs come back with evidence on disk, not a chat transcript.
"Harness" became an engineering discipline. Planning, isolation, and verification are infrastructure, not a prompt.	The runner is ordinary, readable Python code.
Sandboxing moved from advice to product default.	Restricted mode uses provider controls, and the security boundary says plainly what trusted mode is not.
Coordination replaced parallelism as the interesting problem.	One runner per project, and a short queue by design. See review and everyday use.
Agents entered the review seat.	An independent review stage that never sees the implementation's reasoning. People still approve and merge.

Open questions, and where Stageway stands

Some questions have no consensus yet. Stageway takes a position on each, and you can change most of them per project.

Question	Stageway's position
How heavy should the spec process be?	Two lanes. A small change gets a concise plan and runs in one session. A broad change gets four plan files and a validated contract. See lanes.
Should unattended agents skip permission prompts by default?	Trusted execution is the offered default, backed by outcome checks after every stage. Restricted execution is available. Neither is called a sandbox.
Does parallelism help, or just relocate the queue?	One runner per project. Worktrees isolate checkouts; they are not a throughput target.
Do agent-written tests count as verification?	Only after they fail on an assertion before the code exists. Then they are frozen.
What does "good" look like in numbers?	Evidence is recorded per task. Stageway invents no targets.
Does autonomous end-to-end testing generalize?	Disposable environments with observed readiness for local runs, and `test-on-staging` for deployed behavior. Screenshots are not required.

Method and sources

This page synthesizes platform documentation, practitioner writing, and independent research on agentic software delivery, as of Q3 2026. The checkpoint rubric condenses that material; where a practice is pushed mainly by a vendor selling the capability, treat it as [vendor-led] and weigh it accordingly.

Sources cited on this page:

METR, Many SWE-bench-Passing PRs Would Not Be Merged into Main, 10 March 2026.
AIDev, Understanding the Rejection of Fixes Generated by Agentic Pull Requests: Insights from the AIDev Dataset, June 2026.

Checkpoints VER-2 REV-2 REV-3