VAMSHI.

Agentic Engineering

Verification Is the Real Agent Loop

A practical validation stack for coding agents, from cheap deterministic checks to behavioral tests, graders, and human review.

5 min read
ai evals coding agents software testing agent reliability quality engineering
Verification ladder rising from schema and scope checks to tests, behavioral evaluation, model graders, and human review.

The most dangerous coding-agent failure is not broken syntax.

It is a clean diff, a confident explanation, and a green test command that never exercised the requested behavior.

Generation produces a candidate. Verification decides whether that candidate can enter the product.

“Tests pass” is not a result

Imagine an agent fixing an expired-session bug.

It changes the token-refresh helper, adds a unit test for the new branch, and runs the package suite. Everything is green. In the browser, the application still logs the user out because a service worker caches the original 401 response.

The agent verified its implementation. It did not verify the user’s problem.

A useful acceptance check connects the request to observable behavior:

  1. Reproduce the expired-session flow before the change.
  2. Prove the new test fails against the old behavior.
  3. Apply the fix.
  4. Confirm refresh succeeds in the real request path.
  5. Confirm an invalid refresh token still logs the user out.
  6. Inspect the diff for unrelated auth changes.

Now the evidence matches the claim.

Use a verification ladder

Run cheap, objective checks early. Spend human attention only where judgment is actually required.

Level 1: shape and scope

Did the output satisfy its schema? Did the agent touch only allowed files? Did it add secrets, large binaries, or generated artifacts? Did it preserve an API it was told not to change?

These checks are fast and deterministic. They should fail loudly.

Level 2: static correctness

Formatters, linters, type checks, dependency rules, and security scanners catch mechanical defects. They do not prove the feature works, but there is no reason to spend a senior reviewer’s time finding an import cycle a tool can identify.

Level 3: focused tests

The smallest relevant test should demonstrate the change. A good regression test fails before the fix and passes after it.

Agent-written tests deserve scrutiny. A model can weaken an assertion, mock away the real failure, or test the helper it just invented rather than the public behavior.

Level 4: integration and behavior

Exercise the path a user or downstream system actually takes: browser interaction, API boundary, migration, queue, or external tool.

This is where hidden coupling appears.

Level 5: semantic review

Some criteria are difficult to encode: Is the diff unnecessarily broad? Does the explanation match the code? Is the migration operationally safe? Does the solution fit the architecture?

A model grader can help triage these questions. It should be calibrated against human-labeled examples and treated as another instrument, not ground truth.

Level 6: accountable judgment

High-impact changes still need a person who can accept the risk. The reviewer should receive evidence and uncertainty, not a transcript and a request to “take a look.”

Evaluate the workflow, not only the model

A production eval should exercise the complete path:

task intake → context selection → tool use → change
→ verification → final handoff

Start with 20 to 50 cases from real work. That is enough to expose failure patterns without creating an evaluation program nobody maintains.

Include several kinds of cases:

  • ordinary changes the system should complete
  • ambiguous requests where it should ask or narrow scope
  • missing-context tasks where it should stop
  • risky changes that require approval
  • known production failures
  • tasks where doing nothing is correct

Score more than final correctness. Track scope discipline, test quality, evidence reporting, security, reviewability, time, and cost. A system that solves the issue by rewriting half the module may pass a functional test and still fail as an engineering tool.

Keep a held-out set

Once an agent fails a case, the team will improve the prompt, rules, tool, or model around it. That case is now part of the development loop.

Keep other cases held out. Otherwise the eval becomes a list of incidents the harness has memorized how to avoid.

SWE-bench helped make this concrete for coding agents by evaluating systems against real GitHub issues in executable repositories. The deeper lesson is not the leaderboard. It is that the environment and tests create a checkable unit of work.

Turn production escapes into regression gates

When a failure reaches a reviewer or user:

  1. Reduce it to the smallest faithful reproduction.
  2. Identify the failure class, not only the exact diff.
  3. Add variants to the eval set.
  4. Change the appropriate layer: instruction, context, tool, permission, verifier, or model.
  5. Replay the broader suite before release.

If an agent edits generated files once, the regression is not “never edit this one file.” It is “recognize generated artifacts and require the supported regeneration path.”

This is how reliability compounds instead of becoming a longer list of warnings.

Return an evidence packet

The final answer should separate:

  • behavior reproduced
  • files changed
  • checks completed successfully
  • checks attempted but unable to complete
  • checks not run
  • known uncertainty
  • decisions still requiring a person

That format has a useful side effect: it makes unsupported confidence visible. “All tests pass” looks different when the full suite is listed under “not run.”

A capable generator makes work. A capable verifier makes the work reviewable.

Further reading