VAMSHI.

Agentic Engineering

The Harness Is the Product

A production coding agent is the model plus everything that makes its work bounded, repeatable, testable, and safe.

5 min read
coding agents ai architecture developer platforms applied ai agent reliability
Seven system layers surround the coding model: task contract, context, tools, state, permissions, verification, and observability.

A model can write a convincing patch in thirty seconds.

That is not the same as safely changing a production codebase.

Between “here is the issue” and “this is ready to merge” sits a system that chooses context, exposes tools, limits permissions, preserves state, runs checks, records evidence, and knows when to stop. That system is the harness.

The model determines what is possible. The harness determines what happens repeatedly.

Follow one bug all the way through

Suppose a customer reports that the last page of an invoice table is empty.

A chat demo can read the component and suggest an off-by-one fix. A production workflow has more work to do:

  1. Resolve the repository, branch, and package that own the table.
  2. Load the local engineering rules and relevant prior decisions.
  3. Reproduce the failure with a focused test or browser trace.
  4. Make the smallest change that fixes the observed behavior.
  5. Run the focused test, neighboring tests, type checks, and lint checks.
  6. Inspect the diff for unrelated edits and generated-file churn.
  7. Return the patch with evidence, uncertainty, and any checks that could not run.

If the database fixture is missing, the system should say so. If the apparent UI bug requires a schema migration, it should stop and widen the conversation before widening the diff. If the test process times out, “I ran the tests” is not an acceptable final line.

None of this requires the model to become more eloquent. It requires the product around the model to be explicit.

The seven parts of a useful harness

You can draw the boundary in different places, but a production harness usually owns seven concerns.

1. A task contract

The contract states the outcome, relevant scope, constraints, acceptance checks, and stop conditions.

“Fix pagination” is an invitation to guess. “Reproduce and fix the empty final page in the invoice table; preserve the API; add a regression test; stop if the fix requires a data migration” is a task an engineer can review.

2. Context assembly

The harness decides which files, errors, docs, examples, and live records enter the working set. It should be able to explain why each item is present.

Dumping an entire repository into context is not generosity. It is a refusal to retrieve.

3. Tools with narrow contracts

Tools should expose useful operations: search the repository, run the correct test package, inspect a browser state, query an approved service, or create a branch. Inputs should be validated. Outputs should make success and failure distinguishable.

A general shell is powerful, but a typed run_package_tests(package, pattern) tool is easier to govern and easier to evaluate.

4. State that survives the conversation

The transcript is helpful context. It should not be the only task database.

Persist the starting revision, current phase, artifacts created, checks run, approvals received, and unresolved blockers. That state is what makes recovery possible after a process crash or a human handoff.

5. A permission envelope

The workflow should know which files it may change, whether it may use the network, which credentials it can access, and which actions require approval.

Permissions should follow the task’s blast radius, not the model’s confidence.

6. Verification

The agent needs a way to see whether its action worked. For code, that may combine schemas, types, unit tests, browser behavior, diff review, and human judgment.

The verifier is not an ornamental final step. It changes how the agent works. A system that must prove the bug before and after the change behaves differently from one rewarded for producing plausible code.

7. A trace and a review packet

Operators need to know what the agent tried, what changed, which checks passed, how much the run cost, and why it stopped. Reviewers need the compressed version: patch, evidence, risk, and next action.

Raw tool logs are not a handoff.

Put durable knowledge in the right place

A harness improves when repeated failures become durable assets.

  • A repository convention becomes an AGENTS.md rule.
  • A recurring procedure becomes a skill.
  • A risky manual action becomes a narrow tool.
  • A production escape becomes an eval case.
  • A permission near miss becomes a tighter boundary.
  • A wandering run gets a budget and a stop condition.

This is more useful than appending another paragraph to the system prompt. The failure changes the system, not just the next conversation.

OpenAI’s current Codex best-practices guide follows the same progression: clear task context, durable repository guidance, configuration, tools, skills, verification, then scheduling for workflows that have already become stable.

The model is not your architecture

The SWE-agent paper found that the interface between an agent and its computer materially affected performance on software-engineering tasks. That result is easy to underestimate. It means capability is partly a property of the environment we design around the model, not only the weights we call.

This is also why model upgrades can disappoint. A stronger model inside a noisy context pipeline, an ambiguous tool surface, and a weak verifier may simply fail faster or produce more code for a reviewer to unwind.

When evaluating an agent system, ask:

  • Can it prove the requested behavior changed?
  • Can it fail without corrupting shared state?
  • Can another person resume or review the work?
  • Can a repeated mistake become a rule, tool, or eval?
  • Can we swap the model without rebuilding the workflow?

If those answers are weak, the missing capability is probably not another prompt.

The harness is where a model’s intelligence becomes an engineering system.

Further reading