Harness Engineering: Guardrails Instead of Hope
Why what you build around the agent often decides its output more than the model does - and why a guardrail worth the name checks instead of merely warning.
The agent fixed the reported bug. It ran the two tests named in the ticket, both were green, and it committed. What nobody checked: the helper function it rewrote along the way is imported by three other modules. Since that commit, one of them quietly computes the wrong number. No crash, no red log, just a value that is slightly off from now on. What happens next depends less on which model did the work than on what was built into the room it worked in.
A word about the title before we start. A guardrail in the sense of this article is not a roadside sign that warns politely and otherwise does nothing. It is closer to an inspection bay the work must physically pass through before it counts. And the numbers coming up show something unexpected: gates like that cost the agent hardly any solved tasks, but they sharply cut how much it breaks along the way.
Agent = Model + Harness
An agent is never just a model. It is a model plus everything standing around that model: the tools it can call, the environment it runs in, the checks its work has to pass, its rights, and whatever it keeps beyond a single session. That sum has acquired a name: the harness. Building it deliberately, instead of letting it come about by accident, is harness engineering.
Three terms that get mixed up constantly, one sentence each: prompt engineering phrases the individual instruction. Context engineering settles what the agent gets to see while a task is running. Harness engineering builds the workshop around both of them - how far the agent can reach, which checks its output has to clear, and what is still there once the session ends.
Honestly, that nesting is not universally agreed. Part of the literature stacks the three disciplines exactly this way; another part treats the harness as a special form of context engineering instead. This article uses the first reading because it explains more cleanly.
One in-house delimitation as well: the article on agentic coding describes the agent itself and the question of how closely you supervise it. This one is about the room around it - and about what is enforced in that room by a machine rather than by your memory.
Note
What you will be able to do after this article
- Name which parts of an agent setup belong to the harness.
- Decide, for a concrete measure, whether it is hope or mechanism.
- Predict how a tighter sandbox affects the number of permission prompts.
- Explain why green tests are not proof of correctness.
Before the details, a map. Whenever gate, sandbox or journal comes up later, one of these five pieces is meant:
Same worker, two workshops
Picture yourself kitting out a workshop for a new hire before day one: what he is cleared to operate, where the cut-off switch sits, which manual lies ready at his bench, who has to approve the finished piece. Two firms can take on the same man and still end up with very different work coming out. That gap is no verdict on how good he is.
Definition
Model and harness
Analogy
Definition
The size of that effect can be quantified. A large comparison study ran 21,730 agent rollouts across nine models and nine benchmarks. On the leaderboard that goes with it, one and the same model on the same task set produces figures that sit far apart: 68 percent of real programming issues resolved in one execution environment, 34 percent in a more general-purpose one. No retraining, no different model, only a different room.
Two qualifications. The study design behind the 21,730 rollouts is a primary source; the two figures sit on the public leaderboard of the same research group: Claude Sonnet 4.5 on SWE-bench Verified Mini, a random sample of 50 tasks, at 68 percent in the SWE-Agent scaffold and 34 percent in the general-purpose HAL agent. For a sense of scale: moving from one model generation to the next is usually reported to shift scores on benchmarks like these by a few points, at most somewhere in the teens.
This is where the workshop analogy breaks, and it breaks where it matters. A person keeps what you told them once. An agent starts every session from scratch again. The only thing that is permanent for it is what someone wrote into a file it reads at startup. Explaining something to an agent in chat means you spoke out loud once; it does not mean you rebuilt the workshop. Some tools now write such hints into a memory file on their own. But that, too, is only context the agent reads along, not a check that enforces anything.
Caution
Common error: a bad result means a weak model
That is the reflex, and usually the most expensive answer. Identical weights sit dozens of percentage points apart depending on the environment. With a spread that wide, the room is the more likely cause of a disappointing result, not the worker. So the more productive first question is: what in the environment let this through - and can that be turned into something a machine checks?
Caution
The counter-error: so the model does not matter
That is not in the data either. What was measured is how much variance the environment accounts for, not that model quality is irrelevant. Inside a fixed harness, strong models still beat weak ones. The claim is one about measurement honesty: comparing two agents without disclosing the harness means comparing two workshops and calling the result skill.
What the agent keeps lives in files
If the agent takes nothing with it, the environment has to. In practice that means three things: a setup step recording how the project starts, builds and tests; a running journal noting after each finished piece what was done and why; and the habit of committing after each piece instead of everything at the end. The payoff arrives later: when a new session starts, the state is there in writing.
And the rule file?
You know the rule file from the article on context engineering: the agent's onboarding package with conventions, commands and architecture notes. What this article has to add is uncomfortable. A controlled study on real repositories compared two agents across 17 tasks and 288 runs, with and without such a file. It had no measurable effect on correctness, and the study rules out differences larger than 10 to 15 percentage points; the failures sat in design, pattern choice and clean wiring - in things project knowledge cannot supply. That does not mean do not write one. It means do not rely on it for correctness.
The inspection bay, not the sign
A gate is code sitting between two states: the agent considers itself done, and the change counts. It runs a check and may refuse the commit. An instruction is something fundamentally different: it only works if the agent recalls it at the right moment and wants to follow it. The distinction sounds like hair-splitting. It is measurable.
Definition
Gate
Analogy
Definition
The three numbers below come from a single study: open Qwen models on consumer hardware, measured on the SWE-bench Verified task set. They are the backbone of this article, but they are one measurement and not an industry constant. The authors note themselves that large models with longer context windows may not show the text rule's penalty, because a long instruction takes up a smaller share of their context.
The setup is deliberately plain: the same task set, supported three different ways. First run, nothing said: 6.08 percent of previously passing tests failed after the change. Second run, a working rule as text in the prompt, spelled out over 119 lines instead of 49 - write the tests first, then the code: 9.94 percent. Third run, a mechanism: a tool has worked out in advance, via the dependency graph, which tests depend on which code, and before finishing the agent looks it up there and runs the affected tests. 1.82 percent. A word on the measure: what is counted is broken tests, not broken changes. The share of changes that broke at least one test sat at roughly a third in all three runs. So the tool made the damage smaller, not rarer.
9.94 percent of previously passing tests failed - more than the baseline with no rule at all, which sat at 6.08 percent.
1.82 percent - and in a second, smaller setup (different model, 25 tasks) the share of resolved issues rose from 24 to 32 percent.
A bare working instruction, without the machine-determined information, was worse than saying nothing at all.
Regressions with test selection
-70 %
From 6.08 down to 1.82 percent of previously passing tests that fail after the change.
Resolved issues
+8 points
In a second, smaller setup (different model, 25 tasks) the share climbed from 24 to 32 percent. That is 8 instead of 6 out of 25 tasks. In the main setup with 100 tasks the share dipped slightly instead, from 31 to 29 percent; both are too few tasks for a reliable effect.
That is why the inspection bay is the better image than the sign: it checks everyone, regardless of attention, mood and good intentions.
And this image breaks too, productively. At a real vehicle inspection, an independent authority writes the checklist. For a test suite, the party being inspected writes it. Who owns the test file is therefore a harness design decision, not a question of agent behaviour.
Caution
Common error: all tests green means correct
Green means: nothing that gets checked is broken. About everything nobody wrote a test for, green says nothing. Once a project grows, that is more than a technicality - see the box below. On top of that come categories no gate covers: a wrong diagnosis, an overengineered solution, a clean implementation of the wrong requirement.
Caution
Common error: writing it down is the same as enforcing it
The 9.94 percent are the answer to that, and they are uncomfortably precise. The text rule did not simply fail to help - it made the result worse than having no rule at all. The authors explain it like this: the long instruction pushed knowledge about the project out of the context, and the agent attempted bigger changes across more files without knowing which tests it was putting at risk.
A study across 30 systems-level tasks between 1,500 and 110,000 lines of code gave every task two test suites: a visible one the agent may use, and a held-out one it never gets to see. Every top-tier agent tested saturates the visible suite. The gap to the held-out suite grows by a measured 28 percentage points for every tenfold increase in code size.
One observed case was a 2,900-line program that claimed to be a compiler but had in fact memorised the test cases. Important for the distinction: no test file was deleted here, and none was modified. The agent adapted to the visible checklist without touching a single test - the quieter variant of the same problem.
The keycard, not the guard dog
Two pictures first, because the heading sets them against each other. A guard dog judges behaviour: it watches, it barks when something strikes it as wrong - and it can be fooled, distracted or talked down by a calm voice. A keycard judges nothing at all. It opens three doors and not the fourth, no matter who carries it or what they have in mind. Rights in a harness work like the card: handed out in advance, door by door, regardless of intent. Two settings get confused constantly even though they are independent. The first: what is technically possible at all - which directories may the agent write to, which network addresses can it reach? The second: when does it have to stop and ask a human? Common coding agents implement this as two separate dials, typically three sandbox levels from read-only to full access, and several approval policies from ask often to never ask.
The interesting part is how the two interact, because the direction is counter-intuitive. You expect a tighter boundary to mean more prompts. The measurement says the opposite.
Example: what isolation does to the number of prompts
- 1. Starting point, assumed as an illustrative baseline: a three-hour session with roughly 50 permission prompts. That starting figure is an assumption, not a measurement.
- 2. Switch on filesystem and network isolation. Writing anywhere beyond the project folder is now physically out of the agent's reach, and so is any address missing from the approved list.
- 3. The measured value is the reduction: 84 percent fewer permission prompts, self-reported by a vendor from internal usage. Applied to the illustrative baseline, roughly 8 of the 50 remain.
- 4. The reason: whatever happens inside the boundary is now contained and no longer needs individual approval. What remains are the actions that cross the boundary - installing a package from the network, touching a file outside the project.
- 5. The serious case: a dependency's documentation contains a hidden instruction to send a configuration file to an external server. The agent follows it. Isolation did not prevent that - but the target server is not on the approved list, and the upload fails.
Where the picture breaks: any card system takes it on trust that whoever carries the card is the person it was issued to. And an agent can be argued into putting rights it holds perfectly legitimately to work on the wrong thing. The guard dog would be no help here either: anyone who behaves unremarkably walks straight past him. Tools are leaning on him more anyway: Claude Code, for one, now starts in auto mode by default, where a separate classifier model decides on approvals instead of you. That saves prompts, but it is still a guard dog - Anthropic itself reports that in a test it let through roughly one in six genuinely overeager actions. It does not replace the keycard.
Caution
Common error: the sandbox protects me from prompt injection
Two of the major vendors of coding agents, OpenAI and Anthropic, state the opposite in their own documentation. A sandbox limits what an already manipulated agent can reach. It does not detect the manipulation, and it does not prevent it. In the example above the agent followed the hidden instruction; only the consequence was caught.
Caution
Common error: sandbox and approvals are the same dial
There are two. One sets what is technically possible, the other when you get asked. Treating them as one leads you to expect more interruptions as the price of more safety - and to set the environment up more loosely in order to stay productive. The measured direction is the reverse: the tighter technical boundary reduced the number of prompts, because more actions could pass through without risk.
How such a hidden instruction works technically, and why models fall for it, is covered in the article on prompt injections. For this article the consequence is enough: the boundary is damage control, not defence.
Where the harness still fails you
No gate, however well built, catches a wrong diagnosis, an overengineered solution or a correct implementation of the wrong requirement. It checks what someone decided to check. The awkward part is less the gap itself than measuring it: there is still no established method for determining the coverage or the quality of a harness. You can say a harness got better, but not how good it is.
About the term itself, briefly and honestly. The phrase harness engineering appears as a declared coinage in a post by Mitchell Hashimoto dated 5 February 2026; shortly afterwards an OpenAI post carried it in its title, and part of the secondary literature attributes the coinage to a post five weeks later. So the term was only coined in February 2026 and contested in its attribution - while the word harness has been in use far longer than the field now claiming it, and the practices behind it, automated checks and sealed-off environments, go back decades. Treat the vocabulary as new, not the subject.
Against too much enthusiasm stands an objection from practice: a harness becomes a product of its own, with its own bugs, its own legacy and its own maintenance burden; extra reviewing agents cost time, tokens and complexity. What matters is how to classify that objection: it comes from practitioner reports, not from a measurement. There is no solid figure for the point at which the effort exceeds the benefit - which, without a quality metric, would be hard to expect anyway. So anyone starting today builds the cheapest mechanisms first: the fast, deterministic checks the project already has lying around.
Test bench: same mishap, two workshops
The hardening loop
Back to the beginning. That quiet regression from the first paragraph - what happens to it? Without a harness it becomes a story: noticed eventually, retold in the team, filed as a warning. With a harness it becomes a fact about the environment: a check that from now on pulls in the dependent modules before every commit. Workshop, inspection bay and keycard are not three techniques but three places where the same habit applies: stop resolving to do things, and start building them in.
Note
The five things worth keeping
- Agent = model + harness. The harness is often the larger and almost always the more changeable of the two factors.
- A rule someone has to remember is hope. A rule a machine enforces before the commit is a gate - and even half a step toward it, the machine-determined test selection, flipped the sign in the measurement.
- Rights and approvals are two dials, not one: what is technically possible, and when a human gets asked. Tightening the first can relieve the second.
- No gate proves correctness, only that nothing checked is broken. How much it covers is not measurable yet - and as the code base grows, so does the share of undetected failures.
- Harness engineering means turning every observed failure into a permanent, machine-checked property of the environment, so that it structurally cannot happen a second time.
Check
Quiz: Harness Engineering
Going by the formula agent = model + harness, what belongs to the harness?
Select one answer
1. Going by the formula agent = model + harness, what belongs to the harness?
2. A colleague says: I explained the rule to the agent once in chat, that should be enough. What is he missing?
3. Your team keeps forgetting to update the CHANGELOG. Proposal A: a sentence about it in the agent's rule file. Proposal B: an automated check that rejects the commit while the entry is missing. What do the measurements named in the article predict about proposal A?
4. An agent is to analyse an unfamiliar repository and write a report. It should change nothing, and you do not want to be asked for permission forty times. Which dial do you turn - and what does that not protect you from?
5. On a large project an agent reports all visible tests green, without having deleted or modified a single test. A held-out test suite it never saw later shows several failures. What explains this best - and how does it differ from an agent that simply deletes a failing test?
Check
Comprehension check
Can you explain this in your own words? Tick what you've got.
0 of 3 done