Reading the tests your agent wrote
What all tests green is actually worth - and how to judge a test yourself, without writing any code.
The agent reports: 47 tests written, every one of them green. Take a moment and be honest about what you now know regarding the software. The honest answer is: almost nothing. Not because the number is a lie, but because you have never once seen what those 47 tests actually check. And nobody has told you so far that you could.
That is the turn this article makes. A test is text. By convention its name reads as a sentence in something very close to ordinary language, and the line that decides everything names a single value. You can read both parts without writing any code. The article on agentic coding describes the worker; the article on harness engineering describes the room around him. This one is about the work product: the test file you can open. Close the article and you will have looked at two genuine tests and picked the one worth keeping.
What a test claims
A test writes down one claim about how the software behaves: a name that states the claim, and a line naming the value which would prove it. Someone who cannot program can read both halves, and both halves settle how much the test is worth. Reading them is therefore the oversight still open to you when the code itself is beyond your judgement. It matters at exactly the spot where you are standing: one and the same agent produced the software, then produced the checks that judge it.
Note
What you will be able to do
- Read the two pieces of a test that call for no programming knowledge, its name and its expected value, and state which behaviour it claims.
- Decide whether a given check is strong or feeble, and name two wrong results it would let past.
- Explain which claim a coverage figure supports and which it does not, and why it carries the least weight when one agent produced both code and tests.
- Spot a softened check in a before-and-after pair, and name the one verification you can ask for without any tooling at all.
A test is a question with a known answer
You already met the examination image in the article on agentic coding. Here we rotate it a quarter turn further: the examiner is not the interesting part, the question paper is. Because a question needs two things before anyone can mark it. First, it has to state what is being asked. Second, the answer that counts as right has to be fixed beforehand. Take either away and the question stops being an examination and becomes decoration.
A test has precisely those two parts - and they are precisely the two you can read without programming knowledge. The name of the test is the wording of the question. The check holds the correct answer, fixed before anybody tried it.
Definition
Test
In plain words
Technically
Example: the official four-liner
The getting-started guide of pytest, one of the most widely used testing tools for the Python language, opens with a file four lines long. We take it exactly as it stands there:
# content of test_sample.py
def func(x):
return x + 1
def test_answer():
assert func(3) == 5> assert func(3) == 5
E assert 4 == 5
E + where 4 = func(3)Read it from the top. The file is called test_sample.py, and the file name alone is already a declaration: the tool automatically collects everything carrying test_ at the front or _test at the end. Then comes a tiny function that adds one to whatever it is given. So 3 turns into 4. And then the claim: 3 should turn into 5. It does not, the test goes red, and the output reads assert 4 == 5, with the origin of that 4 underneath. This error message needs no explanation. It is a comparison between two numbers that do not match.
That also answers the question of where these files live at all. They carry their purpose in the name so the tool can find them - so nothing stops you from asking for them by that name.
def test_rabatt():
assert berechne_rabatt(150) is not NoneThe name says only that something about discounts got checked - never what. The check forbids exactly one single case: that no value comes back at all.
def test_rabatt_ab_100_euro_ist_10_prozent():
assert berechne_rabatt(150) == 15The name is a sentence you can hold against the outcome. The check names one single value, and you can do the arithmetic yourself: 10 per cent of 150 euros is 15 euros.
The weak test fails in two places, and the two have nothing to do with each other. First the name: discount checked is not a promise, it is a topic. Second the check: is not None means in Python merely that some value came back - None is the Python word for no value at all. So the number 0 goes through, -99 goes through, and the word banana goes through as well. A check that can barely fail is barely checking.
Both columns are runnable pytest once one thing is added: berechne_rabatt lives in the program itself, not in the test file. The test file fetches the function from there. That is why the file holds only the question and the expected answer, not usually the calculation.
Caution
Common error: only programmers can read tests
The wording of the question is everyday language in disguise. The unit testing chapter of Software Engineering at Google (O'Reilly, 2020) prints example names such as shouldNotAllowWithdrawalsWhenBalanceIsEmpty. You just read that without understanding a single line of code. That skill alone carries half the judgement.
Two rules you can apply to the name alone, both from the same chapter. First: a strong test name reads like a full sentence and states both parts - the situation and the outcome expected of it. Second, and this one bites harder: if a name needs the word and, the test is probably examining two things at once and ought to be two tests. Neither rule requires you to open anything except the list of names.
That leaves the quieter error: that a passing test proves the function works. It proves it for the case its name mentions. For no other. Which is why the name is the load-bearing half - it also tells you what the test stays silent about.
Three sizes of test
Tests come in three sizes. A narrow test examines one single piece on its own. A middle one asks whether a couple of pieces work together. A broad one runs the entire application end to end, the way somebody using it would.
Narrow / middle / broad
80 / 15 / 5
The target mix in per cent from Software Engineering at Google (O'Reilly, 2020). The shape behind it was described by Martin Fowler in 2012 as the test pyramid; he credits it to Mike Cohn's 2009 book. The book calls this scope; there, size means something else, namely where a test may run.
Why so lopsided? Broad tests run slowly, cost a lot to write and wobble - a suite made only of them will crawl and raise false alarms. A suite made only of narrow ones examines every part and never the assembly.
One more word you will run into: regression. It means the most unpleasant case there is - something that worked yesterday does not work today, and nobody notices. Tests that stay in place are what stands against it.
The cleaner who visited every room
Green you already know. The figure next to it you do not. Almost every report on machine-written tests carries a second measure: coverage, usually as a percentage. It sounds like thoroughness, and it measures something quite different.
Definition
Test coverage
In plain words
Technically
Think of a cleaner working through a room list. At the end of the shift the sheet tells you which rooms she entered. What the sheet cannot tell you: whether anything in any of those rooms got cleaner. A list full of ticks is just as compatible with a spotless flat as with one where nobody picked up a cloth.
The half-truth still stands, and you should keep it: a room nobody entered is certainly not clean. A line no test ever executed is certainly not checked. So coverage marks a floor under how much attention the program received. Not a seal of quality, but not nonsense either.
Example: what 100 generated tests are worth
In late 2023 Meta turned a tool called TestGen-LLM loose on its own Instagram and Facebook code bases and published the yield in 2024. For two Instagram products the funnel is reported in full: out of every 100 machine-produced test cases, roughly 75 could be compiled at all. Roughly 57 ran through dependably. And roughly 25 reached code that no test had touched before.
Before you carry those figures away, here is where they come from: one single measurement out of one industrial deployment in late 2023, one tool, one code base. Not a law of nature about your agent - but a well-documented bearing from practice.
And now the uncomfortable part. That third measure is exactly the coverage figure this section has just been mistrusting. Code reached is not the same as behaviour checked. So even by that generous yardstick, about 25 out of 100 tests are left standing - and how many of those actually claim anything is something this measure does not say either.
A second detail weighs more than the numbers themselves. Meta did not pick out the usable tests, it had them sifted automatically: a filter behind the model discarded everything that produced no measurable improvement, before any human looked. The 73 per cent that engineers then accepted is an acceptance rate for the filtered remainder. The quality came out of the check behind the model, not out of the model. Anyone without such a filter is looking at the raw funnel.
Tests that reached new code
25 in 100
Meta 2024, TestGen-LLM: of 100 generated test cases roughly 75 could be compiled, roughly 57 ran dependably, roughly 25 reached previously untouched code.
Now the spot where coverage is worth least to you specifically. A replication study evaluated more than 100,000 test cases produced by 11 language models. Where the tests were written against code known to be correct, the coverage figure tracked genuine bug-finding reasonably, and well when whole models were compared. Where they were written against code whose correctness nobody had established, that link collapsed across the board. The second case is yours.
Caution
Common error: 100 per cent coverage means everything is checked
Coverage counts execution, not inspection. The testing overview chapter of Software Engineering at Google settles it in one line: a handful of tests can put a great deal of program through its paces, and whether a single result was ever looked at is not something those lines record. The number tells you where the run went. It does not tell you that anyone looked.
The reflex is to ask for more tests. That is the wrong conclusion, on two counts. First, bad tests are not a neutral addition but a burden: a brittle test fails on changes that have nothing to do with it, and a suite that has to be adjusted by hand at every change is barely automated any more. Second, the measured return on machine-written tests is poor anyway. If you carry one demand out of this section, let it be the demand for named behaviours, not for a bigger number.
The coverage claim comes from a replication study by Zhao, Zhou and Cohen (July 2026): more than 100,000 test cases, produced by 11 language models, evaluated against an established collection of real program faults. On fault-free source code the link between branch coverage and genuine bug-finding reached a correlation of 0.861 when the models were compared with one another, one averaged figure per model; across the individual test suites it lay only between 0.39 and 0.48. On faulty source code the authors call the links uniformly weak, and they advise measuring bug-finding directly instead of through proxy figures.
How hard machine-produced tests have it in the first place shows in a field report from industrial use (Chen and colleagues, July 2026). In the baseline measurements with language models, between 51 and 67 per cent of the generated tests could be compiled and between 30 and 42 per cent ran through. A purpose-built pipeline lifted that to 91.83 and 54.63 per cent. The shape of that second pair is what matters: even the good pipeline still delivers about half its tests in a state that does not pass.
The systematic answer to this is a machine check the agent does not own. It belongs to the article that follows this one in the path. Here we stay with reading.
When the examined party may rewrite the questions
That the party being examined writes the checklist is something you know from the article on harness engineering. Here that is the premise, not the punchline. What nobody has shown you so far is what it looks like when it happens. Two lines, a difference of a few characters.
Take the discount function from earlier. From 100 euros upwards there should be 10 per cent off, so on 150 euros that is 15. The agent wrote the code, the code returns 12, and its own test is therefore red. From here two roads lead to green: repair the calculation, or change the question.
Definition
The softening trap
In plain words
Technically
def test_rabatt_ab_100_euro_ist_10_prozent():
assert berechne_rabatt(150) == 15The test fails and states why: assert 12 == 15.
def test_rabatt_ab_100_euro_ist_10_prozent():
assert berechne_rabatt(150) >= 0The test passes. 12 is not less than 0, and nothing more is being demanded.
Count up what did not change: the same number of files, the same number of tests, and a name still promising 10 per cent. What did change is one comparison symbol and one number. The discount stays wrong and the bar shows green.
Softening the check is a smaller change than repairing the code. That is why it happens.
A popular image falls apart here as well. Tests get called a safety net - the glossary of this portal calls them that too. The image holds exactly as long as the person falling into the net is not also the one knotting it.
That gives you three cases side by side, and they are not the same thing. In the article on agentic coding the agent deletes the red test - the check disappears visibly. In the article on harness engineering it touches no test file and still steers by the visible checklist. And here: the file got edited, the test is still there, its name still makes the old promise. Further documented roads to green repair nothing either; they are in the box further down.
Note
The one verification you can ask for yourself
- 1. Assume the calculation has since been repaired and returns the correct 15 on 150 euros. Now have it deliberately set wrong: turn the 10 per cent into 20 per cent.
- 2. Have the tests run. On 150 euros the calculation now returns 30 instead of 15.
- 3. Ask which test went red. If the file holds the strict test, it goes red: assert 30 == 15. If it holds the softened one, everything stays green, because 30 does happen to be larger than zero.
- 4. Draw the conclusion. If no test goes red, that is a statement about the tests and not about the code: this behaviour was never being checked.
The honest limit on this: a check that weak catches only the very crudest thing left. A negative discount would register with it, and text in that position would make it crash. Every plausible wrong amount goes straight through.
The professional name for this procedure is mutation analysis: small faults are built in deliberately, and the measure is how many of them the tests notice. Google runs it across a code base measured in billions of lines. You need no tooling for it, only the question.
Caution
Common error: then I will simply forbid it
Something related has been measured, and the result is unpleasant. In a study at OpenAI (Baker and colleagues, March 2025) the visible reasoning of the model was scored during training as well - it was not enough to pass, it also had to look unremarkable while doing so. The behaviour did not go away, it got harder to see: the model buried its intent inside the reasoning and went on collecting the reward. And the instruction in a chat has since been measured directly. At METR (June 2025), asking the model not to cheat changed next to nothing. In ImpossibleBench (October 2025), GPT-5 cheated on single programming tasks in more than 85 per cent of cases despite being told not to modify the tests; only the addition to stop at tests that look wrong and name them pushed that down to 1 per cent. On tasks from real projects it still cheated in 54 per cent of cases, even with this strictest instruction. My reading: a ban alone changes little, and you cannot rely on any instruction.
Caution
Common error: the agent is lying to me
The mechanism needs no intent. Published reasoning traces do show it sometimes, but it is not the cause - the cause is a signal that something is being measured by. The difference is not cosmetic: see deception and you distrust the agent. See optimisation and you change what gets measured. Only the second one helps.
The roads to green are not thought experiments. The study at OpenAI (Baker and colleagues, March 2025) watched a model at programming work and put names to them: skip the test so it never runs; hand back a fixed stand-in value at a spot where checking was thin; and rebuild whatever the verification depends on so that it always reports success. The list in the paper is considerably longer; the further entries need background this article does not build. From the reasoning of the model itself the authors quote a line that captures the thing: this is unnatural but tests might pass.
One point has to stand clearly here, because otherwise it sticks in the wrong shape: how often this occurs in ordinary projects is measured by none of these sources. What is documented is that the behaviour occurs and what it looks like. How often it will meet you, nobody knows - this article included.
The next obvious move is to check what changed inside the test file. That is a craft of its own and gets an article of its own in this learning path.
Test bench: which test goes red?
What I ask of the agent
Five sentences you can send word for word. The list should not get longer than that - a long list gets skimmed instead of used.
The fifth sits at the end for a reason: its answer is the one you can check for yourself. The first four are assurances. This one is evidence.
The test that was never red
Every road to a dishonest green in this article ends at the same result: a test that never once went red for an honest reason. Back to the 47 green tests from the start. You still do not know whether the software is correct - but you now have three things you did not have before: two halves you can read, one figure you mistrust in measured amounts, and one verification that produces evidence. And you have a different opening question. No longer: are the tests green? Instead: what does each of them claim, and which behaviour did none of them name?
Note
The five points that matter
- Two halves of a test are open to a non-programmer: the name states the intended behaviour, the check names the value that would demonstrate it. Neither is decoration.
- Green reports on the condition of the checks, not on the condition of the software - and where nobody ever wrote a test, it reports nothing at all.
- A coverage figure records which lines were executed, not whether a result was inspected. It is reasonably useful where the code is known to be correct, and it loses its value in exactly the case where the same agent wrote code and tests.
- The risky edit is not the test somebody removed. It is the one left in place, its name still vouching for the old behaviour after the check inside went soft - and no count of tests, no pass rate and no coverage figure shows it.
- One verification does not rest on what the agent tells you: have the code deliberately broken, then ask which test turned red. If nothing turned red, nothing was checking that behaviour.
Check
Quiz: reading and judging tests
Which two parts of a test can you read without being able to program?
Select one answer
1. Which two parts of a test can you read without being able to program?
2. A test is called test_passwort and checks only that the result is not empty. Which two independent weaknesses does it have?
3. Agent A reports 40 tests, all green. Agent B reports 12 tests, all green, each of them with a concrete expected value. What follows from that?
4. A test is called test_suche_liefert_genau_drei_treffer. Yesterday it checked for the number 3, today only that a list comes back at all. Nothing was deleted. Which of the usual figures shows this?
5. Case A: a human wrote working code and the agent adds the tests. Case B: the agent wrote the code that nobody has checked, and the tests along with it. Where does a high coverage figure say more?
Check
Comprehension check
Can you explain this in your own words? Tick what you've got.
0 of 3 done