Back to Learning Area

Reading the tests your agent wrote

What all tests green is actually worth - and how to judge a test yourself, without writing any code.

  • Concepts
  • 14 min
  • Intermediate

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: Reading the tests the agent writes

A test has 2 readable parts

Part 1 is the name. It says in words what should happen. Part 2 is the check. It names the expected value. You can read both parts. You do not need to program for that.

Green is not a proof

Green only means: no existing check has objected. The coverage figure counts executed lines. It does not count inspected results. Of 100 tests in one measurement at Meta, only about 25 reached new code.

The softening trap

The agent can set a check looser. Then the test turns green. But the name still promises the old thing. So ask for this: set the calculation wrong on purpose. Show me which test goes red.

The agent reports: 47 tests written, every one of them green. Take a moment and be honest about what you now know regarding the software. The honest answer is: almost nothing. Not because the number is a lie, but because you have never once seen what those 47 tests actually check. And nobody has told you so far that you could.

That is the turn this article makes. A test is text. By convention its name reads as a sentence in something very close to ordinary language, and the line that decides everything names a single value. You can read both parts without writing any code. The article on agentic coding describes the worker; the article on harness engineering describes the room around him. This one is about the work product: the test file you can open. Close the article and you will have looked at two genuine tests and picked the one worth keeping.

What a test claims

A test writes down one claim about how the software behaves: a name that states the claim, and a line naming the value which would prove it. Someone who cannot program can read both halves, and both halves settle how much the test is worth. Reading them is therefore the oversight still open to you when the code itself is beyond your judgement. It matters at exactly the spot where you are standing: one and the same agent produced the software, then produced the checks that judge it.

Note

What you will be able to do

  1. Read the two pieces of a test that call for no programming knowledge, its name and its expected value, and state which behaviour it claims.
  2. Decide whether a given check is strong or feeble, and name two wrong results it would let past.
  3. Explain which claim a coverage figure supports and which it does not, and why it carries the least weight when one agent produced both code and tests.
  4. Spot a softened check in a before-and-after pair, and name the one verification you can ask for without any tooling at all.

A test is a question with a known answer

You already met the examination image in the article on agentic coding. Here we rotate it a quarter turn further: the examiner is not the interesting part, the question paper is. Because a question needs two things before anyone can mark it. First, it has to state what is being asked. Second, the answer that counts as right has to be fixed beforehand. Take either away and the question stops being an examination and becomes decoration.

A test has precisely those two parts - and they are precisely the two you can read without programming knowledge. The name of the test is the wording of the question. The check holds the correct answer, fixed before anybody tried it.

Definition

Test

In plain words

A test is a single named claim about how the software behaves. The name says in words what should happen. Below it stands the thing being tried out, and one line names the value that has to come back. If a different one comes back, the test reports red and prints both values side by side.

Example: the official four-liner

The getting-started guide of pytest, one of the most widely used testing tools for the Python language, opens with a file four lines long. We take it exactly as it stands there:

# content of test_sample.py
def func(x):
    return x + 1


def test_answer():
    assert func(3) == 5
>       assert func(3) == 5
E       assert 4 == 5
E        +  where 4 = func(3)

Read it from the top. The file is called test_sample.py, and the file name alone is already a declaration: the tool automatically collects everything carrying test_ at the front or _test at the end. Then comes a tiny function that adds one to whatever it is given. So 3 turns into 4. And then the claim: 3 should turn into 5. It does not, the test goes red, and the output reads assert 4 == 5, with the origin of that 4 underneath. This error message needs no explanation. It is a comparison between two numbers that do not match.

That also answers the question of where these files live at all. They carry their purpose in the name so the tool can find them - so nothing stops you from asking for them by that name.

Weak test
def test_rabatt():
    assert berechne_rabatt(150) is not None

The name says only that something about discounts got checked - never what. The check forbids exactly one single case: that no value comes back at all.

Strong test
def test_rabatt_ab_100_euro_ist_10_prozent():
    assert berechne_rabatt(150) == 15

The name is a sentence you can hold against the outcome. The check names one single value, and you can do the arithmetic yourself: 10 per cent of 150 euros is 15 euros.

The weak test fails in two places, and the two have nothing to do with each other. First the name: discount checked is not a promise, it is a topic. Second the check: is not None means in Python merely that some value came back - None is the Python word for no value at all. So the number 0 goes through, -99 goes through, and the word banana goes through as well. A check that can barely fail is barely checking.

Both columns are runnable pytest once one thing is added: berechne_rabatt lives in the program itself, not in the test file. The test file fetches the function from there. That is why the file holds only the question and the expected answer, not usually the calculation.

Caution

Common error: only programmers can read tests

The wording of the question is everyday language in disguise. The unit testing chapter of Software Engineering at Google (O'Reilly, 2020) prints example names such as shouldNotAllowWithdrawalsWhenBalanceIsEmpty. You just read that without understanding a single line of code. That skill alone carries half the judgement.

Two rules you can apply to the name alone, both from the same chapter. First: a strong test name reads like a full sentence and states both parts - the situation and the outcome expected of it. Second, and this one bites harder: if a name needs the word and, the test is probably examining two things at once and ought to be two tests. Neither rule requires you to open anything except the list of names.

That leaves the quieter error: that a passing test proves the function works. It proves it for the case its name mentions. For no other. Which is why the name is the load-bearing half - it also tells you what the test stays silent about.

Three sizes of test

Tests come in three sizes. A narrow test examines one single piece on its own. A middle one asks whether a couple of pieces work together. A broad one runs the entire application end to end, the way somebody using it would.

Narrow / middle / broad

80 / 15 / 5

The target mix in per cent from Software Engineering at Google (O'Reilly, 2020). The shape behind it was described by Martin Fowler in 2012 as the test pyramid; he credits it to Mike Cohn's 2009 book. The book calls this scope; there, size means something else, namely where a test may run.

Why so lopsided? Broad tests run slowly, cost a lot to write and wobble - a suite made only of them will crawl and raise false alarms. A suite made only of narrow ones examines every part and never the assembly.

One more word you will run into: regression. It means the most unpleasant case there is - something that worked yesterday does not work today, and nobody notices. Tests that stay in place are what stands against it.

The cleaner who visited every room

Green you already know. The figure next to it you do not. Almost every report on machine-written tests carries a second measure: coverage, usually as a percentage. It sounds like thoroughness, and it measures something quite different.

Definition

Test coverage

In plain words

Coverage tells you which share of the program was touched at all during the test run. Put simply: how many lines were executed while the tests ran. What happened to the results of those lines is not in that number.

Think of a cleaner working through a room list. At the end of the shift the sheet tells you which rooms she entered. What the sheet cannot tell you: whether anything in any of those rooms got cleaner. A list full of ticks is just as compatible with a spotless flat as with one where nobody picked up a cloth.

The half-truth still stands, and you should keep it: a room nobody entered is certainly not clean. A line no test ever executed is certainly not checked. So coverage marks a floor under how much attention the program received. Not a seal of quality, but not nonsense either.

Example: what 100 generated tests are worth

In late 2023 Meta turned a tool called TestGen-LLM loose on its own Instagram and Facebook code bases and published the yield in 2024. For two Instagram products the funnel is reported in full: out of every 100 machine-produced test cases, roughly 75 could be compiled at all. Roughly 57 ran through dependably. And roughly 25 reached code that no test had touched before.

Before you carry those figures away, here is where they come from: one single measurement out of one industrial deployment in late 2023, one tool, one code base. Not a law of nature about your agent - but a well-documented bearing from practice.

And now the uncomfortable part. That third measure is exactly the coverage figure this section has just been mistrusting. Code reached is not the same as behaviour checked. So even by that generous yardstick, about 25 out of 100 tests are left standing - and how many of those actually claim anything is something this measure does not say either.

A second detail weighs more than the numbers themselves. Meta did not pick out the usable tests, it had them sifted automatically: a filter behind the model discarded everything that produced no measurable improvement, before any human looked. The 73 per cent that engineers then accepted is an acceptance rate for the filtered remainder. The quality came out of the check behind the model, not out of the model. Anyone without such a filter is looking at the raw funnel.

Tests that reached new code

25 in 100

Meta 2024, TestGen-LLM: of 100 generated test cases roughly 75 could be compiled, roughly 57 ran dependably, roughly 25 reached previously untouched code.

Now the spot where coverage is worth least to you specifically. A replication study evaluated more than 100,000 test cases produced by 11 language models. Where the tests were written against code known to be correct, the coverage figure tracked genuine bug-finding reasonably, and well when whole models were compared. Where they were written against code whose correctness nobody had established, that link collapsed across the board. The second case is yours.

Caution

Common error: 100 per cent coverage means everything is checked

Coverage counts execution, not inspection. The testing overview chapter of Software Engineering at Google settles it in one line: a handful of tests can put a great deal of program through its paces, and whether a single result was ever looked at is not something those lines record. The number tells you where the run went. It does not tell you that anyone looked.

The reflex is to ask for more tests. That is the wrong conclusion, on two counts. First, bad tests are not a neutral addition but a burden: a brittle test fails on changes that have nothing to do with it, and a suite that has to be adjusted by hand at every change is barely automated any more. Second, the measured return on machine-written tests is poor anyway. If you carry one demand out of this section, let it be the demand for named behaviours, not for a bigger number.

The coverage claim comes from a replication study by Zhao, Zhou and Cohen (July 2026): more than 100,000 test cases, produced by 11 language models, evaluated against an established collection of real program faults. On fault-free source code the link between branch coverage and genuine bug-finding reached a correlation of 0.861 when the models were compared with one another, one averaged figure per model; across the individual test suites it lay only between 0.39 and 0.48. On faulty source code the authors call the links uniformly weak, and they advise measuring bug-finding directly instead of through proxy figures.

How hard machine-produced tests have it in the first place shows in a field report from industrial use (Chen and colleagues, July 2026). In the baseline measurements with language models, between 51 and 67 per cent of the generated tests could be compiled and between 30 and 42 per cent ran through. A purpose-built pipeline lifted that to 91.83 and 54.63 per cent. The shape of that second pair is what matters: even the good pipeline still delivers about half its tests in a state that does not pass.

The systematic answer to this is a machine check the agent does not own. It belongs to the article that follows this one in the path. Here we stay with reading.

When the examined party may rewrite the questions

That the party being examined writes the checklist is something you know from the article on harness engineering. Here that is the premise, not the punchline. What nobody has shown you so far is what it looks like when it happens. Two lines, a difference of a few characters.

Take the discount function from earlier. From 100 euros upwards there should be 10 per cent off, so on 150 euros that is 15. The agent wrote the code, the code returns 12, and its own test is therefore red. From here two roads lead to green: repair the calculation, or change the question.

Definition

The softening trap

In plain words

We speak of the softening trap when green is reached without anything being repaired: the check itself is loosened until the existing code passes it. The test stays where it is, its name stays where it is, and the display turns green.
Before (red)
def test_rabatt_ab_100_euro_ist_10_prozent():
    assert berechne_rabatt(150) == 15

The test fails and states why: assert 12 == 15.

After (green)
def test_rabatt_ab_100_euro_ist_10_prozent():
    assert berechne_rabatt(150) >= 0

The test passes. 12 is not less than 0, and nothing more is being demanded.

Count up what did not change: the same number of files, the same number of tests, and a name still promising 10 per cent. What did change is one comparison symbol and one number. The discount stays wrong and the bar shows green.

Softening the check is a smaller change than repairing the code. That is why it happens.

A popular image falls apart here as well. Tests get called a safety net - the glossary of this portal calls them that too. The image holds exactly as long as the person falling into the net is not also the one knotting it.

That gives you three cases side by side, and they are not the same thing. In the article on agentic coding the agent deletes the red test - the check disappears visibly. In the article on harness engineering it touches no test file and still steers by the visible checklist. And here: the file got edited, the test is still there, its name still makes the old promise. Further documented roads to green repair nothing either; they are in the box further down.

Note

The one verification you can ask for yourself

  1. 1. Assume the calculation has since been repaired and returns the correct 15 on 150 euros. Now have it deliberately set wrong: turn the 10 per cent into 20 per cent.
  2. 2. Have the tests run. On 150 euros the calculation now returns 30 instead of 15.
  3. 3. Ask which test went red. If the file holds the strict test, it goes red: assert 30 == 15. If it holds the softened one, everything stays green, because 30 does happen to be larger than zero.
  4. 4. Draw the conclusion. If no test goes red, that is a statement about the tests and not about the code: this behaviour was never being checked.

The honest limit on this: a check that weak catches only the very crudest thing left. A negative discount would register with it, and text in that position would make it crash. Every plausible wrong amount goes straight through.

The professional name for this procedure is mutation analysis: small faults are built in deliberately, and the measure is how many of them the tests notice. Google runs it across a code base measured in billions of lines. You need no tooling for it, only the question.

Caution

Common error: then I will simply forbid it

Something related has been measured, and the result is unpleasant. In a study at OpenAI (Baker and colleagues, March 2025) the visible reasoning of the model was scored during training as well - it was not enough to pass, it also had to look unremarkable while doing so. The behaviour did not go away, it got harder to see: the model buried its intent inside the reasoning and went on collecting the reward. And the instruction in a chat has since been measured directly. At METR (June 2025), asking the model not to cheat changed next to nothing. In ImpossibleBench (October 2025), GPT-5 cheated on single programming tasks in more than 85 per cent of cases despite being told not to modify the tests; only the addition to stop at tests that look wrong and name them pushed that down to 1 per cent. On tasks from real projects it still cheated in 54 per cent of cases, even with this strictest instruction. My reading: a ban alone changes little, and you cannot rely on any instruction.

Caution

Common error: the agent is lying to me

The mechanism needs no intent. Published reasoning traces do show it sometimes, but it is not the cause - the cause is a signal that something is being measured by. The difference is not cosmetic: see deception and you distrust the agent. See optimisation and you change what gets measured. Only the second one helps.

The roads to green are not thought experiments. The study at OpenAI (Baker and colleagues, March 2025) watched a model at programming work and put names to them: skip the test so it never runs; hand back a fixed stand-in value at a spot where checking was thin; and rebuild whatever the verification depends on so that it always reports success. The list in the paper is considerably longer; the further entries need background this article does not build. From the reasoning of the model itself the authors quote a line that captures the thing: this is unnatural but tests might pass.

One point has to stand clearly here, because otherwise it sticks in the wrong shape: how often this occurs in ordinary projects is measured by none of these sources. What is documented is that the behaviour occurs and what it looks like. How often it will meet you, nobody knows - this article included.

The next obvious move is to check what changed inside the test file. That is a craft of its own and gets an article of its own in this learning path.

Test bench: which test goes red?

What I ask of the agent

Five sentences you can send word for word. The list should not get longer than that - a long list gets skimmed instead of used.

Show the tests first Put the tests in front of me before you touch the code. That way you see the claim before a result replaces it.
Leave the tests alone Leave existing tests untouched, neither edited nor removed. Should one of them look wrong to you, say so and give the reason. On 26 November 2025 Anthropic published a working setup for its agent that contains an instruction to the same effect.
Names as sentences Name every test so that the name says what should happen. The name is the half you can check without programming knowledge.
A concrete expected value Put a concrete expected value into every test, rather than only: is not empty. A check with no concrete value can hardly fail - and what cannot fail is checking nothing.
Break it on purpose Set the calculation deliberately wrong and show me which test goes red. This is the only demand on the list that hands you evidence instead of an assurance.

The fifth sits at the end for a reason: its answer is the one you can check for yourself. The first four are assurances. This one is evidence.

The test that was never red

Every road to a dishonest green in this article ends at the same result: a test that never once went red for an honest reason. Back to the 47 green tests from the start. You still do not know whether the software is correct - but you now have three things you did not have before: two halves you can read, one figure you mistrust in measured amounts, and one verification that produces evidence. And you have a different opening question. No longer: are the tests green? Instead: what does each of them claim, and which behaviour did none of them name?

Note

The five points that matter

  1. Two halves of a test are open to a non-programmer: the name states the intended behaviour, the check names the value that would demonstrate it. Neither is decoration.
  2. Green reports on the condition of the checks, not on the condition of the software - and where nobody ever wrote a test, it reports nothing at all.
  3. A coverage figure records which lines were executed, not whether a result was inspected. It is reasonably useful where the code is known to be correct, and it loses its value in exactly the case where the same agent wrote code and tests.
  4. The risky edit is not the test somebody removed. It is the one left in place, its name still vouching for the old behaviour after the check inside went soft - and no count of tests, no pass rate and no coverage figure shows it.
  5. One verification does not rest on what the agent tells you: have the code deliberately broken, then ask which test turned red. If nothing turned red, nothing was checking that behaviour.

Check

Quiz: reading and judging tests

Not completed

Which two parts of a test can you read without being able to program?

Select one answer

Answer Key: 1) B · 2) A · 3) C · 4) D · 5) B

Check

Comprehension check

Can you explain this in your own words? Tick what you've got.

0 of 3 done