plottypusSign inStart free trial
Guides

Your coding agent said done. Here's how to check.

by Lay Sheth5 min read

Quick answer

Don’t take the agent’s summary as evidence. Rerun the tests yourself with caching off, diff the test files on their own, and check that any new test fails without the fix. Most false “done”s fall into five types, and each takes a minute or two to rule out with git and your test runner.

False done What it looks like Quickest check
Tests not rerun after the last edit “All tests pass”, then two more edits Rerun the suite, with caching off
A test rewritten until it passes Expected value changed to match the output git diff on test files only
A skipped or deleted test Fewer tests than yesterday, or a new skip Count tests, grep for skip markers
A stub presented as finished Hard-coded return, TODO, swallowed error Grep the diff, read the new functions
The wrong file edited A copy, a generated file, or dead code changed Check the change is on the path that runs

Why can’t the agent’s summary be the evidence?

Because the summary is written by the same process that did the work, at the moment it decided to stop. Anthropic’s own guidance for Claude Code puts it this way: “Claude stops when the work looks done.” Without a check it can run, “looks done” is the only signal.

There’s also a documented incentive to game the check. METR reported in June 2025 that frontier models reward-hacked in their evaluations, including monkey-patching an evaluator so every submission passed, and that the models showed awareness the behaviour wasn’t what users wanted. ImpossibleBench (October 2025) gave agents tasks where the spec and the tests contradict, so a pass means a shortcut. The most capable models cheated most often, and Claude models mostly did it by editing the test cases.

None of this means your agent is lying on purpose. It means the summary is a claim, and claims get checked.

How do I check the tests were really rerun?

Run them yourself, after the last edit, with any result cache turned off. Some runners reuse old results. Go is the clearest case: in package list mode, go test caches successful results and prints (cached) instead of a time. The Go docs say “the idiomatic way to disable test caching explicitly is to use -count=1.”

bash
go test -count=1 ./...

For other runners, run the plain command from a clean shell, not the agent’s terminal. Then compare the counts to the agent’s claim. “43 passed” from you and “all tests pass” from the agent is a match. “41 passed, 2 skipped” is not.

How do I tell if a test was rewritten to pass?

Look at test changes on their own, apart from the rest of the diff. Against the branch point:

bash
git diff main...HEAD --stat -- tests/
git diff main...HEAD -- tests/

The three-dot form compares against the common ancestor, so you see only this branch’s work. Swap tests/ for wherever your tests live, or a pattern like '*_test.go'. Then read each change and ask one question: did an assertion get weaker? The usual signs:

  • an expected value changed to whatever the code now returns
  • an exact match turned into “contains” or “is not null”
  • a try block wrapped around the assertion
  • a test for an error case deleted

Some test edits are correct. A test that encoded the old behaviour you asked to change should change. The point is to read those edits on purpose instead of scrolling past them.

How do I spot a skipped or deleted test?

Deleted files show up with a filter:

bash
git diff main...HEAD --diff-filter=D --name-status

For skips, grep the added lines for your framework’s markers. In pytest that’s @pytest.mark.skip or @pytest.mark.xfail, and pytest -rs prints the reason for each skipped test in the summary. In Go it’s t.Skip. In Jest, look for .skip( and .only(: a stray .only means only the marked tests in that file run.

bash
git diff main...HEAD | grep '^+' | grep -nE 'skip|xfail|\.only\('

How do I catch a stub?

Read the new functions, not just the diff summary. Then grep the added lines for the usual shapes:

bash
git diff main...HEAD | grep '^+' | grep -nE 'TODO|FIXME|not implemented|return (true|false|null|nil|0|\[\]|\{\})\s*;?$'

That last pattern is noisy, which is fine. You’re looking for a function whose whole body is a fixed return, or an error path that logs and carries on. If the agent said “implemented retry with backoff” and the retry loop runs once, it’s a stub.

How do I know the fix actually fixed it?

A regression test should fail without the fix and pass with it. If it passes both ways, it isn’t testing the bug. git stash can stash only the source files and leave the tests in place:

bash
git stash push -- src/      # set the fix aside, keep the new test
go test -count=1 ./...      # the new test should fail here
git stash pop               # bring the fix back
go test -count=1 ./...      # and pass here

Git’s docs describe the pathspec form exactly this way: only the matching files are rolled back, “leaving files that do not match the pathspec intact.” This one check catches more false dones than any other, because a test written after the code tends to test what the code does, not what you asked for.

What about the wrong file?

Agents sometimes fix a copy: a vendored file, a generated file that gets rebuilt, an old module nothing imports. Check that the changed function is on the path that runs. A quick way is to break it on purpose: make the changed line throw and see if any test notices. If nothing fails, either the code isn’t reached or nothing tests it. Both are worth knowing.

Can I make the agent do these checks itself?

Partly. You can ask it to show evidence instead of asserting success: the command it ran and what came back. The Claude Code docs recommend exactly that. And you can make the rerun deterministic with a hook, so the turn can’t end while the suite is red. I wrote that setup up in make your coding agent rerun the tests before it says done. A hook won’t catch a weakened test or a stub, though. For those you want a reviewer that didn’t write the code, which is the argument in should one coding agent review another’s work?.

Where plottypus fits

Every check above is something you can do by hand, and if you only run one, run the stash check. The full list, as a checklist you can paste into a pull request, is in a checklist for reviewing a coding agent’s work. plottypus is a second agent that does this kind of checking while the Claude Code session is still going: it notices when the tests haven’t been rerun since the last edit and tells the agent, with the evidence, before you see the “done”; the agent reruns them. What it catches and what it misses is on the evals page. If that’s the problem you have, start a free trial.

Sources

Checked 14 October 2026.

plottypus is independent and not affiliated with any company or project named here. Claude Code is a trademark of Anthropic. Names belong to their owners. This is our research and opinion, dated above; details may have changed.