# How we test plottypus: results, method and cost

Canonical: https://plottypus.com/evals

Last updated: 2026-10-14

> Every plottypus eval, dated, with its set and size: eight code-review tools under one judge, comments right, cost, time, method and what we leave out.

Every result we have, as of 14 Oct 2026. Each number names its set, its size and its date.

### Martian Code Review Bench, same pull requests, same judge

| Tool | Precision | Recall | F1 | n | F1, validation |
| --- | --- | --- | --- | --- | --- |
| Qodo (extended v2) | 64.0% | 58.2% | 61.0 | 20 | 59.0 |
| Augment | 56.9% | 52.7% | 54.7 | 20 | 57.9 |
| **plottypus** | 60.5% | 47.3% | 53.1 | 20 | 55.8 |
| Cursor Bugbot | 55.8% | 43.6% | 49.0 | 20 | 44.0 |
| Devin | 66.7% | 36.4% | 47.1 | 20 | 52.9 |
| CodeRabbit | 38.0% | 54.5% | 44.8 | 20 | 41.6 |
| GitHub Copilot (v2) | 32.6% | 56.4% | 41.3 | 20 | 48.1 |
| Claude Code | 37.9% | 40.0% | 38.9 | 20 | 44.8 |

14 Oct 2026. [Martian Code Review Bench](https://github.com/withmartian/code-review-benchmark). Judge: Sonnet 5.5 with the benchmark's own judge prompt, one judge for every tool. Re-scored from each tool's published comments on the same pull requests. Held-back: 20 Go and Java pull requests never used in tuning; plottypus ran once. F1, validation: 18 pull requests no rule was written from; plottypus is the mean of 4 passes (F1 53.5 to 57.7). At this size, a gap under about 5 points of F1 is within noise.

- With every tool scored by the same judge, plottypus ranked third of eight on F1 on both sets, ahead of CodeRabbit, GitHub Copilot and Claude Code by 7 to 14 points both times.
- On 20 held-back pull requests, 17 plottypus comments missed a known bug, against 49 for CodeRabbit and 64 for GitHub Copilot, scored by the same judge.

In four numbers (14 Oct 2026, reviewer Sonnet 5.5, dollars are tokens at list price):

- 60% of comments right. Held-back, run once: 20 pull requests in Go and Java, n = 44 comments, 95% interval 46 to 74%. 60.5% on the same languages we tuned on.
- $0.24 median per pull request, in model cost. $0.64 worst per pull request. Held-back, run once, n = 20 pull requests in Go and Java.
- 39.5 s from the mistake to the message to Claude, median. Our own synthetic tests, n = 18 planted mistakes, 16 caught.
- $1.25 an hour of live review, in model cost, on average. One real session replayed twice, at $1.22 and $1.28 an hour.

## pull request results, two sets.

|  | Validation (same languages we tuned on) | Held-back (Go and Java, run once) |
| --- | --- | --- |
| Serious bugs found | 9 of 11 a pass (82%), range 8 to 10 across passes | not published |
| All known bugs found | 29 of 56 a pass, range 27 to 30 across passes | not published |
| F1 | 55.8, range 53.5 to 57.7 across passes | not published |
| Comments right | 60.5% of comments right | 60% of comments right, 95% interval 46 to 74% |
| Pull requests | 18, TypeScript, Ruby and Python | 20, Go and Java |
| Passes | 4 | 1 |
| Known bugs, of them serious | 56, 11 | 55, 23 |
| Comments sent, of them wrong | not shown | 44, 17 |
| Cost per pull request, median | $0.19 per PR | $0.24 per PR |
| Cost per pull request, worst | not shown | $0.64 per PR |
| Time per pull request, median | about 60 s | 170 s |
| Time per pull request, worst | not shown | 506 s |
| File lookups per pull request, median | not shown | 11 |

Validation repos: [cal.com](https://github.com/calcom/cal.diy) (TypeScript, 7), [discourse](https://github.com/discourse/discourse) (Ruby, 7), [sentry](https://github.com/getsentry/sentry) (Python, 4). Held-back repos: [grafana](https://github.com/grafana/grafana) (Go, 10), [keycloak](https://github.com/keycloak/keycloak) (Java, 10). All from the [Martian code-review benchmark](https://github.com/withmartian/code-review-benchmark).

- One run, so each figure carries its interval.
- This was the first scored held-back run. Three early experiments on 8 Oct called the reviewer on some of these pull requests by mistake; nothing from them was scored or looked at.
- In the run, the answer files were missing at first. They were rebuilt from the public known bugs and only the scoring was rerun; no review was rerun.

## live sessions: our own synthetic tests.

Our own 15 scripted Claude Code sessions with 18 planted mistakes and 2 clean controls, run once on the shipped code. (14 Oct 2026, our own synthetic tests, 15 sessions, n = 18 mistakes + 2 clean, 1 pass)

- 16 of 18 planted mistakes caught
- 13 of 16 of those reached Claude before it finished
- 0 false flags; both clean sessions stayed silent
- 39.5 s median, mistake to message (167.6 s at the 90th percentile)

- We wrote these sessions and their mistakes ourselves, and we tuned delivery on them. They show the plumbing works, not how it does on your code.
- The other half of our scripted sessions is held back and has not been run yet.

## what we're improving.

- **Serious bugs in languages we never tuned on**: On 20 Go and Java pull requests, run once, it found fewer of the serious known bugs than on the languages it was tuned on. Some it noticed and then cleared as fine in the step where it checks its own suspicion. That step is what we are working on, and any fix will be tested on fresh, untouched Go and Java pull requests before we claim it.
- **Live use**: Not yet measured at scale. Every number here is from a replay or a scripted run, not from a team using it day to day.

- Serious bugs in languages we did not tune on: it finds fewer of them on Go and Java than on the languages it was tuned on.
- It talks itself out of real problems. On the held-back set it noticed serious bugs and then cleared them as fine in the step that checks its own suspicion.
- Subtle concurrency, like a counter that can lose updates when two jobs run at once: missed in every pass on one public pull request.
- In our own scripted sessions it missed 2 of 18 planted mistakes, and 3 of the 16 it caught reached Claude after it had finished.
- Every number here is from a replay or a scripted run. None is from a team using it live yet.

## three sets, and why three.

- **Design set** (not published): The pull requests we designed the reviewer on, in TypeScript, Ruby and Python. We tuned on them, so they say how well we fitted, not how well it works. (tuned on)
- **Validation set** (same languages we tuned on): 18 public pull requests from the same benchmark, in the same languages we tuned on. No tuning rule was written from them, but we used them to choose the model and to reject one prompt, so these numbers lean optimistic. The rule for passing was committed before the run. (18 PRs · 4 passes · rule committed first (ccb5626))
- **Held-back set** (run once): 20 public pull requests in Go and Java from the Martian code-review benchmark, run once, never tuned on. We read only its totals. Once run, it is spent: a fix cannot be checked on it. (20 PRs · 1 pass · frozen commit 20fa6d8)

A set you tune on flatters you. A fresh set in the same languages shows whether the tuning was general or fitted. A held-back set in other languages, run once on frozen code, is the closest we can get to code we have never seen.

## two real examples, one found, one missed.

Both are public pull requests from the validation set, rated serious by the answer key. Held-back items are never opened, so none is shown.

### cal.com, pull request 8087 (Critical)

Link: https://github.com/calcom/cal.diy/pull/8087

Result: found in 4 of 4 passes.

Known bug, in the answer key's words: "The code uses forEach with async callbacks, which causes asynchronous operations (e.g., calendar/video event deletions, payment refunds) to run concurrently without being awaited. This 'fire-and-forget' behavior leads to unhandled promise rejections, race conditions, and incomplete cleanup…"

Our comment, at `packages/trpc/server/routers/viewer/bookings.tsx:553`: "forEach(async) is used for the calendar deletions on reschedule. The deletions are not awaited, and the surrounding try/catch cannot catch errors from the async callback. … A failure in getCalendar or deleteEvent becomes an unhandled promise rejection, which can crash a Node process … Use Promise.all(refs.map(async ...)) with an await."

```
bookingRefsFiltered.forEach(async (bookingRef) => {
  const calendar = await getCalendar(credentialsMap.get(bookingRef.type));
```

### cal.com, pull request 14943 (High)

Link: https://github.com/calcom/cal.diy/pull/14943

Result: missed in 4 of 4 passes.

Known bug, in the answer key's words: "Using retryCount: reminder.retryCount + 1 reads a possibly stale value and can lose increments under concurrency; consider an atomic increment via Prisma (increment: 1) to avoid race conditions (also applies to the similar update in the catch block)."

No comment matched it in any of the 4 passes. The answer key names no file or line, so we show none.

```
retryCount: reminder.retryCount + 1
```

## the method, and its limits.

- **answer key**: The benchmark's known bugs for each pull request, each with a severity.
- **serious**: A known bug the benchmark rates Critical or High.
- **judge**: A language model decides whether a comment matches a known bug, using the benchmark's own judge prompt word for word. It is a Claude model, the same family as the reviewer, and it sends one comment with all of that pull request's known bugs in one call, where the benchmark makes one call per pair.
- **found**: A known bug counts as found if at least one comment matches it. A comment that matches no known bug counts as wrong.
- **precision, recall, F1**: Precision is the share of comments that are right. Recall is the share of known bugs found. F1 is the balance of the two (their harmonic mean), from 0 to 100.
- **hand check**: On 8 Oct we checked 20 judge decisions by hand (10 matches, 10 non-matches, drawn with a fixed seed) and agreed with all 20. Outside the sample we found one we would reverse (too strict). The reviewer has changed since that run.
- **right % is likely a floor**: On the design and validation sets we read the "wrong" pile against the code (one reader): about half were real defects the answer key does not list, and none were false. So the share of right comments is likely a floor. The held-back pile was not read.
- **cost**: Dollars are tokens counted at list price.

## cost and time.

- Held-back, per pull request: $0.24 median per PR, $0.64 worst per PR
- Held-back, per known bug found: $0.24 per bug found
- Held-back, time per pull request: 170 s median, 506 s worst
- Validation, per pull request: $0.19 per PR, about 60 s
- Live review: About $1.25 per hour of Claude working: one real session replayed twice, at $1.22 and $1.28 an hour.

Reviewer: Sonnet 5.5 (claude-sonnet-5-5), high effort on pull requests and medium effort in live sessions. Dollars are tokens counted at list price.

We do not show a cost for the synthetic sessions yet: two of our own records disagree, and we will publish it once they match.

## what we do not publish, and why.

- **Numbers from the 6 pull requests we designed the reviewer on**: We tuned on them, so they say how well we fitted, not how well it works.
- **Results from our own recorded work sessions**: They are private, there is only one we can replay, and the reviewer was tuned on it. Only its cost per hour is shown.
- **Earlier runs (8 to 12 October)**: They measured an older reviewer, and their dollar figures came from a tool estimate, not from tokens at list price.
- **The held-back pull requests one by one**: We only read the totals, so the set stays untouched by us.