plottypusSign inStart free trial

Evals · as of 14 Oct 2026

how we test plottypus.

Every result we have, as of 14 Oct 2026. Each number names its set, its size and its date.

Reviewer Sonnet 5.5Benchmark Martian code-review benchmarkStart free trial

Martian Code Review Bench: same pull requests, same judge

Tools on the Martian Code Review Bench, scored by one judge
ToolPrecisionRecallF1nF1, validation
Qodo (extended v2)64.0%58.2%61.02059.0
Augment56.9%52.7%54.72057.9
plottypus60.5%47.3%53.12055.8
Cursor Bugbot55.8%43.6%49.02044.0
Devin66.7%36.4%47.12052.9
CodeRabbit38.0%54.5%44.82041.6
GitHub Copilot (v2)32.6%56.4%41.32048.1
Claude Code37.9%40.0%38.92044.8

14 Oct 2026 · Martian Code Review Bench · judge Sonnet 5.5 with the benchmark's own judge prompt, one judge for every tool

Re-scored from each tool's published comments on the same pull requests. Held-back: 20 Go and Java pull requests never used in tuning; plottypus ran once. F1, validation: 18 pull requests no rule was written from; plottypus is the mean of 4 passes (F1 53.5 to 57.7). At this size, a gap under about 5 points of F1 is within noise.

  • With every tool scored by the same judge, plottypus ranked third of eight on F1 on both sets, ahead of CodeRabbit, GitHub Copilot and Claude Code by 7 to 14 points both times.
  • On 20 held-back pull requests, 17 plottypus comments missed a known bug, against 49 for CodeRabbit and 64 for GitHub Copilot, scored by the same judge.
  • 60%of comments rightHeld-back, run once: 20 pull requests in Go and Java, n = 44 comments, 95% interval 46 to 74%. 60.5% on the same languages we tuned on.
  • $0.24median per pull request, in model cost$0.64 worst per pull request. Held-back, run once, n = 20 pull requests in Go and Java.
  • 39.5 sfrom the mistake to the message to Claude, medianOur own synthetic tests, n = 18 planted mistakes, 16 caught.
  • $1.25an hour of live review, in model cost, on averageOne real session replayed twice, at $1.22 and $1.28 an hour.

14 Oct 2026 · reviewer Sonnet 5.5 · dollars are tokens at list price

The report

pull request results, two sets.

Public pull requests from the benchmark, side by side. Validation is in the same languages we tuned on, averaged over 4 passes, with its range across passes. Held-back is 20 pull requests in Go and Java, held-back, run once, with its 95% interval.

Validation and held-back results
MetricValidation same languages we tuned onHeld-back Go and Java, run once
Serious bugs found9 of 11 a pass (82%), range 8 to 10 across passesnot published
All known bugs found29 of 56 a pass, range 27 to 30 across passesnot published
F155.8, range 53.5 to 57.7 across passesnot published
Comments right60.5% of comments right60% of comments right, 95% interval 46 to 74%
Every row: size, counts, cost and time
More rows
MetricValidationHeld-back
Pull requests18, TypeScript, Ruby and Python20, Go and Java
Passes41
Known bugs, of them serious56, 1155, 23
Comments sent, of them wrongnot shown44, 17
Cost per pull request, median$0.19 per PR$0.24 per PR
Cost per pull request, worstnot shown$0.64 per PR
Time per pull request, medianabout 60 s170 s
Time per pull request, worstnot shown506 s
File lookups per pull request, mediannot shown11

Validation reposcal.comTypeScript, 7discourseRuby, 7sentryPython, 4

Held-back reposgrafanaGo, 10keycloakJava, 10

Notes on the held-back run

  • One run, so each figure carries its interval.
  • This was the first scored held-back run. Three early experiments on 8 Oct called the reviewer on some of these pull requests by mistake; nothing from them was scored or looked at.
  • In the run, the answer files were missing at first. They were rebuilt from the public known bugs and only the scoring was rerun; no review was rerun.

live sessions: our own synthetic tests.

Our own 15 scripted Claude Code sessions with 18 planted mistakes and 2 clean controls, run once on the shipped code.

  • 16 of 18planted mistakes caught
  • 13 of 16of those reached Claude before it finished
  • 0false flags; both clean sessions stayed silent
  • 39.5 smedian, mistake to message (167.6 s at the 90th percentile)

14 Oct 2026 · our own synthetic tests · 15 sessions · n = 18 mistakes + 2 clean · 1 pass

  • We wrote these sessions and their mistakes ourselves, and we tuned delivery on them. They show the plumbing works, not how it does on your code.
  • The other half of our scripted sessions is held back and has not been run yet.

what we're improving.

Serious bugs in languages we never tuned on
On 20 Go and Java pull requests, run once, it found fewer of the serious known bugs than on the languages it was tuned on. Some it noticed and then cleared as fine in the step where it checks its own suspicion. That step is what we are working on, and any fix will be tested on fresh, untouched Go and Java pull requests before we claim it.
Live use
Not yet measured at scale. Every number here is from a replay or a scripted run, not from a team using it day to day.
  • Serious bugs in languages we did not tune on: it finds fewer of them on Go and Java than on the languages it was tuned on.
  • It talks itself out of real problems. On the held-back set it noticed serious bugs and then cleared them as fine in the step that checks its own suspicion.
  • Subtle concurrency, like a counter that can lose updates when two jobs run at once: missed in every pass on one public pull request.
  • In our own scripted sessions it missed 2 of 18 planted mistakes, and 3 of the 16 it caught reached Claude after it had finished.
  • Every number here is from a replay or a scripted run. None is from a team using it live yet.

three sets, and why three.

  1. Design setnot published

    The pull requests we designed the reviewer on, in TypeScript, Ruby and Python. We tuned on them, so they say how well we fitted, not how well it works.

    tuned on

  2. Validation setsame languages we tuned on

    18 public pull requests from the same benchmark, in the same languages we tuned on. No tuning rule was written from them, but we used them to choose the model and to reject one prompt, so these numbers lean optimistic. The rule for passing was committed before the run.

    18 PRs · 4 passes · rule committed first (ccb5626)

  3. Held-back setrun once

    20 public pull requests in Go and Java from the Martian code-review benchmark, run once, never tuned on. We read only its totals. Once run, it is spent: a fix cannot be checked on it.

    20 PRs · 1 pass · frozen commit 20fa6d8

A set you tune on flatters you. A fresh set in the same languages shows whether the tuning was general or fitted. A held-back set in other languages, run once on frozen code, is the closest we can get to code we have never seen.

two real examples, one found, one missed.

Both are public pull requests from the validation set, rated serious by the answer key. Held-back items are never opened, so none is shown.

cal.com · pull request 8087

CriticalFound in 4 of 4 passes

Known bug, in the answer key's words

The code uses forEach with async callbacks, which causes asynchronous operations (e.g., calendar/video event deletions, payment refunds) to run concurrently without being awaited. This 'fire-and-forget' behavior leads to unhandled promise rejections, race conditions, and incomplete cleanup…

Our comment packages/trpc/server/routers/viewer/bookings.tsx:553

forEach(async) is used for the calendar deletions on reschedule. The deletions are not awaited, and the surrounding try/catch cannot catch errors from the async callback. … A failure in getCalendar or deleteEvent becomes an unhandled promise rejection, which can crash a Node process … Use Promise.all(refs.map(async ...)) with an await.

Show the code
bookingRefsFiltered.forEach(async (bookingRef) => {
  const calendar = await getCalendar(credentialsMap.get(bookingRef.type));

cal.com · pull request 14943

HighMissed in 4 of 4 passes

Known bug, in the answer key's words

Using retryCount: reminder.retryCount + 1 reads a possibly stale value and can lose increments under concurrency; consider an atomic increment via Prisma (increment: 1) to avoid race conditions (also applies to the similar update in the catch block).

Our comment

No comment matched it in any of the 4 passes. The answer key names no file or line, so we show none.

Show the code
retryCount: reminder.retryCount + 1

the method, and its limits.

answer key
The benchmark's known bugs for each pull request, each with a severity.
serious
A known bug the benchmark rates Critical or High.
judge
A language model decides whether a comment matches a known bug, using the benchmark's own judge prompt word for word. It is a Claude model, the same family as the reviewer, and it sends one comment with all of that pull request's known bugs in one call, where the benchmark makes one call per pair.
found
A known bug counts as found if at least one comment matches it. A comment that matches no known bug counts as wrong.
precision, recall, F1
Precision is the share of comments that are right. Recall is the share of known bugs found. F1 is the balance of the two (their harmonic mean), from 0 to 100.
hand check
On 8 Oct we checked 20 judge decisions by hand (10 matches, 10 non-matches, drawn with a fixed seed) and agreed with all 20. Outside the sample we found one we would reverse (too strict). The reviewer has changed since that run.
right % is likely a floor
On the design and validation sets we read the "wrong" pile against the code (one reader): about half were real defects the answer key does not list, and none were false. So the share of right comments is likely a floor. The held-back pile was not read.
cost
Dollars are tokens counted at list price.

cost and time.

Held-back, per pull request
$0.24 median per PR, $0.64 worst per PR
Held-back, per known bug found
$0.24 per bug found
Held-back, time per pull request
170 s median, 506 s worst
Validation, per pull request
$0.19 per PR, about 60 s
Live review
About $1.25 per hour of Claude working: one real session replayed twice, at $1.22 and $1.28 an hour.

Reviewer: Sonnet 5.5 (claude-sonnet-5-5), high effort on pull requests and medium effort in live sessions. Dollars are tokens counted at list price.

We do not show a cost for the synthetic sessions yet: two of our own records disagree, and we will publish it once they match.

what we do not publish, and why.

Numbers from the 6 pull requests we designed the reviewer on
We tuned on them, so they say how well we fitted, not how well it works.
Results from our own recorded work sessions
They are private, there is only one we can replay, and the reviewer was tuned on it. Only its cost per hour is shown.
Earlier runs (8 to 12 October)
They measured an older reviewer, and their dollar figures came from a tool estimate, not from tokens at list price.
The held-back pull requests one by one
We only read the totals, so the set stays untouched by us.