A← Index

reviewer

An AI code reviewer that runs on my machine, measured against CodeRabbit.

RoleSolo: design & build
TypeDeveloper tool · LLM evaluation
StackRust · Claude · GitHub CLI
SourcePrivate

IWhy I built it

I wanted pull-request review like CodeRabbit that runs on my own machine, with my own model choice. I also wanted to know if it was good. A review tool that sounds smart but misses bugs, or raises false alarms, costs more time than it saves.

So the evaluation came before most of the features. Every change to the prompt or the pipeline runs against the same set of real bugs.

52PRs from real bug fixes
16open-source projects
~$0.09model cost per review
0.63false alarms per clean PR

IIHow a review runs

reviewer <PR-url or branch> runs from the terminal. No server and no daemon are needed.

Read the intent
It reads the PR title and description, so it judges the change against what the author meant to do.
Check out and gather context
It checks out the head commit in a detached worktree, runs the linters and collects the callers and definitions of the changed code.
Find and merge
Claude reviews the diff in one or more passes. The passes run in sequence, so the second pass reuses the prompt cache. Duplicate findings within three lines are merged, and the list is capped at ten.
Post
It posts inline comments with the GitHub CLI. reviewer watch runs on a systemd timer and reviews every new PR head.

IIIMeasured against CodeRabbit

I built 52 test PRs from public bug-fix history in projects such as Hono, ripgrep, bat, Django, Clap and Caddy. 42 contain a known bug and 16 are clean. A script forks each project, applies the change and opens the PR, so CodeRabbit and reviewer review the same PRs. A scorer matches findings to the known bug lines, with an optional LLM judge, and reports intervals instead of one number.

ToolRecallPrecisionF1False alarms per clean PR
reviewer, 2 passes (3 runs)0.370.240.290.63
CodeRabbit (1 run)0.480.230.311.18

Sonnet as the model, judged scoring, 31 October 2026.

CodeRabbit still finds more bugs. reviewer is close on F1 and raises about half as many false alarms on clean PRs. In a three-PR cost check, the model cost was about $0.09 per review.

IVWhat measuring changed

An agentic deep pass, where the model searches the repository with tools, sounded like the obvious upgrade. On the eval set it scored F1 0.33 against 0.38 for the plain review, so it did not become the default.

Two finder passes raised recall but tripled the cost at first. The passes ran in parallel and could not share the prompt cache. Running them in sequence halved the cost, from about $0.18 to $0.09 per review.

Limits. CodeRabbit has one run in the comparison, and the set has 52 PRs, so the intervals are wide. The source is private. The benchmark PRs are public.

SignedArjun