reviewer
An AI code reviewer that runs on my machine, measured against CodeRabbit.
IWhy I built it
I wanted pull-request review like CodeRabbit that runs on my own machine, with my own model choice. I also wanted to know if it was good. A review tool that sounds smart but misses bugs, or raises false alarms, costs more time than it saves.
So the evaluation came before most of the features. Every change to the prompt or the pipeline runs against the same set of real bugs.
IIHow a review runs
reviewer <PR-url or branch> runs from the terminal. No server and no daemon are needed.
reviewer watch runs on a systemd timer and reviews every new PR head.IIIMeasured against CodeRabbit
I built 52 test PRs from public bug-fix history in projects such as Hono, ripgrep, bat, Django, Clap and Caddy. 42 contain a known bug and 16 are clean. A script forks each project, applies the change and opens the PR, so CodeRabbit and reviewer review the same PRs. A scorer matches findings to the known bug lines, with an optional LLM judge, and reports intervals instead of one number.
| Tool | Recall | Precision | F1 | False alarms per clean PR |
|---|---|---|---|---|
| reviewer, 2 passes (3 runs) | 0.37 | 0.24 | 0.29 | 0.63 |
| CodeRabbit (1 run) | 0.48 | 0.23 | 0.31 | 1.18 |
Sonnet as the model, judged scoring, 31 October 2026.
CodeRabbit still finds more bugs. reviewer is close on F1 and raises about half as many false alarms on clean PRs. In a three-PR cost check, the model cost was about $0.09 per review.
IVWhat measuring changed
An agentic deep pass, where the model searches the repository with tools, sounded like the obvious upgrade. On the eval set it scored F1 0.33 against 0.38 for the plain review, so it did not become the default.
Two finder passes raised recall but tripled the cost at first. The passes ran in parallel and could not share the prompt cache. Running them in sequence halved the cost, from about $0.18 to $0.09 per review.
Limits. CodeRabbit has one run in the comparison, and the set has 52 PRs, so the intervals are wide. The source is private. The benchmark PRs are public.