12 November 2026 · 4 min read
How accurate is CodeRabbit? A test on 52 real bug fixes
Recall, precision and false alarms for CodeRabbit and a local AI reviewer, on the same pull requests.
IThe short answer
On 52 pull requests built from public bug-fix history, CodeRabbit found about half of the known bugs. Its recall was 0.48 and its precision was 0.23. On clean pull requests, it raised 1.18 false alarms per PR.
My own local reviewer, reviewer, found fewer bugs, with a recall of 0.37. It raised about half as many false alarms: 0.63 per clean PR. The F1 scores were close: 0.31 for CodeRabbit and 0.29 for reviewer.
So CodeRabbit catches more bugs. A local reviewer with a narrower prompt can be quieter on clean code. Neither tool finds most bugs on its own.
IIHow the test works
Each test PR comes from a real upstream bug fix in an open-source project, such as Hono, ripgrep, bat, Django, Clap and Caddy. A script forks the project, applies the change and opens the PR. Some PRs contain the known bug. Others are clean.
Both tools review the same PRs. A scorer matches each review comment to the known bug lines, with an optional LLM judge, and reports recall, precision and F1 with intervals.
The benchmark PRs are public in mrwick1/reviewer-bench, so you can run any other review tool on the same set.
IIIResults
| Tool | Recall | Precision | F1 | False alarms per clean PR |
|---|---|---|---|---|
| CodeRabbit (1 run) | 0.48 | 0.23 | 0.31 | 1.18 |
| reviewer, 2 passes (3 runs) | 0.37 | 0.24 | 0.29 | 0.63 |
Sonnet as the model for reviewer, judged scoring, 31 October 2026.
Recall is the share of known bugs that a tool flagged. Precision is the share of its comments that pointed at a real bug. A false alarm is a comment on a PR that has no known bug.
In a three-PR cost check, reviewer cost about $0.09 in model usage per review. That cost depends on the model and on the PR size.
IVWhich one to use
Use CodeRabbit when you want the most bugs caught and a hosted service. Its docs list GitHub, GitLab, Azure DevOps and Bitbucket, plus IDE and CLI integrations.
Use a local reviewer when you want your own model choice, your code to stay on your machine, or fewer comments on clean PRs. You pay the model cost per review.
Whichever tool you pick, measure it on your own bugs first. A reviewer that sounds smart but raises false alarms costs more time than it saves.
VLimits
CodeRabbit has one run in this comparison, and the set has 52 PRs, so the intervals are wide. CodeRabbit changes over time, and a later run can give other numbers. reviewer is my own tool and its source is private. I built it, so read the numbers with that in mind.
VIQuestions
How many bugs does CodeRabbit find?
On 52 pull requests built from real bug fixes, CodeRabbit found about 48% of the known bugs (recall 0.48), with precision 0.23. This was one run, judged on 31 October 2026.
Does CodeRabbit give false positives?
Yes. On pull requests with no known bug, CodeRabbit raised 1.18 false alarms per PR in this test. A local reviewer with two passes raised 0.63.
Is there a local alternative to CodeRabbit?
Yes. You can run an AI reviewer on your own machine with your own model. My reviewer runs from the terminal, posts inline GitHub comments and cost about $0.09 per review in model usage. Its source is private, but the benchmark PRs are public.
How can I benchmark an AI code reviewer?
Build pull requests from real upstream bug fixes, add clean pull requests, run each tool on the same set, and match the comments to the known bug lines. Report recall, precision and false alarms per clean PR, with intervals.