01 Quality
Recall and false positives.
Which combinations found the real issues without flooding review with noise.
03 Client confidential · AI reliability engineering
AI that finds real bugs without inventing them.
Eight review skills, multiple models and harnesses, and a benchmark grounded in an eight-sprint bug inventory.
The work.
We analyzed the complete bug inventory from eight sprints, including real P1 and P2 reports, then prepared a sample of 12 diffs with seven known positives. Three of those positives were P1 or P2 issues.
Eight code-review skills were tested across different models and harnesses. We also adjudicated 80 additional findings, including false positives, so the comparison could measure more than whether a tool found the seeded issues.
The result.
The recommendations reduced false-positive noise in code review and improved the precision of issue-severity assessment.
- Client
- Confidential
- Benchmark
- Historical bugs and prepared diffs
- Compared
- Review skills, models and harnesses
- Recommendations
- CI and local use
Benchmark inventory
Known positives. Adjudicated noise.
8Sprints of bug reports analyzed.
8Code-review skills tested.
12Prepared diffs in the sample.
7Known positives in those diffs.
3Known positives at P1 or P2.
80Extra findings adjudicated, including false positives.
02 Judgement
Severity precision.
How accurately each combination assessed the seriousness of a finding.
03 Economy
Findings per dollar.
The cost of useful findings, not just the volume of review output.
The comparison produced separate recommendations for CI and local use.
Local implementation workflow
Review. Fix. Prove it twice.
After the first development pass, the agent enters a review-fix loop. It keeps going until two consecutive reviews are green.
The average number of review cycles needed to reach that condition fell from five to two.
5Average review cycles before.
2Average review cycles after.
2 greenConsecutive reviews required to finish.
Measure the reviewer
Know what the workflow finds. And what it invents.
We measure your review workflow and show you where quality, cost or false-positive noise is slowing delivery. Explore our AI reliability engineering service.