How we measure
Every number on the landing comes from a run you can reproduce. The datasets, the scoring, and the honesty rules — and the head-to-head protocol we publish before the results, on purpose.
What we report today
Three measurements back the claims on the landing. Each is labelled with its dataset and sample size — nothing is rounded up, and there is no fabricated competitor column.
| Measurement | Result | Dataset |
|---|---|---|
| Specificity (no false alarms on clean code) | 1.00 | board v2 — 58 inverted real-fix pairs; the fine-tuned model raised zero findings on the clean side. |
| Recall (reintroduced defects caught) | 0.66 | same 58-pair set; balanced accuracy 0.828 vs 0.655 for the untuned base model. |
| Verdict accuracy (fine-tune A/B) | 83.3% → 95.8% (+12.5pp) | held-out set, n=48; specificity to 100%; invalid-JSON outputs 4/48 → 0. |
| Held-out real Drupal MR pairs | recall 0.70 / specificity 0.95 | v7 on real merge-request pairs outside the training data. |
Why "inverted-fix" pairs
Each pair takes a real merged Drupal fix and inverts it: the "defect" side re-introduces the exact bug the maintainer fixed, the "clean" side is the merged code. That gives a ground-truth label for every case — a finding on the defect side is a true positive, a finding on the clean side is a false alarm. It measures the two things that matter for a second reviewer: does it catch real, maintainer-confirmed defects, and does it stay quiet on code that's actually fine.
Head-to-head protocol (planned)
We have not yet run a blind head-to-head against CodeRabbit. The protocol is fixed and public before the run, so the result can't be reverse-engineered to flatter us. When it runs, we publish every metric it produces — including any that favour CodeRabbit.
- Dataset: 20–30 real, public Drupal merge requests (drupalcode.org core + contrib), each a pre-fix diff that re-introduces a known defect, mixed across security (XSS / SQLi / access), Drupal idioms (DI, cache metadata,
#[Hook], deprecated→D11), plus a clean control set to measure false alarms. Public MRs only — no private code touches either tool. - Procedure: the same diffs through both tools — Drureview via
drupal-codereviewer review --diff-file <mr>.diff --output json, CodeRabbit via its review comments on each MR. A reviewer scores each finding against the gold labels blind (not knowing which tool produced it): true-positive / false-positive / missed, tagging Drupal-idiom catches. - Metrics reported: recall (overall + Drupal-idiom subset), specificity / false-alarm rate on the clean set, the Drupal-idiom catches a generic tool missed, plus n, date, and tool/model versions. The MR list and gold labels ship with the results so anyone can reproduce.
Honesty rules
- No fabricated metrics, ever. Numbers appear only with their dataset and n.
- If CodeRabbit wins on overall recall, we say so. The pitch is specificity + Drupal-idiom + on-prem privacy + one-time — not "catches more bugs".
- External vendor-published stats may appear only if clearly attributed as vendor claims, never presented as our measurement.
- The method, the MR list, and the raw scores are published — reproducible or it doesn't count.
Source for the head-to-head protocol: docs/COMPARISON_METHODOLOGY.md on GitHub.