How we validate ourselves

We measure how often our findings are wrong, on codebases we did not tune against. A language is not presented as validated until that measurement has passed both a machine gate and an independent human one.

The number that matters is how much of what we report is noise — findings that do not hold up when someone examines them properly. A detector that fires constantly is not thorough; it is expensive, because every false finding costs a person time and costs the whole report credibility.

So findings are adjudicated one at a time against the real codebase, and each is resolved into one of a few outcomes: it stands, it is wrong and the detector must change, or the detector was right but said the wrong thing and the advice must change. Only the first counts as a true finding.

Detectors are improved by running them against real open-source projects and fixing what they get wrong. That work cannot then be used to prove they are accurate — a detector tuned until it is quiet on a specific codebase tells you nothing about the next one.

So the two activities use different code. Before a validation cycle begins, part of the corpus is set aside as the validation set. The improvement work during that cycle runs only on the remainder, and never on the set-aside part. When the cycle ends, the system is measured against the part it was never allowed to touch.

Being precise about what this does and does not guarantee. The separation holds within a cycle: nothing measured at the end of a cycle was tuned against during it. Cycles are long, and between them the corpus is refreshed and the sets are drawn again — a project used for improvement in an earlier cycle may appear in a later validation set. Real codebases change substantially over that span, so by then it is a different codebase in every way that matters to a measurement.

1. The measurement. The system runs across the validation set and every finding is adjudicated, producing a noise rate for that language. This gate is machine-run and repeatable.
2. Independent human agreement. A sample of the same judgements is reviewed by a rater from outside the team, and their verdicts are compared with ours. What is tested is not the score but whether our judgements are consistent with a competent outsider's.

A language that has not cleared both gates is not presented as validated. Passing the first alone is not enough, and the second is deliberately outside our control.

When adjudication shows we were wrong, the fix goes into the detector, not into the report. The cases are:

• The detector is wrong — it is corrected, and the correction must demonstrably change the outcome, not merely silence that one case.
• The detector is right but the advice is wrong — the finding stays and the wording is fixed. A correct finding explained badly is still a defect, and suppressing it would hide a real problem.
• We cannot yet stand behind it — the measurement becomes advisory: reported and explained, but excluded from the score until it earns its place.

This is where the honesty machinery described in How we measure comes from. The distinction between "we checked and it is clean" and "we could not establish this" exists because we measure how often we are wrong, and know what we are not yet entitled to claim.

Noise rates, per-language status and the corpus itself are not recorded here — they change continuously. A validated result carries its own figures, alongside the rubric version it was measured under.