The Codebase Assurance Index, explained

What we measure — the Codebase Assurance Index

The Codebase Assurance Index is one reproducible 0–100 score for a whole codebase — whatever languages it is written in, all of them at once. Ten lenses roll up into it: most dimensions measured by deterministic tools, a few given an advisory, tolerance-banded LLM read, all of them versioned and published. Same code in, same number out. A measurement, not an opinion.

Survey a repo — free Browse the open standard →

A quick orientation

What makes the Codebase Assurance Index different.

Reproducible

Every dimension is computed by a deterministic tool reading your code. Commit + frozen rubric → exactly one score. Same commit, same rubric, same advisory data — same number.

Depth is never gated

Every survey — including the free first report — computes the full CAI: all dimensions, all lenses. You never pay for depth; you pay for breadth and cadence.

The same rubric for every language

A product built in several languages isn't scored one language at a time and averaged. Every file is measured against the same rubric and folded into the same ten lenses, so a 74 means the same thing wherever the code came from. When we add support for a new language, the rubric doesn't change — which means your existing scores don't move.

Verifiable

Read the exact rule each dimension was scored by, and run the same measurement on your own code — the algorithm and rubric are open.

Verify a score yourself →

What "reproducible" means here

Same commit, same rubric version, same number — on any machine, on any day. The deterministic dimensions are just tools and thresholds. The few that use an LLM are tolerance-banded: the model's read has to fall inside a band to count, and a result that sits on a boundary is re-run, not rounded. The rubric is versioned, and a scoring change ships as a new version with a published changelog — so your trend line is never quietly re-based under you.

How we measure

Benchmarks, allocation-aware APIs and async hygiene.

Dimensions →

The full vocabulary — every dimension, its evaluator and rubric version — lives on the open standard. Browse the catalog →

The firewall

Nothing moves the number but the code.

The deterministic score sits on one side of a firewall; an advisory LLM read sits on the other — and it can never cross. That's the difference between a measurement and asking an LLM, which answers differently every time.

The AI only ever advises

A few findings get an advisory, tolerance-banded LLM read that can never, by construction, move the headline number. It explains in plain English; it never scores. The measurement stays pure.

Your inputs never score

Your compliance declarations, a suppressed finding, your contract profile — they change what the artifact says, never the CAI. A declaration is presentation; the score is measurement. Neither party to a contract can tilt the number — only the code changing moves it (or a disclosed advisory refresh like a new CVE).

The full firewall, drawn on the standard →

From the lenses to one number

How the lenses roll up.

The CAI is a weighted roll-up of the lens scores under the frozen rubric - not an average you can't see inside.

Core always counts; conditional lenses only when they apply

The five core lenses always contribute. The conditional lenses contribute only when the code calls for them, and the weights re-normalise — so a repo is never penalised for a lens that doesn't apply.

A critical lens caps the headline

The roll-up can't read Strong while a lens reads Critical: a single critical-band lens caps the CAI, so the one number can't hide a serious failure in one dimension behind strong scores elsewhere.

Mined from git history, not just the code

Four behavioural dimensions — hotspots, bus factor, knowledge freshness, change coupling — are read deterministically from your git history and scored into the same CAI.

If we can't measure it, we say so

Some lenses go deeper in some languages than others. Where a lens can't be measured properly it returns not measured, with the reason — and the weights re-balance, exactly as they do for a lens that doesn't apply. Your score is never quietly reduced because of the language you chose, and a gap is never dressed up as a zero.

So a contract floor of CAI ≥ 80 means every always-on lens is Strong or better with no lens Critical — decomposable, not opaque. The authoritative spec: codeassuranceindex.info/spec

The architecture, mapped

The proof no line-scanner can produce.

The Architecture lens isn't a metric on a file — it's the shape of the whole system. This is a live C4 map of a real published repository: its bounded contexts and the calls between them, coloured by health. Read straight from the code and rendered the same way the app draws it.

How to read the number

One fixed scale, five bands — and a pin that never moves for anyone.

Every score renders on the same worst→best scale: Critical / Weak / Adequate / Strong / Exemplary, cut at 25 / 50 / 70 / 90. The pin marks the score's exact spot — position on the fixed scale *is* the reading, never a corpus-relative rank.

Noise is how these tools die

Every team has installed something that flagged three thousand issues on day one. It got suppressed, then ignored, then nobody opened it again. A finding you don't trust is worse than no finding, because it costs attention and returns nothing. So detectors are tuned per language against real production code, and each one is tested twice: that it fires on the actual defect, and that it stays silent on the idiom. A shared base class isn't low cohesion. A Rails concern isn't a god object. An interface a façade is obliged to implement isn't dead code.

Calibrated, not noisy

A reading you can act on — not a thousand findings to triage.

Watchdog is tuned against real production code in each language it surveys, so the normal patterns a line-level checker trips over — a repository that shares a base class, a test that asserts through a harness, an interface a façade is obliged to implement — are not reported as faults. Zero setup, no rule-tuning weekend: the false alarms are tuned out before you ever see them.

Tuned on real code

Every detector is tested to fire on the real defect and stay quiet on ordinary code, against public reference projects in each language. .NET has been tuned longest, and each language is held to the same test suite before it goes live. The published surveys are open to read, so you can check the findings against a repository you already know rather than take a rate on trust.

Disagree, and it learns

Any scored finding can be disputed in one click — routed to human triage, and a confirmed false-positive becomes a detector test so it can't recur. The instrument sharpens with use; the score never bends to the dispute.

Quiet by design

Findings are ranked by what moves the grade, folded so one stray stub barely registers, and a lens returns not-measured, with the reason rather than a phantom zero. Volume is never mistaken for rigour.

Calibration is an ongoing programme — idiom-heavy codebases still surface residual noise we keep tuning down, and every dispute feeds the next round.

Worked example — cohesion

LCOM4 counts how many disconnected clusters a class splits into — and provably mis-measures some good designs. A well-designed domain aggregate (an Order with AddItem, ChangeShipping, Cancel) is cohesive by its invariant, yet looks like a god-class to the raw metric. Watchdog recognises the shapes LCOM4 provably mis-measures — domain aggregates, data-access repositories, source-generated view-models, contract-mandated plumbing — and exempts them, while still flagging the real god-object. The result: a cohesion signal you can act on, not a list to triage.

The same logic applies to a Java aggregate, a Kotlin sealed hierarchy or an Elixir context, because what is being recognised is how the code holds together — not the language it is written in.

Slop by shape, not authorship

The hollow code that compiles — measured deterministically.

The failure mode of fast, high-volume code isn't bad syntax — a linter catches that. It's code that looks finished and does nothing. Watchdog measures that emptiness directly — by the shape of the code. And shape carries across languages: a CalculateTax that just returns 0 is the same finding in C#, in Java, in Python and in Go, scored the same way in each.

IC1

Stubs that look implemented

A CalculateTax that returns 0, an async method that never waits, empty types, dead branches. It compiles and type-checks — in any of them — and it does nothing.

D10

Tests that assert nothing

Passing tests with no assertions, in xUnit, JUnit, pytest, RSpec, Jest or ExUnit alike; skipped tests dressed up as coverage. Coverage counts the test file; it never asks whether the test checks a result.

Errors made invisible

An empty catch {}, a swallowed except: pass, an ignored error return, a rethrow that loses the stack trace. Type-based checks pass; the failure disappears.

D17

Untracked debt & dead code

TODO/FIXME/HACK, blanket suppressions, commented-out code, unreferenced symbols — counted raw at best by other tools; scored here.

Copy-paste, never parameterised

Near-duplicate blocks — type-aware detection, scored by density across the codebase.

We don't guess whether a machine typed it — stylometric "AI detection" is a credibility trap, and we refuse to make the claim. We measure the hollowness by shape, so a rushed human and an eager model produce the same finding and the same fix.

Rubric versioning

Freeze the rubric, keep the score constant.

Watchdog scores with a versioned rubric. Any change that can move a score for unchanged code bumps the rubric version.

Versioned and contestable

The rubric is contestable: a scoring change that isn't reflected in the published spec fails our CI, so every number stays re-derivable from a rule you can read.

Rubric versions & governance →

Contract rubrics

Pin a repository to a frozen rubric and the ruler stops moving under you — the same commit re-scores to the same number under that rubric, so any movement you see is the asset changing, never the ruler. Advisory data still refreshes, so a new CVE can legitimately move a security finding — a real signal, disclosed in the changelog.

The noise benchmark

The method is public before the numbers are.

Watchdog measures how much of its own output is worth acting on, under a protocol written and published before the measurement runs. Publishing the method first is the point: a rate whose definition can be chosen after the result is not a measurement. No Watchdog noise result exists yet, and none will be quoted here until the run completes and its evidence is published alongside it.

Available now

The measurement protocol - the frozen five-class taxonomy, the noise formula and its denominator rule, corpus and freshness rules, judging and statistics.
Reproduction instructions - how to re-run the measurement against the same corpus.
The five-class judge prompt - the exact instructions the machine judge is given.

Read the protocol and judge prompt

Published with the run

The training-repository exclusion list, sealed before the holdout is selected.
The corpus manifest, pinned to exact commits and hashed before any tool is scanned.
Raw and normalised findings, every machine and human verdict, and the agreement statistics.
Statistical outputs, sensitivity analyses and every deviation from the protocol.

A valid result is published whether it meets our target or misses it.

Watchdog's target is an audited broad-noise rate below 5% on a fresh, sealed holdout that the engine has never been trained on. That is a target, not an achieved result.

Get the measurement. No depth is ever gated.

Survey a repo — free Verify a score yourself →