How the number
is actually made
A benchmark is only as trustworthy as its definitions. This is the full protocol behind the 186 domains on the benchmark page: what counts as correct, how the domains are labeled, what we compare against, and the one distinction that matters most, between a benchmark and the real world.
Top-1 and top-3
The task is single-answer: given a domain, name the organization that operates it. Top-1 counts a domain as correct when our highest-confidence entity is the right operator. Top-3 counts it when the right operator is anywhere in our first three ranked entities.
On the current set that is 183 of 186 for top-1 (98.39%) and 99.46% for top-3. Every number on the benchmark page and this one is read from the same committed result file at build time, so the copy cannot drift from the measurement.
What "correct" means, exactly
A prediction matches the label when the two names are equal after canonical normalization: Unicode NFKC folding, stripping common legal suffixes (Inc, LLC, Corp, GmbH and the like), removing punctuation, collapsing whitespace, and lowercasing. Labels are stored already normalized, and the engine emits normalized names, so the comparison is apples to apples.
It is deliberately strict on the entity, and we would rather show you that than hide it. It does not strip every national legal form, so mullvad vpn does not match mullvad vpn ab: we identified the brand but not the exact registered entity, and that scores as a miss. This makes the headline a conservative floor, not a generous one.
186 domains, hand-labeled, machine-enforced
The set spans 12 difficulty categories, from clean corporate sites to redacted, CDN-masked, and post-acquisition edge cases. Each entry captures the same seven required signals so the engine scores every domain on equal footing, and each label is tied to a dated, independent source.
The protocol is not aspirational, it is enforced by the schema that loads the set. An entry is rejected unless its label is already normalized, its source URL and source type are present, and all seven required signals are captured. Labels older than 90 days raise a staleness warning. In short: the set cannot contain an unsourced or undated answer.
Seven required signals
Accepted label sources
Where we are grading our own opinion
For most domains the operator is a fact. For subsidiaries, acquisitions, and genuinely ambiguous cases it is a choice, and the label encodes ours: we attribute to the operating entity, not always the ultimate legal parent. Reasonable people can disagree, so we say so rather than pretend the answer key is objective everywhere.
Our own misses are exactly these calls. We score espn.com as espn, not Disney, even though Disney owns it, because ESPN is who operates the domain. An independent source proves a filing or a certificate exists; it does not settle brand-versus-parent. That call is ours, and the benchmark is partly measuring agreement with it.
98.39% of what? Against a naive tool, 70.43%
An accuracy number means nothing without something to beat. We compare against the dumbest reasonable tool: read the WHOIS registrant organization, and treat a redacted or privacy-proxied record as no answer. Scored with the identical match function, that baseline gets 70.43% of the set, so the engine's real contribution is the 28-point gap, not the raw 98.39%.
The gap is not spread evenly, and that is the point. On clear, acquisition, and rebrand domains the naive baseline already scores 100%, so the engine adds nothing and the category is simply easy. All of the lift is concentrated where WHOIS fails: privacy-redacted and privacy-conflicting domains go from 0% to 100%, and ambiguous from under 40% to 100%. The per-category table on the benchmark page shows every row.
Small samples, honest error bars
Some categories hold only ten to fifteen domains, and a point estimate on a sample that small can mislead. So every accuracy figure carries a Wilson 95% confidence interval. The overall top-1 is 98.39%, with a 95% interval of 95.37% to 99.45%.
Read the intervals, not just the points. A category showing 100% on ten domains is not proof of perfection: its honest lower bound sits around 70%. We would rather publish the wide interval than a precise-looking number the sample cannot support.
In-sample benchmark vs real-world
Here is the honest limit of this benchmark. The scoring engine has been developed and tuned against these 186 domains. Measuring on the same set you tuned against gives an upper bound, not the accuracy a stranger's domain will see tomorrow. We treat the 98.39% as exactly that: a ceiling on a curated set.
The real-world number comes from a separate suite that runs against the live production API, with live infrastructure, live redaction, and all the noise a curated capture smooths away. That out-of-sample figure is the one to trust for what your own domains will do, and it is lower. The benchmark page reports both, side by side, with dates, rather than letting the flattering number stand alone.
We are not standing up a frozen hold-out set and calling it science this week. A hold-out is only meaningful if it stays untouched over time, so we are building that as an ongoing discipline rather than manufacturing a number for it now.
Do the confidence scores mean anything?
A confidence score is only useful if it is honest. We bucket predictions by the confidence we reported and check how often each bucket was actually right. The average gap between claimed and actual, the expected calibration error, is 0.0354, about 3.5 points.
One caveat we will not bury: a large share of predictions land at full confidence and are trivially right, which flatters any aggregate calibration figure. The bucket that actually tests calibration is the low-confidence one, where we report cautiously and beat our own score. The full bucket table is on the benchmark page.
Deterministic, dated, and honest about the gaps
How it runs
The benchmark is deterministic: the same captured inputs always produce the same score, so a result is auditable rather than a lucky sample. It reruns on every change to the scoring engine, and the committed result file is dated and versioned, so a regression cannot ship quietly and the number on the site always traces to a specific run.
What we hold back, and why
We do not publish the raw domain list: it is our regression set, and publishing it invites gaming and leaks the exact cases we test against. We also do not publish per-signal weights or the vendors behind each source. The outcomes are open and audited; the recipe stays ours.