A buyer evaluating a platform eventually asks whether it is tested. The answer is always yes, and the answer is close to useless. A suite reports on what it looked at. It says nothing at all about what it never opened, and it reports the same green either way.
So the question worth asking a supplier is the second one: what is in the corpus, and what happens when that corpus quietly gets smaller? We put it to ourselves last week, on the public storefront of VARGATES Medical, and did not much like the answer.
What the check was for
The storefront carries counts — clinical cases, anatomy models, specialties. A number on a marketing page is a promise about stock, and stock changes. So one of our checks exists solely to refuse a count typed into a page by hand: a figure has to be read from the catalogue that serves it, or be declared in a list with a written reason. It had been green for months.
That rule is about numerals, and it is worth saying where it stops: a sentence that says “hundreds” states no figure, so this check cannot see it. One is live on /platform as we publish this. That is a sentence for an owner to settle, not a guard.
It was green because it could not see seven of the pages it was meant to cover.
Two spellings of the same link
The check begins by asking the site what its pages are: it reads the header and collects every internal address there. It collected them like this.
{ labelKey: "nav.blog", href: "/blog" } // read
<Link href="/platform"> // not readA menu entry declared as data and the same entry written as markup are the same thing to a visitor. To a regular expression they are not. Seven pages lived in that difference:
/platform/hardware/cases/catalog/demo/dashboard/login
Two of them, /platform and /hardware, are top-level menu entries. The first had been stating counts a reader would reasonably take as inventory while the catalogue behind them served fewer — and, for one of the two tiles a catalogue could fill, a library that was empty.
The pin that reported a blind spot as a repair
There had been an eighth page outside the corpus: the homepage. It was found months ago, in the same way, and it was fixed by writing the string "/" into the collector by hand.
The pin is gone. The homepage now arrives the way every other page does: the header links to it.
A page that is eight lines long
A second mechanism kept the same pages out. Having found a page, the walk followed its imports into the shared component library and stopped there. But /platform is an eight-line file that does nothing except render a client component sitting beside it — a file of over two hundred lines. This framework pushes everything interactive across that boundary, which is exactly where a marketing page's visible text ends up.
The scanner read the four lines of code that render nothing, found no claims in them, and reported the page clean. It was not wrong about what it read.
The corpus is printed now, and pinned in both directions
The old collector covered 20 pages and 51 files. The same walk, reading both spellings and following a page into its own client component, finds 27 pages and 77 files. Both numbers are now asserted, and printed on every run:
storefront corpus: 27 pages, 77 filesPinned in both directions, which is the part that matters. Growth is a decision: a new menu entry is new public surface, and its claims want reading. Shrinkage is the failure this whole exercise was about — a renamed route, a header rewritten into a shape the collector does not read, a component moved. A corpus that quietly shrinks produces precisely the same silent green as a clean storefront.
The same shape, twice more
With the corpus corrected, two further guards turned out to have the same shape of defect. They share no code and do opposite things, but each reached the code it protects and could not see inside it, because no fixture had ever put the subject where it would fail.
The first reads sentences on a page and reports ones left in English. Its pattern for "the text between two tags" excluded braces — so any sentence with a value interpolated into it vanished from the guard entirely. That is most sentences that carry a number. An English sentence had been sitting inside an otherwise fully translated administrator panel, reporting completions over the last thirty days to a Russian-speaking administrator, and the file carried no line in the guard's baseline at all.
The second reads a number written beside its label. It looked for the value in value position:
value: "159", // caught
value: isCases ? "159" : t("stats.value"), // silent
value: metrics.cases ?? "159", // silentThe second and third forms are not exotic mutations invented to embarrass the rule. They are the natural regression here: the tile disappears while the catalogue is empty, the next person reads the gap as a bug and writes ?? "159" so the old number shows again — restoring the exact claim that was removed. Six forms are now named in a list, each with its own fixture, and the names are pinned to each other — not to a count, because a count is edited by changing one digit. A seventh form cannot arrive without its fixture.
The numbers nobody can check
Not every figure can be sourced from a catalogue. Some are positions rather than row counts: the same content can honestly be counted more than one way, and which number a buyer should read is a decision for a person, not for a script that picks the larger.
Those live in a register, each with a written reason. The register is pinned to its exact contents — adding an entry means editing the pin, which puts the decision into a diff a human reads. It holds four entries today. The point is not that four is a good number. The point is that it cannot become five quietly.
If you are the one evaluating a supplier
Three questions worth putting to anyone, including us:
What is in your test corpus, and where is that number written down?
What happens if the corpus shrinks? A check that silently covers less is the most expensive kind, because its output is indistinguishable from the good outcome.
Which numbers on your public pages are read from the product, and which are typed by a person?
We would rather answer those with the machinery than with a slide. If you are evaluating VARGATES Medical for an institution or a territory, ask us the awkward version of any of them.