The sitemap Google was never given: nine days and three impressions

This section has carried a new material every day since 1 September. On 12 September we read what Google Search had done with its first nine days: three impressions across the whole section — one for the section page and one each for two materials. The rest of the site was being shown roughly 20 to 56 times a day over the same period, so the property worked and the data had arrived.

A zero has two causes

A page that earns nothing in search is in one of two states, and from the outside they look identical. Either people were shown it and did not need it, or nobody was shown it because the search engine had not been told it was there. The first is a reason to change what we write about. The second is a delivery defect, and changing the subject in answer to it would treat the symptom and leave the cause in place. So delivery was ruled out first.

What the console believed

A sitemap is the file through which a site tells a search engine which addresses it serves. Ours is built by the site itself, is listed in robots.txt, and was correct. Search Console did not hold a registration for it. It held seven others — the sitemaps of the WordPress site this domain used to run, all submitted on 15 November 2022.

Every one of them answered 404, and every one carried an error. That is the part worth keeping. An error on every sitemap had been the normal state of the property for almost three years, and against that background the absence of the file we actually publish did not stand out at all. Google could still find our file through robots.txt; nobody had ever handed it over.

The repair took minutes. The live sitemap was submitted and fetched within a second, with no errors and 42 addresses — the number the live file listed — and the seven dead registrations were removed. Making sure nobody has to spot this by hand again took the rest of the day.

A healthy registration is not a page that was read

The next question was whether the materials themselves were in the index. On 12 September, of the twelve materials then in the sitemap, seven were indexed and five were reported as Discovered – currently not indexed: Google knew the address and had never fetched it.

A related gap sat on our side. Materials here are published by an HTTP call, not delivered with a deployment of the site. The IndexNow notification — the protocol through which a site tells Bing, Yandex and other participating engines that an address has appeared — had stayed a command run by hand after a deploy. Once publication stopped being a deploy, nothing pushed a new material to those engines. Google does not take part in IndexNow; for Google the sitemap remains the channel.

Publication now announces the material in the same request, under four rules:

• a material that has never been public is not announced: its address answers 404, and a crawler sent there learns only that the page does not exist;

• a material that is withdrawn from public view is announced, because its address has just stopped answering, and that is a change a crawler has to be told about;

• a re-publication that changes nothing a reader would see is not announced, so repeated deliveries of the same text do not spend the quota;

• a draft that becomes public is announced even though its text has not changed, because for the first time there is something to open.

A refused announcement does not undo the publication. The material is already on the site; the response and the log say that the announcement failed, and why.

The file itself was short

The last defects were inside the sitemap itself, and the largest of them no test could see. The same code produced 61 addresses on a developer's machine and 42 in production. Part of the list came from a table in the repository, read from disk at the moment the sitemap was requested. The server function's bundle does not carry that directory — deliberately, since it would add megabytes to every function — so in production the read failed, and the failure was caught and turned into an empty list. Nothing broke. The site simply told search engines less about itself.

Every test stayed green, because tests read the sitemap where the files are, and a test run cannot reproduce the environment in which the file is missing. The set of addresses is now produced by a script from the site's own routes, kept in the repository and compiled into the bundle. The disk read is gone, an empty source is reported as an error instead of passing for “nothing to list”, and a page added to the site's code without regenerating the list turns a test red. The same change removed two smaller gaps even from the developer's count: an old exclusion list written for the previous site was hiding a live page of ours, and a hand-kept list of routes had fallen eight routes behind.

On 14 September — after those repairs, and with the materials published since — the live sitemap listed 76 addresses, and the copy Google last fetched held 76.

What watches this now

Each of these failures was silent, so each now has something that says so out loud. The section's report:

• compares the number of addresses in the live sitemap with the number Search Console holds, and prints both when they differ;

• fails when the property holds no sitemap registration or more than one, names any registration with errors on a line of its own, and requests every registered address to confirm it answers;

• asks, for every published material, whether it is in the index and when it was last crawled, and fails when a material more than seven days old is still outside.

A separate search report splits search clicks into searches for our own names and everything else. In the section's report, a check that could not be carried out prints “not measured” instead of passing. An unperformed check looks exactly like a passed one, and here that resemblance lasted almost three years.

What this does not claim

None of it moves a position. It settles only whether an address has been presented at all — necessary, not sufficient. Between 15 August and 11 September, every search click that the query breakdown could attribute came from someone typing one of our names, and the other queries we appeared for averaged a position around 48 — well past the first page of results.

As of 14 September, ten of the thirteen materials in this section have a recorded crawl. Three do not: yesterday's, one from 9 September, and the very first, from 1 September, which Google now reports it does not know at all — exactly the case the index check exists to report. A crawl is an event, not a ranking. Whether people find us through what we do rather than by our name is measured again on 12 October, over a closed window.