XML sitemap

A sitemap answers one question: which URLs does this site consider worth knowing about. That is a discovery aid, and it is genuinely useful on sites where internal linking is poor, where pages are added faster than they are linked, or where sections sit deep in the structure.

It is not a ranking signal and it is not an instruction. A page listed in a sitemap that Google judges thin, duplicated or low value stays out of the index, and Search Console will report it under Discovered, currently not indexed. That status is frequently read as a crawling problem when it is a verdict.

On a small, well-linked site a sitemap adds almost nothing, because every page is already reachable in two clicks. That is worth saying to clients who have been told a sitemap will fix their indexing, since it is one of the more common pieces of bad advice in circulation.

Where it genuinely earns its place is as a statement of intent that can be compared against reality. Submitted versus indexed is a two-number diagnostic, and the difference between them is the list of pages the site wanted indexed and did not get.

A sitemap should contain canonical, indexable URLs that return 200. Every entry is a claim that this address is the definitive version of a page worth indexing, and anything failing that description contradicts the claim.

The usual contaminants are all produced automatically. Redirecting URLs left behind after a migration. Pages carrying a noindex tag, which state the opposite of what the sitemap states. Parameter variants that canonicalise elsewhere. URLs returning 404 because the sitemap regenerates on a schedule rather than on publish.

None of those breaks anything outright, and that is the problem: the site is simply telling Google two different things and letting it choose. A sitemap that is half wrong is a weaker signal than no sitemap at all, because it says the site does not know its own state.

The check is mechanical and takes minutes: crawl the sitemap, confirm every entry returns 200, is self-canonical, and carries no noindex. On most client sites that crawl finds something on the first run.

The last-modified date is the one part of a sitemap that carries real information, and it is the part most implementations get wrong.

Done properly, it says when the content of the page genuinely changed, and a crawler can use it to prioritise: pages that changed are worth revisiting, pages that have not are not. Done as most CMS plugins do it, it is set to the date the sitemap was generated, so every URL claims to have changed this morning.

A field where everything changed today carries no information, and a crawler that learns a site lies about it stops using it. Google has been explicit that it ignores lastmod when it does not trust it, so the cost of getting this wrong is not a penalty, it is the silent loss of the one useful signal in the file.

Worth checking rather than assuming, because it is invisible from the front end. Open the sitemap, look at twenty entries, and see whether the dates differ from each other. If they are all identical and recent, the field is decoration.

The most valuable thing a sitemap provides is not discovery. It is a denominator. Search Console reports how many of the submitted URLs are indexed, and that comparison converts a vague worry into a list.

On arriving at a new client site, this is usually the fastest useful diagnostic available. Six hundred submitted and two hundred indexed is a specific problem with a specific set of pages behind it, and the coverage report names the reason for each: duplicate without user-selected canonical, crawled but not indexed, soft 404, excluded by noindex. Each reason points somewhere different.

It also makes the sitemap worth splitting. Separate files for products, articles and category pages mean the indexed ratio can be read per type, and a section that indexes poorly becomes visible instead of being averaged into a single number that looks acceptable.

What this diagnostic is not is a crawl budget finding. On a site of a few thousand pages, a low indexed ratio is a quality, duplication or intent problem, and the coverage reason will say which.