Crawl budget

Crawl budget is not a number Google allocates. It is the outcome of two independent things, and they fail differently, so it is worth holding them apart.

Crawl capacity is how hard Googlebot is willing to push. It watches response times and errors, and backs off when a site struggles, because crawling is not worth degrading a site for its actual visitors. A slow server therefore lowers the ceiling directly, and no amount of sitemap work raises it.

Crawl demand is how much Google wants the site. Popularity, staleness and how often content genuinely changes all feed into it. A site whose pages never change earns fewer visits, and correctly so.

The practical read: capacity problems look like a fast site being crawled thoroughly or a slow site being crawled sparsely. Demand problems look like a site being crawled sparsely regardless of speed, which is usually a signal about the content rather than the infrastructure.

Google published guidance on this and it is unusually direct: crawl budget is a concern for sites with upwards of roughly a million unique pages, or for large sites whose content changes frequently, such as a listings site with a fast-moving inventory. Most agency clients are neither.

Below that threshold, a page that is not indexed is almost always not indexed for a different reason, and Search Console names it. Discovered but not indexed, crawled but not indexed, duplicate without user-selected canonical, soft 404. Each of those points somewhere specific, and none of them points at budget.

This matters because crawl budget is the most satisfying wrong answer available. It sounds technical, it sounds like expertise, and it moves responsibility to the crawler rather than to the content. An agency that diagnoses it on a six hundred page brochure site has bought itself a quarter of work that cannot produce anything.

The honest sequence is to read the index coverage reasons first, and only reach for budget if the site is genuinely large and the logs show the crawler spending itself somewhere useless.

Where crawl budget does bite, the cause is rarely the number of real pages. It is the number of addresses, and those two diverge violently once anything is parameterised.

Faceted navigation is the classic. Five filters with six values each, combinable and orderable, produce tens of thousands of addresses from a few hundred products. Add a sort parameter and a page number and the space is effectively unbounded. A crawler will work through it, because it has no way of knowing the combinations are worthless.

Calendars are the purest form of the problem: a date widget with next and previous links generates an infinite series, and crawlers have been found working through years of empty months. Session identifiers in URLs do the same thing with a different mechanism, generating a fresh address for every visit.

The handling is to stop the addresses existing rather than to ask for more budget. Facets with no search demand should not be linked in a crawlable way, calendars should stop generating links past a horizon, and session state belongs in a cookie. Canonicals help with consolidation but they do not save the fetch, because the crawler has to fetch the page to read the tag.

Server response time is the lever with the most direct effect and the one least often presented as SEO work. Halving response time raises crawl capacity mechanically, and it helps real visitors at the same time, so it is rarely a hard case to make.

After that it is subtraction rather than addition. Remove the addresses that should not exist, fix redirect chains so a fetch does not cost three, return proper 404s instead of soft ones, and stop linking to things that are going to be excluded anyway.

Sitemaps help discovery, not budget. They tell a crawler what exists; they do not raise the ceiling, and submitting a larger sitemap on a site with a crawling problem tends to make the symptom worse rather than better.

The measurement worth doing, on a site large enough to warrant any of this, is server log analysis: what Googlebot actually fetched, how often, and how much of it was worth fetching. Everything else in this area is inference, and on a large site the inference is usually wrong.