Crawl Budget: What It Is and When You Should Care

By Faaiz Mahmood · Updated 11 September 2026 · 8 min read

Crawl budget is the most over-applied concept in technical SEO. It is real, it matters enormously on large sites, and it is irrelevant to the overwhelming majority of people who read about it. Before spending a weekend on it, it is worth finding out which group you are in.

1. What it actually is

Google does not crawl the web without limit. Every site gets an effective allowance of fetches over a period, and Google describes it as the product of two separate things.

Crawl capacity. How hard Google thinks it can hit your server without degrading it. Googlebot watches response times and errors. Serve quickly and reliably and it crawls more freely. Start returning 500s or taking three seconds a page and it backs off, deliberately, to avoid knocking you over.

Crawl demand. How much Google wants your pages. Popular URLs, pages that genuinely change, and sites that regularly publish something worth having all raise demand. Stale pages and pages Google has already decided are low value lower it.

Both have to be there. A blazing server on a site nobody wants gets crawled lightly, because capacity was never the constraint.

2. Whether this applies to you

Google's own guidance is blunt about this, and it is worth repeating. If your site has fewer than a few thousand URLs, it will generally be crawled efficiently and crawl budget is not something you need to manage.

You should care if you recognise your site here:

• Over ~10,000 URLs, particularly if the count grows on its own.
• Faceted navigation. A shop with six filters that combine freely can generate more URL permutations than it has products, by orders of magnitude.
• A large site that changes constantly, like classifieds or job listings, where a page crawled a week late is worthless.
• New pages taking weeks to appear in search when the site is otherwise healthy.

If none of those describe you and pages are not being indexed, the cause is almost certainly content quality or a blocking rule, not budget. Check noindex vs disallow before going further here.

3. Reading the Crawl stats report

Search Console has the data under Settings, then Crawl stats. Three things are worth looking at.

Total crawl requests over time. You want this roughly stable or rising. A sustained drop that lines up with a server change or a traffic fall is worth investigating.

Average response time. This is the number with the most leverage on the whole page. Googlebot throttles itself against a slow server, so response time is effectively a ceiling on your crawl rate. Getting a 1,200ms average down to 300ms can raise crawl volume substantially without touching anything else.

Crawl requests by response code. Here is the cheapest win available. If a meaningful share of requests are returning 404s, 301s or 500s, that share of your allowance is being spent on nothing. A site burning 30% of its crawl on redirect chains is a genuinely common finding.

4. Where budget actually leaks

In rough order of how much they cost on a typical large site:

Faceted navigation and parameters. The biggest offender by a wide margin. Every combination of colour, size, sort order and page number is a distinct URL returning near-identical content. Left open, this can multiply a 5,000 product catalogue into hundreds of thousands of crawlable addresses.

Internal search results. Pages of the form /search?q=anything are infinite by definition. There is no reason for a crawler to be in there.

Redirect chains. Each hop is a separate fetch. A five-hop chain spends five requests to deliver one page. See redirects.

Soft 404s. A missing page returning 200 with an empty template looks like a real page to a crawler, so it keeps getting crawled forever. HTTP status codes covers why this is worse than a plain 404.

Thin, auto-generated archives. Tag pages holding one post each, date archives nobody reads, author pages on a one-author blog.

5. Fixing it, in the order that works

Make the server faster. Unglamorous and the highest leverage thing on the list. Caching, a CDN, and cutting slow database queries raise the ceiling for everything else.

Block the infinite spaces in robots.txt. Parameters and internal search are exactly what robots.txt is for, because the goal is to stop the fetch rather than to hide the page:

User-agent: *
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*?utm_

Sitemap: https://example.com/sitemap.xml

One warning worth repeating: if those URLs are already indexed, blocking them now freezes them in the index, because the crawler can no longer read a noindex tag. Remove them first, then block. The robots.txt generator will format the rules correctly.

Fix the chains and the dead ends. Point every redirect straight at its final destination. Return a real 404 or 410 for pages that are gone.

Clean the sitemap. Canonical, indexable, 200-status URLs only, with honest lastmod dates. Google uses those dates to prioritise recrawls, so an accurate sitemap directs budget toward what changed.

Tidy internal linking. Crawlers follow links. Pages buried six clicks deep get crawled rarely. If something matters, link to it from somewhere shallow.

6. Things that do not help

A few persistent myths, worth naming so you do not waste time:

• The crawl-delay directive. Google ignores it completely. Bing supports it. Adding it does nothing for your Google crawl rate.
• Submitting the sitemap repeatedly. Resubmitting an unchanged file does not buy extra crawling.
• Requesting indexing for hundreds of URLs. The tool is rate limited and intended for one-off fixes.
• nofollow on internal links to save budget. This has not worked as a crawl control since nofollow became a hint in 2019.

7. A realistic expectation

Crawl budget work is maintenance, not a growth lever. Done well on a large site it means new pages get found in hours instead of weeks, and changes to existing pages are noticed promptly. That is worth real money on a classifieds site or a large shop.

It will not make a page rank that would not otherwise have ranked. If Google is crawling your pages and choosing not to index them, no amount of budget work changes that verdict. The answer there is a better page.

For a full pass over the rest of your setup, the technical SEO audit runs through the checks in a sensible order.