Crawl budget: when it matters and when it does not
What crawl budget is, which sites actually need to think about it, and how to stop search engines wasting time on URLs that do not matter.
· 2 min read
"Crawl budget" is one of the most discussed and least needed topics in technical SEO. For most sites it is not a problem at all. For a few, it decides how quickly new and changed pages appear in search.
What it is
Google describes crawl budget as the combination of two things:
- Crawl capacity: how much Googlebot can crawl without overloading your server. It rises when the site responds quickly and falls when it slows down or returns errors.
- Crawl demand: how much Google wants to crawl, based on how popular and how often updated your pages are, and how many URLs it knows about.
Who needs to think about it
Google's own guide on crawl budget is written for:
- very large sites, roughly a million or more unique pages whose content changes moderately often;
- medium or large sites, roughly ten thousand or more pages, with content that changes very quickly, such as daily;
- sites with many pages reported as "Discovered - currently not indexed" in Search Console.
If your site has a few thousand pages and new pages are usually crawled within days, crawl budget is not your problem. Look at content quality and internal linking instead.
What wastes crawling
- Faceted navigation and parameters creating huge numbers of URL combinations.
- Infinite spaces, such as calendars that link to every future month.
- Duplicate URLs from session IDs, tracking parameters and inconsistent internal links.
- Soft 404s and long redirect chains.
- Slow server responses, which lower crawl capacity.
How to improve it
- Keep crawlers away from URLs that do not need crawling, with robots.txt rules for parameter and filter patterns. See robots.txt vs noindex: blocking crawling is the right tool here, noindex is not.
- Return proper status codes: 404 or 410 for pages that are gone, so they are crawled less over time.
- Keep sitemaps clean and current, listing only canonical, indexable URLs with accurate last-modified dates.
- Link internally to the URLs you want crawled, not to their variants. See Internal linking.
- Make the server fast and stable.
How to see it
The Crawl Stats report in Google Search Console (under Settings) shows requests per day, response times, status codes and what Googlebot fetched. Server logs show the full picture of what crawlers request.
Work on a very large site? Share what worked in the Technical SEO community.