Crawl budget

Crawl budget is the number of URLs a search engine is willing and able to crawl on your site in a given period. It depends on how fast your server responds and on how much Google thinks your pages are worth revisiting. Google describes it as the combination of crawl capacity, what your server can handle, and crawl demand, how much it wants your content.

It is a real constraint only on large sites, and it is invoked far more often than it applies. A site of a few hundred pages will not run out. Even so, the way crawlers spend their time on a small site still affects how quickly new and updated pages get picked up.

Where it bites is on sites that generate URLs: faceted navigation, search result pages, session parameters, endless pagination, and old URLs that keep returning errors. A crawler spends its allowance on those and never reaches the pages that matter. It is one of the arguments for keeping a generated URL set small and deliberate rather than publishing every combination a template can produce.

These terms get used interchangeably and mean different things. Google’s documentation describes crawl budget as the result of crawl capacity and crawl demand. “Index budget” is an industry phrase for a related idea: Google does not index every page it crawls.

The practical consequence: crawling and indexing are separate steps. A status of “Crawled, currently not indexed” in Search Console usually points to quality or duplication, and more crawling will not change it.

• In Google Search Console, open Settings and then Crawl stats. Note total requests per day, average response time and host status.
• Look at the breakdown by response. A high share of 404s, redirects or server errors means crawling is spent on URLs that give nothing back.
• Look at the breakdown by purpose, discovery of new URLs versus refresh of known ones. If new pages take weeks to be discovered, check internal links and the sitemap.
• Crawl the site with Screaming Frog and compare the number of URLs found with the number you expect. A large gap usually means parameters, filters or pagination are generating URLs.
• If you can access server logs, analyze Googlebot requests to see which sections it visits most. Screaming Frog Log File Analyser does this.
• Clean up: redirect or remove dead URLs, stop internal links from pointing to parameter variants, keep sitemaps to canonical indexable URLs, and improve server response times, with PageSpeed Insights for page-level speed.
• Check Crawl stats again after a few weeks to confirm the share of useful requests went up.

• Crawl stats in Search Console: requests per day, response times and the share of 404s and redirects.
• Old URLs still returning errors, and whether they deserve a redirect.
• Parameters, filters and internal search pages that generate near-infinite URLs.
• Thin or template pages that should be improved, merged or set to noindex.
• XML sitemaps that list only canonical, indexable URLs.

Search engines are no longer the only bots visiting. AI companies run their own crawlers, such as GPTBot and OAI-SearchBot for OpenAI and PerplexityBot for Perplexity, and they all use server resources. On a small site this rarely matters. On a large or slow site heavy bot traffic can slow responses, and Google lowers its crawl rate when a server slows down. Decide in robots.txt which AI crawlers you allow, knowing that blocking a search crawler such as OAI-SearchBot or PerplexityBot can keep your pages out of that assistant’s search answers.

AI Overviews rely on Google’s index, so anything that delays Google’s discovery of new pages also delays their chance of appearing in an AI answer. In local SEO the risk shows up when location pages are generated in bulk: hundreds of near-identical city pages compete with the useful ones for crawl attention.

1. Small business.Rarely a real limit. Focus on fixing 404s and making sure every new page is linked internally and listed in the sitemap.
2. E-commerce.The classic case. Filters, sorting, variants and internal search can multiply URLs many times over, and canonicals, parameter handling and robots.txt rules decide where the crawl goes.
3. Multilingual site like ours.Each language adds a full set of URLs, and every hreflang pair has to be crawled on both sides. Keeping both versions free of dead links matters more as the site grows.
4. Local service business.Location pages should exist only for areas you actually serve and have something useful to say about. The same reasoning led us to set our thin template pages to noindex.

A quick way to know which group you are in: compare the number of pages you expect with the number of URLs a Screaming Frog crawl finds. If the crawl finds many times more, the site is generating URLs and crawl budget deserves attention.

• Worrying about crawl budget on a small site while ignoring basic indexation problems.
• Blocking pages in robots.txt that also need a noindex, which stops Google from seeing the directive.
• Leaving years of old URLs returning 404 after a redesign.
• Publishing every page a template can generate just because it is possible.

Crawl efficiency is part of the technical side of traditional SEO, and it matters most when a site grows through templates, as with location pages in local SEO.