Google Updates Crawl Budget Guidance: 304 and Limits
Google updated its official crawl budget documentation, adding a direct recommendation to return HTTP code 304 (Not Modified) to Googlebot for pages that haven’t changed, and, for the first time, explicitly explaining that the crawl limit is shared across all Google crawlers. This isn’t a new algorithm or a one-off event; it’s a clarification of rules technical SEO has already been living by for a while, but now it’s officially documented rather than just established industry practice.
What happened
On July 22-23, 2026, Google updated the “Optimize your crawl budget” section of its Search Central documentation (part of the Crawling infrastructure section). The documentation clarifies three things.
First, every site starts with the same conservative crawl capacity limit. Google states directly that if there’s demand to crawl more, and the site consistently responds quickly and reliably, the system automatically raises this limit over time. In other words, the limit isn’t fixed forever; it’s “earned” through reliable server performance.
Second, this limit is shared across all Google crawlers at once. Each crawler (the regular Googlebot, Googlebot-Image, AdsBot, and others) has its own crawling demand, but the upper bound on capacity is a single, site-wide ceiling. If one crawler is loading the server heavily, that leaves less capacity for the others.
Third, the documentation now includes a direct recommendation: support the HTTP code 304 (Not Modified). If a page hasn’t changed since the last crawl, the server can return 304 instead of the full content; Google reuses the cached version, and the server saves bandwidth and resources. The practical effect is that Googlebot spends the freed-up crawl budget on pages that have actually changed, instead of re-fetching content that hasn’t.
These recommendations don’t apply indiscriminately to every site. Google explicitly names the audience: large sites with over a million pages, and medium-sized sites with over 10,000 pages that update frequently.
Why it matters
For sites that fall under these thresholds, marketplaces, large catalogs, news outlets, crawl budget directly affects how quickly new or updated pages even make it into search results. If Googlebot is spending its resources re-crawling thousands of unchanged product listings instead of new listings or updated prices, indexing of new content slows down invisibly, without any error or warning in Search Console.
304 caching used to be a well-known web performance practice, recommended more for site speed than for SEO. Now it’s an official recommendation specifically in the context of crawl budget, an argument that’s easier to bring to a development team when you need to justify prioritizing a technical task that won’t show visible results within a single sprint.
How it works
| Before | Now | What to do |
|---|---|---|
| The crawl capacity limit was described generally, without details on sharing between crawlers | The limit is now explicitly described as shared across all Google crawlers at once | Check in Search Console → Crawl Stats which crawler types are using which share of the limit |
| 304 caching was general web performance advice | Direct Google recommendation specifically for saving crawl budget | Check via server response headers (curl -I with If-Modified-Since) whether the server supports 304 at all |
| It was unclear why new large sites got crawled more slowly than expected | Explained: every site starts with a conservative limit that grows over time with stable server performance | Don’t try to force the limit artificially; focus on server response stability instead of trying to “beat” the conservative start |
Technically, nothing breaks or gets disabled automatically; this isn’t a forced transition like some other Google updates. It’s documentation that formalizes what some technical SEO specialists were already implementing, and it gives a clearer pointer to where to look if a large site is crawling more slowly than you’d like.
Who this applies to
Google itself named the direct threshold of applicability: sites with over a million pages, and medium-sized sites with over 10,000 pages that change frequently. That’s marketplaces, large e-commerce catalogs, classifieds boards, news portals, and aggregators.
For a typical business site with a few dozen or a few hundred pages, a landing page, a services site, a small online store, this update effectively changes nothing: at that volume, Google manages to crawl the entire site with plenty of headroom even without any special measures. Spending development resources on 304 caching for the sake of crawl budget doesn’t make sense here; the priorities remain the basics: load speed, correct indexing, no duplicate content.
A separate category is scaling sites: if your catalog currently has a few thousand products but is planned to grow past 10,000 within six months, it’s worth building 304 support into the architecture ahead of time rather than retrofitting it later.
Editorial analysis
We haven’t yet run a separate measurement of exactly how much reindexing speed improves after implementing 304 specifically following this documentation update; it’s too fresh to have gathered statistics across client accounts. But checking server response codes is already a standard part of the technical SEO audit we run for any site with a large catalog, well before this clarification from Google. This news doesn’t change our methodology; it confirms officially that the direction we’ve taken is correct, and gives us an argument that’s easier to present to a client or development team.
A practical example
Imagine a catalog with 40,000 product pages, of which roughly 5% (price, availability, description) actually gets updated each week. If the server returns code 200 with full content for every page regardless of whether it changed, Googlebot spends the same amount of resources on the 2,000 pages that genuinely changed and the 38,000 pages that stayed the same. If the server correctly returns 304 for those 38,000 unchanged pages, the freed-up resource goes toward re-crawling the 2,000 that did change, faster, and the updated price or availability status shows up in search results sooner.
What to do right now
- Check the size and update frequency of your catalog: does the site fall under Google’s thresholds (1 million+ pages, or 10,000+ pages that change frequently)?
- If it does, check with a developer whether the server actually returns 304 for unchanged pages, or always serves a full 200 response.
- Look at the breakdown of response codes and crawler types in Search Console → Crawl Stats.
- For sites that are actively growing, build 304 support into the technical spec ahead of time, not after page volume already becomes a problem.
- For smaller sites, don’t prioritize this; check the basics first: load speed, duplicate pages, correct robots.txt and sitemap.
Expert tip. Before rolling out 304 broadly, test it selectively on non-critical pages first, such as archived products, old news items, or utility pages. This lets you confirm the caching is configured correctly before extending it to the whole catalog.
Something to watch for. 304 only works when the server accurately determines that a page hasn’t changed. If a caching layer (CDN, proxy) is misconfigured and returns 304 even for a page with an updated price or description, Google simply won’t see the change, and reindexing of the real update will be delayed. Check the logic for determining “the page hasn’t changed” separately from the mere fact that a 304 code is technically being returned.
FAQ
What is crawl budget? It’s the amount of resource Google is willing to spend crawling a specific site over a given period: how many pages Googlebot manages to visit, and how often, based on crawling demand and the server’s capacity.
Do I need to urgently change anything on my site because of this update? No, not for most sites. The recommendation specifically targets large catalogs and sites with over 10,000 pages that update frequently; for a typical business site with dozens or a few hundred pages, crawl budget usually isn’t a bottleneck.
What is a 304 Not Modified code, and how does it differ from 200? A 200 code means the server returned the full content of the page. A 304 code means the page hasn’t changed since the last request, and the client (in this case, Googlebot) can use its existing cached version without re-downloading the content.
Does this update directly affect a site’s search rankings? No, it’s not a ranking factor. The effect is indirect: proper handling of 304 and crawl budget speeds up how quickly Google notices and reindexes changes on a site, but doesn’t affect the page’s ranking position itself.
How do I check whether my server supports 304?
You can send a request with an If-Modified-Since or If-None-Match header to a specific page (for example, via curl -I) and check the response code. If the page hasn’t changed since the specified date but the server still returns 200 with full content, 304 support isn’t there.
Does this apply to Google Merchant Center and product feeds separately? The documentation addresses Googlebot’s crawling of site pages, not the Merchant Center product feed; these are different data update mechanisms. The direct 304 recommendation applies specifically to a site’s HTML pages.
Where in Search Console can I check current crawl statistics? Under “Settings → Crawl Stats” for a specific property; it shows the breakdown by response code, file type, and Google crawler over the last 90 days.
Conclusion
This documentation update doesn’t change the rules of the game for most sites; it formalizes what technical SEO specialists on large projects already knew from practice, and gives a clearer reference point for anyone working with large catalogs or scaling sites. The main practical takeaway is simple: if your site is large and updates frequently, check whether the server correctly returns 304 for unchanged content, and whether it mistakenly does the same for content that actually changed.
If you’d like to check whether crawl budget is even a bottleneck for your site, and what exactly should be fixed first, let’s discuss it in a technical audit. These checks are part of our SEO services, and if your site is still being built or reworked, 304 support and other technical requirements can be built in right from the development stage.