Google confirmed at Search Central Live Asia Pacific 2025 that "soft 404" pages still consume a site's crawl budget even when the server returns a normal 200 OK status. Search Analyst Gary Illyes shared the clarification during an on-stage Q&A session in Singapore.
Why soft 404s drain crawl budget
Illyes explained that Googlebot first records the HTTP status, then inspects the page content. If the text signals "not found" or other low-value messaging, the crawler labels the URL as a soft 404. The page is removed from the index, but the fetch and render still count toward the site's allotted crawl requests. Hard 4XX errors do not have this impact because Google drops them before any content analysis.
Soft 404 quick facts
- Definition: An HTML page that says a resource is missing yet returns 200 OK.
- Effect: Excluded from search results and charged against crawl budget.
- Where to spot them: Search Console > Indexing > Pages > "Soft 404".
- Typical causes: Out-of-stock products, empty category pages, expired profiles, blank search results.
- Fixes: Serve 404 or 410 for removed content, 301-redirect moved items, or keep the page live with useful information.
Why this matters
Google has flagged soft 404s since 2011 and revised its crawl-budget policy in 2017, yet it had never explicitly stated that these pages pull from the quota. For large e-commerce and news sites with millions of URLs, wasted crawls can delay discovery of new or updated content.
Sources
Remarks by Gary Illyes at Search Central Live Asia Pacific 2025 and an attendee summary from Kenichi Suzuki on LinkedIn.
Inside Google's Universal Commerce Protocol that lets AI agents tap carts, catalogs and loyalty pricing
Google quietly upgrades AI shopping protocol: what Cart, Catalog and Identity Linking change next
Google and DocMorris Launch AI Health Companion for Europe - What Changes Next
Worried About Endless 404 Reports In Search Console? John Mueller Reveals What They Really Mean.svg)
