Google Search Advocate John Mueller has addressed ongoing reports of "phantom" noindex errors in Google Search Console, responding to a site owner who said Search Console was flagging noindex issues the site could not reproduce.
Google on phantom noindex errors in Search Console
Mueller discussed the issue in a recent Bluesky exchange that began with a site owner's question posted about persistent noindex errors. In earlier cases he had checked, Mueller said a noindex directive was always present somewhere in the response sent to Google, even when it was not obvious to site owners.
He noted that in some situations the noindex directive was only shown to Google, which made debugging significantly harder.
The reported case involved a long-running Search Console issue. Key points included:
- The site had been seeing noindex errors in Search Console for four months.
- The site owner said there were no noindex directives in the page HTML or in robots.txt.
- Despite this, Search Console continued to report a robots meta tag noindex error on affected URLs.
Mueller reiterated that previous examples he had reviewed always contained a noindex instruction somewhere in the response to Google. He invited the site owner to send example URLs privately so he could investigate further.
Technical context and troubleshooting methods
A noindex directive tells Google not to include a page in its search index. When a URL submitted in a sitemap also sends a noindex signal, Search Console can report the issue as "Submitted URL marked 'noindex'."
Search Engine Journal writer Roger Montti outlined several technical scenarios that can create hidden noindex signals, including cached HTTP response headers at the server or CDN level that preserve an outdated noindex header for some clients. He highlighted cases where Cloudflare returned a status code 520, which Cloudflare can use when blocking a user agent.
According to Montti, inconsistent Cloudflare responses can lead different HTTP header checkers to see different results. For example, one tool, such as KeyCDN, might receive a 520 status from Cloudflare while another, such as SecurityHeaders.com, sees a normal 200 status. This variation can hide or expose a noindex header depending on the tool used.
Montti recommended using Google's Rich Results Test to see exactly what Google receives when it fetches a URL. The tool sends a request from Google data center IP addresses and shows both the HTTP response and the rendered page. Montti wrote that a page sending a noindex signal will show no structured data results in the Rich Results Test, which helps reveal cases where servers present different content or headers to Google than to other clients.
The Rich Results Test currently uses the Google-InspectionTool user agent instead of the standard Googlebot string. Montti noted that requests from this tool originate from Google IP ranges and pass reverse DNS checks for Google domains. He also suggested testing pages with tools configured to use the Googlebot user agent string, citing Google's User Agent Switcher extension for Chrome and the Screaming Frog crawler as useful options for this kind of troubleshooting.
Official tools and documentation
Google documents the robots meta tag and noindex directive in its Search Central guidance for site owners. The official Rich Results Test is available for checking structured data implementation and examining how Google fetches and renders a page.
Google also publishes information about its crawlers and inspection tools, including user agents such as Google-InspectionTool and Googlebot. Montti's article in Search Engine Journal summarized these troubleshooting approaches and detailed his Cloudflare header observations in the context of phantom noindex reports.
.svg)
.svg)
.svg)