On 5 June 2025, San Francisco startup Perplexity AI accused security provider Cloudflare of wrongly blocking traffic from its AI answer assistants. Perplexity argues the requests are human-initiated, comply with published identification guidelines, and should not be subject to the robots.txt restrictions intended for automated crawlers.
Perplexity AI Assistants vs. Cloudflare Blocking
In a blog post titled Agents or Bots? Making Sense of AI on the Open Web, Perplexity challenged a Cloudflare article that grouped the service with bots said to ignore robots.txt. Perplexity says its assistant retrieves pages only after a user submits a question, discards the text once an answer is generated, and therefore behaves like a standard browser session.
The company claims Cloudflare’s automated rules label this traffic as unverified scraping, preventing users from accessing publicly available information. Key points from Perplexity’s response include:
- Every request carries the “PerplexityBot” user-agent string.
- The same IP address that fetches the content returns the answer directly to the user.
- Fetched text is not stored, indexed, or used for model training.
- Premium content behind paywalls is never displayed.
- The firm has applied to join Cloudflare’s Verified Bots allowlist.
Perplexity warns that misclassification could stifle emerging AI tools that respect publishers’ boundaries.
Background Context
Cloudflare’s 30 May 2025 post “Protecting Websites From Abuse by AI Models” introduced new safeguards to curb large-scale scraping that disregards robots.txt. The company operates a Verified Bots program that currently whitelists compliant crawlers such as Google and Bing, while unverified services face stricter rate limits.
Founded in January 2022, Perplexity has raised more than 70 million dollars and offers domain-specific assistants for finance, academic research, and news - all of which rely on real-time web retrieval.
The debate turns on a 1994 convention: robots.txt tells search engines which pages to crawl. Observance is voluntary, and little guidance exists for AI systems that fetch content only when prompted by a user. Industry bodies, including the Interactive Advertising Bureau, continue to call for clearer standards, while publishers negotiate direct licensing deals with firms such as OpenAI and Meta.
Source Citations
- Perplexity AI – “Agents or Bots? Making Sense of AI on the Open Web” (5 June 2025)
- Cloudflare – “Protecting Websites From Abuse by AI Models” (30 May 2025)
- Cloudflare – Verified Bots Documentation
.svg)
.svg)
.svg)