Cloudflare Boots Perplexity's Bot for Stealth Crawling - What Sparked the Sudden Ban

Cloudflare has removed Perplexity AI’s web crawler from its Verified Bots program and is now blocking the bot network-wide. The security action, announced on 18 June 2024, follows evidence that Perplexity’s crawler ignored robots.txt directives and disguised itself to continue collecting web pages.

Why Cloudflare acted

In a detailed post according to Cloudflare, engineers found that Perplexity rotated IP addresses, switched Autonomous System Numbers, and spoofed a Chrome 124 user agent to bypass standard blocks. These tactics violate the Verified Bots policy, which requires fixed IP ranges, consistent user-agent strings, and strict observance of robots.txt rules. Cloudflare has now added managed firewall rules that automatically deny both declared and stealth traffic linked to Perplexity.

Key details

• Action date: 18 June 2024
• Affected user agents: “PerplexityBot,” “Perplexity-User,” and traffic masquerading as Chrome 124 on macOS
• Policy breached: Verified Bots requirements for robots.txt compliance and transparent IP disclosure
• Enforcement: Perplexity delisted as a verified bot; new heuristics block all related requests
• Site owner impact: Publishers that still want to allow the crawler must now create explicit firewall exceptions

Program background

Cloudflare launched the Verified Bots initiative in 2021 to give reputable crawlers unthrottled access to protected sites while limiting abuse. Participants agree to publish fixed IP addresses, use stable identifiers, and honor site directives. Perplexity AI - which powers a question-answering platform that relies on large-scale web data - was initially approved under this program before the recent violations surfaced.

Perplexity’s response

In a statement published the same day, Perplexity argued that user-initiated assistant requests differ from automated scraping and said it is reviewing Cloudflare’s claims. The company added that it is “open to constructive dialogue” about industry standards for responsible crawling. Read the full Perplexity response.