Cloudflare blocks AI crawlers by default
2025-07-01 · Publisher : Cloudflare · Domain : Artificial Intelligence
Cloudflare, which handles about 20% of web traffic, reverses the rule: AI crawling now requires the site's permission, and a pay-per-crawl model is tested in private beta.
What changes
- Every new domain is now blocked to AI crawlers by default, unless the owner explicitly allows them.
- Pay per crawl lets owners set a per-request price and returns an HTTP 402 "Payment Required" code to non-paying crawlers.
- AI crawlers are asked to declare their purpose: training, inference or search.
- More than one million customers already used the AI crawler blocking offered since September 2024.
- Condé Nast, Gannett, Reddit and Pinterest back the initiative.
On July 1, 2025, billed by the company as "Content Independence Day", Cloudflare becomes the first internet infrastructure provider to block by default AI crawlers that access content without permission or compensation. The CDN and security provider handles about 20% of global web traffic. ### What happened - **Blocking by default**: at sign-up, each new domain is asked whether it allows AI crawlers; otherwise, access is denied. The model moves from opt-out to opt-in. - **Declared purpose**: AI companies can state whether their crawler is used for training, inference or search, to help publishers decide. - **Pay per crawl**: in private beta, the publisher sets a flat per-request price and chooses, for each crawler, to allow, charge or block. The system relies on the HTTP 402 code, cryptographic authentication of crawlers (Web Bot Auth) and Cloudflare as merchant of record. - **Supporters**: Condé Nast, Dotdash Meredith, Gannett, Pinterest and Reddit, among others. ### Why it matters AI answer engines consume content without sending back as many visits as traditional search engines, which undermines publishers' business model. By acting at the infrastructure level, Cloudflare gives websites collective bargaining power and lays the groundwork for a market for content access, including for agents that could pay on their own within a budget. ### Caveats The mechanism only covers sites served by Cloudflare and assumes crawlers identify themselves honestly. Pay per crawl is only in private beta and its adoption by AI companies remains to be proven. ### Key takeaways For any website, AI crawler access becomes an explicit choice (allow, charge or block) rather than a given.
Links
- Cloudflare — Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large (communiqué)
- Cloudflare Blog — Introducing pay per crawl
- Cloudflare Blog — Content Independence Day: no AI crawl without compensation!
- MIT Technology Review — Cloudflare will now block AI bots from crawling its clients' websites by default
Tags : IA, Droit d'auteur, Web, Médias, Cloudflare