Cloudflare separates search access from AI-training preferences

Cloudflare introduced Disallow AI Training on 15 September for site owners who want search indexing without permitting training use. The setting publishes the applicable preference in robots.txt while allowing mixed-use crawlers classified as accountable. Other training crawlers are blocked. Cloudflare named Apple, Google and Microsoft as operators meeting requirements or making time-bound commitments.

The company also changed Block and Block on pages with ads to apply to mixed-use crawlers. Blocking such a crawler can therefore remove its search access as well. The narrower setting depends on operator mechanisms and commitments; it is different from a technical guarantee about every subsequent use of fetched content. Cloudflare described further controls over AI summaries as future work, rather than a capability fully delivered with this announcement.