DX Today | No-Hype Podcast & News About AI & DX
Rick Spair
Podcast
Episodes
Listen, download, subscribe
Two Thousand Crawls for One Click: Cloudflare Flips the Web's AI Default, September 15, 2026
Two Thousand Crawls for One Click: Cloudflare Flips the Web's AI Default Today Cloudflare's new crawler defaults take effect, sorting automated visitors into three declared purposes, Search, Agent, and Training, and blocking the last two by default on ad supported pages. Chris and Laura dig into the mixed use crawler problem that puts Googlebot, Applebot, and Bingbot on notice, the crawl to referral ratios showing some AI companies taking thousands of pages for every visitor they send back, and the Pay Per Use marketplace meant to be the carrot behind the stick. They also ask the harder question: is a single private infrastructure company setting the terms of machine access to the open web a fix, or a new problem wearing a friendlier face? Hosted by Chris and Laura. The DX Today Podcast brings you daily deep dives into the most consequential stories in the AI ecosystem. #AICrawlers #Cloudflare #OpenWeb #AIPolicy #PublisherEconomics
DX Today | No-Hype Podcast & News About AI & DX RSS Feed
