A robots.txt census of the Tranco top 1,000,000 domains (June 2026): which AI crawlers does each
site block? Every domain's /robots.txt is fetched and parsed for 20 AI-crawler user-agents (GPTBot,
CCBot, ClaudeBot, Google-Extended, Bytespider, …) and whether each is Disallow: /'d. This is the
dataset behind the Crawlora AI-Crawler Blocking Index. CC BY 4.0.
998,497 domains scanned · 629,565 (63.1%) serve a robots.txt.
9.33%… See the full description on the dataset page:
https://huggingface.co/datasets/crawlora-net/ai-crawler-blocking-index.
