osamamumtaz01/

ai-crawler-user-agents

Machine-readable list of all 28 known AI crawler and agent user-agent strings — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, Applebot-Extended, and more — with each bot's operator, purpose, and observed robots.txt compliance.

	Fields




	

Field Description

userAgent Token to match in robots.txt / server logs (e.g. GPTBot)

operator Company running the crawler

purpose training, search, or both… See the full description on the dataset page: https://huggingface.co/datasets/osamamumtaz01/ai-crawler-user-agents.