llm-tokens-atlas is an open, reproducible benchmark of LLM tokenization across
5 providers (Anthropic claude-opus-4-7, Google gemini-2.5-pro, OpenAI
gpt-4o, Mistral mistral-large-latest, Cohere command-r-08-2024) across
5 prompt formats (Markdown, XML, JSON, YAML, Plain text), evaluated on
12,500 real-world prompt requests (n=2,500 per provider, 500 unique prompts × 5
formats × 5 providers). For each (prompt, provider, model, format) cell we… See the full description on the dataset page:
https://huggingface.co/datasets/faraa2m/llm-tokens-atlas.
