The Txa 1 Tokenizer is a highly efficient, production-ready tokenizer engineered for the Txa 1 (4B MoE) model family. Built upon the battle-tested Mistral v1 foundation, it has been fine-tuned to balance high compression rates with extreme processing speed on H100/H200 hardware.

This tokenizer natively supports ChatML formatting, making it instantly compatible with modern inference engines like vLLM, Ollama, and LM Studio.

Developed by Rx, Founder & CEO of Axtrio AI.

📊 Benchmark Arena

We pitted the Txa 1 Tokenizer against industry heavyweights in our Tokenizer Arena.

1. Speed Analysis (Throughput)

Higher is better. Measures raw tokenization speed on H100 hardware.

2. Compression Efficiency

Lower is better. Measures how many tokens are needed to represent complex Code & English.

3. Vocabulary Architecture

Comparison of dictionary sizes. Txa 1 stays lean (32k) to maximize VRAM efficiency for the 4B MoE architecture.

🔧 Technical Specifications

Feature	Specification
Base Architecture	Byte-Pair Encoding (Mistral v1 Foundation)
Vocabulary Size	32,003 Tokens (Efficient & Lean)
Added Special Tokens	`<
Optimization	Code & Logic Compression
Compatibility	Fully Compatible with `LlamaTokenizerFast`

💻 Usage

Quick Start

python

from transformers import AutoTokenizer

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("AxtrioAI/Txa1-4B-Tokenizer")

# Test ChatML Format
chat = [
    {"role": "user", "content": "Hello Txa, can you help me debug python?"},
    {"role": "assistant", "content": "Certainly! Please paste your code below."}
]

# Apply template
formatted_prompt = tokenizer.apply_chat_template(chat, tokenize=False)
print(formatted_prompt)

axtrioai/Txa1-4B-Tokenizer

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Txa 1 Tokenizer: The Foundation of Axtrio AI ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

⚡ Overview