TokSuite–Llama-3.2 is part of TokSuite, a suite of language models designed to study the impact of tokenizer choice on language model behavior under controlled conditions.

This model uses the Llama-3.2 tokenizer and is otherwise identical to the other TokSuite-Llama models (toksuite/meta-llama-Llama-3.2-1B and toksuite/meta-llama-Llama-3.2-1B-seed_777_model_seed_222) in architecture, training data, training budget and only differ in the initialization, denoted by the model_seed.

Tokenizer

Tokenizer: Llama-3.2
Tokenization method: BPE
Vocabulary size: 128,256
Out-of-vocabulary handling: Byte-fallback
Language coverage: Multilingual
Pretokenization source: GPT-4

Processing details:

Numbers: Group by 3
Contractions: GPT-4
Unicode normalization: None
Whitespace / boundary markers: Learned
Zerowidth chars: Token

Why Llama-3.2?

Llama-3.2 was included in TokSuite to represent a multilingual BPE tokenizer with modern GPT-4–style pretokenization and a medium-to-large vocabulary size. As described in the tokenizer selection rationale of the TokSuite paper, Llama-3.2 exemplifies a contemporary tokenizer design that combines subword segmentation with preprocessing conventions used in recent large language models.

Including Llama-3.2 enables TokSuite to study tokenizer behavior in settings where:

BPE segmentation is paired with GPT-4–style pretokenization,
vocabulary size is substantially larger than early English-centric tokenizers,
and multilingual text is handled through a single shared tokenizer.

This makes Llama-3.2 a representative example of modern multilingual BPE tokenization.

Model Architecture

Architecture: Decoder-only Transformer (Lingua's Llama-3.2-1B configuration)
Non-embedding parameters: ~1B
Context length: 4096 tokens
Framework: Meta Lingua
Initialization: Shared super-vocabulary initialization across TokSuite models

The architecture and training setup are identical across all TokSuite models; only the tokenizer differs.

Training Data

The model was trained on a multilingual corpus totaling approximately 100B tokens, composed of:

English: 40B tokens from FineWeb-Edu
Multilingual: 60B tokens evenly distributed across:
- Chinese (ZH)
- Turkish (TR)
- Italian (IT)
- Farsi (FA)

You can find the pretraining dataset here: toksuite/toksuite_pretraining_data

All TokSuite models are trained using a fixed token budget, following common practice in large-scale language model training.

Training Procedure

Training steps: 100,000
Sequence length: 4096
Batch size: 256 sequences
Optimizer: AdamW
Peak learning rate: 1e-3
Learning rate schedule: Cosine decay with 2,000 warm-up steps
Weight decay: 0.1

TokSuite Robustness Benchmark

TokSuite–Llama-3.2 variants are evaluated on the TokSuite robustness benchmark, which measures sensitivity to real-world text perturbations, including:

orthographic and spelling variations,
diacritics presence and absence,
keyboard and input-method noise,
Unicode formatting and homoglyphs,
OCR and spacing artifacts,
LaTeX and STEM-style formatting.

Tokenization Robustness under Multilingual Text Perturbations
Values represent relative performance drop, computed as (Acc_clean − Acc_perturbed) / Acc_clean, where lower values indicate greater robustness.

Perturbation types include:

Input: non-native keyboard input and romanization
Diacr.: optional diacritics
Orth.& Gram.: orthographic and grammatical errors
Morph: morphological variations including derivations, inflections, and contractions
Noise: homoglyph substitutions, OCR artifacts, typos, and spacing errors
LaTeX: LaTeX-style mathematical formatting
STEM: scientific diagrams and notational conventions
Unic.: Unicode styling characters

NEN denotes non-English inputs and EN denotes English inputs. The Avg column reports the average relative performance drop across all perturbation categories.

Model	Input (NEN)	Diacr. (NEN)	Orth. & Gram. (EN)	Orth. & Gram. (NEN)	Morph (EN)	Morph (NEN)	Noise (EN)	Noise (NEN)	LaTeX (EN)	STEM (EN)	Unic. (EN)	Avg ↓
toksuite/meta-llama-Llama-3.2-1B-seed_777_model_seed_888	0.31	0.44	0.12	0.15	0.25	0.13	0.09	0.24	0.09	0.26	0.58	0.24
toksuite/meta-llama-Llama-3.2-1B	0.31	0.56	0.11	0.11	0.25	0.10	0.09	0.26	0.13	0.29	0.60	0.26
toksuite/meta-llama-Llama-3.2-1B-seed_777_model_seed_222	0.29	0.54	0.09	0.12	0.20	0.13	0.14	0.29	0.19	0.37	0.59	0.27
Avg	0.30	0.51	0.11	0.12	0.24	0.12	0.10	0.26	0.13	0.31	0.59	0.25

Intended Use

This model is intended for:

research on tokenization and robustness,
multilingual NLP analysis,
controlled ablation studies,
benchmarking tokenizer behavior under noise.

It is not instruction-tuned, aligned, or optimized for deployment.

Limitations

Trained on a limited set of five languages.
Not optimized for instruction following or dialogue.
Fixed token budget constrains exposure to raw text depending on tokenization efficiency.
Intended strictly for research purposes.

Ethical Considerations

TokSuite models are released to support scientific investigation of tokenization effects.
They may reflect biases present in large-scale web data and should not be used in high-stakes or user-facing applications without additional safeguards.

Citation

If you use this model, please cite:

bibtex

@article{toksuite2025,
  title={TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior},
  author={Altıntaş, Gül Sena and Ehghaghi, Malikeh and Lester, Brian and Liu, Fengyuan and Zhao, Wanru and Ciccone, Marco and Raffel, Colin},
  year={2025},
  arxiv={https://arxiv.org/abs/2512.20757},
}

toksuite/meta-llama-Llama-3.2-1B-seed_777_model_seed_888

TokSuite – Llama-3.2 - Model Initialization seed=888

Model Summary