Byte-Level BPE Tokenizer (vocab = 49,152)
A production-grade byte-level BPE tokenizer trained on a mixture of
permissively-licensed English text, code (via FIM) and math, with 256
reserved special-token slots covering ChatML / chain-of-thought /
tool-call / document-boundary primitives.
The companion pretokenized corpus that uses this exact
tokenizer.json
is available at
BlackwoodAI/blackwood-pretrain-49152.
At a glance
| Property | Value |
|---|
| Vocab size | 49,152 |
| Algorithm | Byte-level Byte-Pair Encoding |
| Pre-tokenizer | ByteLevel (add_prefix_space=False) |
| Decoder | ByteLevel |
| Post-processor | ByteLevel (trim_offsets=False) |
| Initial alphabet | Full ByteLevel alphabet (256 bytes) |
| Min merge frequency | 2 |
| Unknown token | <|unk|> (rarely fires — byte-level fallback covers all UTF-8) |
| Special tokens | 256 atomic tokens (IDs 0–255) |
| Library | tokenizers==0.21.0 |
The 256 special-token block is large by design — the first ~26 IDs cover
common roles (chat, CoT, tool-call, document/source boundaries), and the
remaining 230 slots are reserved for downstream fine-tune specialisations
so new roles can be aliased without retraining BPE merges.
Special-token catalog (IDs 0–255)
256 reserved slots, stable across releases.
Core (IDs 0–2)
| ID | Token | Purpose |
|---|
| 0 | <|endoftext|> | BOS / EOS / document boundary |
| 1 | <|pad|> | Right-padding |
| 2 | <|unk|> | Unknown (byte-level fallback handles all UTF-8) |
Chat / ChatML (IDs 3–7)
| ID | Token | Role |
|---|
| 3 | <|im_start|> | Message start |
| 4 | <|im_end|> | Message end |
| 5 | <|system|> | System role |
| 6 | <|user|> | User role |
| 7 | <|assistant|> | Assistant role |
Reasoning / CoT (IDs 8–11)
| ID | Token | Purpose |
|---|
| 8 | <think> | Open scratchpad |
| 9 | </think> | Close scratchpad |
| 10 | <answer> | Open final answer |
| 11 | </answer> | Close final answer |
Tool calling — generic (IDs 12–13)
| ID | Token | Purpose |
|---|
| 12 | <|tool_call|> | Open tool invocation |
| 13 | <|tool_result|> | Open tool result |
Document structure (IDs 14–15)
| ID | Token | Purpose |
|---|
| 14 | <|doc_sep|> | Document boundary within a packed sequence |
| 15 | <|source_sep|> | Source / domain boundary within a packed sequence |
Extended markers (IDs 16–25)
For long-horizon tool / planning workflows. Compose with the generic
tool-call wrappers.
| ID | Token | Purpose |
|---|
| 16 | <|fn_name|> | Function name |
| 17 | <|fn_args|> | Function arguments |
| 18 | <|fn_response|> | Function return value |
| 19 | <|tool_error|> | Tool error signal |
| 20 | <|plan_step|> | Multi-step planning marker |
| 21 | <|reflect|> | Self-correction signal |
| 22 | <|memory|> | Persistent-memory access |
| 23 | <|search|> | Search marker |
| 24 | <|file_op|> | Filesystem operation marker |
| 25 | <|terminate|> | Clean stop |
Reserved (IDs 26–255)
<\|reserved_010\|> through <\|reserved_239\|> — 230 slots reserved
for downstream fine-tune specialisations.
Measured fertility (tok/char, real-corpus samples)
Numbers reproduced from test_report.json:
| Source | tok/char | tok/byte |
|---|
| smollm-cosmopedia | 0.186 | 0.185 |
| smollm-fineweb-edu | 0.204 | 0.204 |
| dclm-edu | 0.230 | 0.230 |
| finemath | 0.303 | 0.302 |
| proof-pile-2 | 0.297 | 0.297 |
| dolma3-mix-150B | 0.272 | 0.272 |
| olmo-mix-1124 | 0.227 | 0.226 |
| dolma3-dolmino (stack-edu FIM) | 0.237 | 0.235 |
| Domain (synthetic) | tok/char |
|---|
| English prose | 0.224 |
| Python code | 0.335 |
| Rust code | 0.392 |
| LaTeX math | 0.399 |
| JSON | 0.336 |
| Chinese | 0.821 |
| Arabic | 0.944 |
| Emoji-heavy | 0.614 |
Notes:
- English prose at 0.22 is competitive with Llama-3-class tokenizers.
- Non-Latin scripts (CJK, Arabic) are not optimised — this is an
English / code / math tokenizer.
- Any UTF-8 input is covered losslessly thanks to ByteLevel.
Training corpus
Trained on 15 GB of UTF-8 text streamed across 7 sources (see
stats.json for exact byte / doc counts):
| Source | Share |
|---|
| smollm-corpus (cosmopedia-v2 + fineweb-edu-dedup) | 30 % |
| dclm-edu | 20 % |
| dolma3-dolmino-100B | 15 % |
| finemath | 15 % |
| proof-pile-2 (arxiv + algebraic-stack + open-web-math) | 10 % |
| olmo-mix-1124 | 5 % |
| dolma3-mix-150B | 5 % |
All sources hold permissive licenses (ODC-BY / CC-BY-4.0 / various
permissive).
Round-trip + correctness tests
Released with a 6-test verification suite:
- Vocab size == 49,152
- All 256 special tokens at correct sequential IDs (0–255)
- Special tokens encode as single atomic tokens
- Round-trip
decode(encode(x)) == x exact on synthetic samples
- Round-trip exact on real corpus samples across every source
- ChatML / CoT / tool-call template strings parse as expected token streams
All six pass; details in test_report.json.
Usage
Raw tokenizers library
1from tokenizers import Tokenizer
2from huggingface_hub import hf_hub_download
3
4tok_path = hf_hub_download(
5 repo_id="BlackwoodAI/blackwood-0.7b-bpe49152-tokenizer",
6 filename="tokenizer.json",
7)
8tok = Tokenizer.from_file(tok_path)
9ids = tok.encode("<|im_start|><|user|>hi<|im_end|>").ids
10print([tok.id_to_token(i) for i in ids])
11# ['<|im_start|>', '<|user|>', 'hi', '<|im_end|>']
transformers
1from transformers import AutoTokenizer
2tok = AutoTokenizer.from_pretrained(
3 "BlackwoodAI/blackwood-0.7b-bpe49152-tokenizer",
4 use_fast=True,
5)
ChatML inference prompt
1prompt = (
2 "<|im_start|><|system|>You are a helpful coding assistant.<|im_end|>\n"
3 "<|im_start|><|user|>Write fizzbuzz in Python.<|im_end|>\n"
4 "<|im_start|><|assistant|>"
5)
6input_ids = tok.encode(prompt).ids
CoT template
1prompt = (
2 "<|im_start|><|user|>What's 17 × 23?<|im_end|>\n"
3 "<|im_start|><|assistant|>"
4 "<think>17 × 20 = 340, 17 × 3 = 51, total = 391.</think>"
5 "<answer>391</answer><|im_end|>"
6)
Tool-call template
1prompt = (
2 "<|im_start|><|user|>What's the weather in NYC?<|im_end|>\n"
3 "<|im_start|><|assistant|>"
4 "<|tool_call|><|fn_name|>get_weather"
5 "<|fn_args|>{\"city\":\"NYC\"}"
6 "<|tool_result|><|fn_response|>{\"temp_f\":72,\"condition\":\"clear\"}"
7 "<answer>It's 72°F and clear in NYC.</answer>"
8 "<|terminate|><|im_end|>"
9)
Notes on byte-level encoding
add_prefix_space=False: encoding "hello" differs from " hello" by
the first token. Be consistent at training and inference.- No normaliser is applied. Casing, accents, control bytes and Unicode
variants are all preserved. Layer NFKC normalisation before encoding
if your downstream needs it.
- No subword regularisation (BPE-Dropout, SentencePiece unigram
sampling) is applied at the tokenizer level. Add it downstream if your
use case needs it.
License
Tokenizer artifact: Apache-2.0.
Training corpora retain their upstream licenses (ODC-BY / CC-BY-4.0 /
various).
