You can deploy this model and start earning money today!

Nj-1111/
Copernicus-Tokenizer

Views

No views yet

Copernicus Tokenizer

Domain-general BPE tokenizer trained from scratch on 3.96 million documents spanning natural language, code, mathematics, and scientific text.

Parameter	Value
Algorithm	Byte-Pair Encoding (BPE)
Vocabulary size	32,685
Merges	32,493
Byte encoding	GPT-2 byte-level (256-char alphabet)
Min frequency	3

Quick start

python

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Nj-1111/Copernicus-Tokenizer")

ids = tokenizer("Hello, world!")
print(ids)

Use in a training loop

python

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained("Nj-1111/Copernicus-Tokenizer")

inputs = tokenizer(
    ["Hello world", "def foo(): pass"],
    truncation=True,
    max_length=2048,
    padding="max_length",
    return_tensors="pt",
)

Special tokens

Token	Role
`<\|endoftext\|>`	BOS / EOS
`<\|unk\|>`	Unknown
`<\|pad\|>`	Padding
`<think>` / `</think>`	Chain-of-thought delimiters
`<\|user\|>` / `<\|assistant\|>` / `<\|system\|>`	Chat roles
`<\|im_start\|>` / `<\|im_end\|>`	ChatML-style markers
`<\|tool_call\|>` / `<\|tool_result\|>`	Tool use

Training data

Domain	Source
Natural language	Wikipedia (multilingual), Common Crawl
Code	The Stack
Mathematics	MATH dataset, arXiv
Science	PubMed, S2ORC

Training code: github.com/Nj-1111/copernicus-tokenizer