The kl3m-004-char-16k-casedcase-sensitive tokenizer is a domain-specific character-based tokenizer trained
on a stratified sample of nearly 2M documents across general, legal, and financial domains from the kl3m-data project,
including American English, British English, Spanish, German, French, Italian, and other common EU languages.
This tokenizer uses the standard Byte-Pair Encoding (BPE) tokenizer from tokenizers/transformers, but modifies the
training process to restrict the vocabulary to tokens that are at most 3 characters long. Models trained with this tokenizer
should be able to handle a number of use cases that are otherwise difficult to handle with standard tokenizers, such as
low-resource spell-checking, OCR correction, whitespace normalization, and other tasks that require a high degree of character-level
granularity.
Model Details
Summary
Vocabulary: 16,384 tokens
Tokenizer type: BPE with 1-4 character tokens
Special token support: Both causal and masked language modeling
Language(s) (NLP): Primarily English, Spanish, German, French, with a small percentage of other EU languages.
Special Tokens for both Embedding and Generative Models
For both training and inference efficiency, we intended this tokenizer vocabulary to be
usable for both embedding and generative models. As such, we included special tokens
suitable for both causal and masked language modeling tasks.
<|start|>: 0
<|end|>: 1
<|pad|>: 2
<|unk|>: 3
<|sep|>: 4
<|cls|>: 5
<|mask|>: 6
We also added a number of chat and instruction tokens that were not included in kl3m-001-32k, including:
<|system|>: 7
</|system|>: 8
<|user|>: 9
</|user|>: 10
<|instruction|>: 11
</|instruction|>: 12
These tokens are identical to those used in the kl3m-003-64k tokenizer.
Replication
The entire data collection and preprocesing pipeline is being made available, along with
training data, as part of the ALEA InstituteKL3M project.
This tokenizer is intended to be used for English, Spanish, German, or French language tasks where
character-level details are important, such as OCR correction, spell-checking, or tasks where word boundaries
are not well-defined.
For a standard BPE "word" tokenizer with a larger vocabulary size, consider using the kl3m-004-128k-cased or
kl3m-004-128k-uncased tokenizers.
Recommendations
The kl3m-004-char-16k-cased tokenizer may be particularly useful when character-level details are important but
resource constraints are not as severe. For smaller vocabularies with better resource efficiency, consider using the
kl3m-004-char-4k-cased or kl3m-004-char-8k-cased tokenizers.
How to Get Started with the Model
Use the code below to get started with the model.
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained('alea-institute/kl3m-004-char-16k-cased')
