Metric	Value	Status
Vocabulary Size	`16,384`	✅ Standard
Round-Trip Integrity	`100% Lossless`	✅ Verified
Unknown Token (`<unk>`) Rate	`0.00%`	✅ Verified
Punctuation Isolation	`Clean`	✅ Isolated
Ge'ez Wordspace (`፡`) Handling	`Atomic Token`	✅ Verified
Base Architecture	`SentencePiece`	✅ Native

Quickstart Usage

python

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("BeitTigreAI/tigre-spm-tokenizer")

text = "ሰላም፡ እሊ ናይ ሂጋ ትግሬ ክታበት ቱ።"

# Encode to tokens & IDs
tokens = tokenizer.tokenize(text)
input_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("IDs   :", input_ids)

# Lossless Decoding
decoded_text = tokenizer.decode(input_ids, clean_up_tokenization_spaces=False)
print("Decoded:", decoded_text)
assert text == decoded_text

Repository Structure

tokenizer.model: Native SentencePiece binary model file.
tokenizer.json: Serialized fast tokenizer representation for Python/Rust environments.
tokenizer_config.json: Metadata and special token configuration mapping.
special_tokens_map.json: Explicit <pad>, <s>, </s>, and <unk> assignments.