You can deploy this model and start earning money today!

issdandavis/
scbe-tokenizer-sacred-tongues-qwen25-coder-7b

Name: scbe-tokenizer-sacred-tongues-qwen25-coder-7b
Availability: InStock

Views

No views yet

Status: experimental. Research artifact, not a production candidate. Canonical model: scbe-coding-agent-vtc-qwen15-v1-gguf.

Sacred Tongues tokenizer - Qwen2.5-Coder-7B

This is a tokenizer only. There are no model weights in this repo. Load it with AutoTokenizer, not AutoModel.

Numbers below are from sacred_tongues_extension_report.json in this repo.

Twelve tongue names were added to the vocabulary, lowercase and capitalised: kor'aelin avali runethic cassisivadan umbroth draumric.

Vocab size: 151,665 -> 151,677 (12 requested, 12 added)
Effect: every one of the twelve went from 2-4 pieces to 1 piece

token	pieces before	pieces after
kor'aelin	4	1
cassisivadan	4	1
runethic	3	1
umbroth	3	1
draumric	3	1
avali	2	1

That is the whole claim: the six tongue names are now atomic instead of being split across several sub-word pieces. No downstream model quality is claimed.

python

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("issdandavis/scbe-tokenizer-sacred-tongues-qwen25-coder-7b")