spyysalo/

harmony-latin-plus

Version of the gpt-oss tokenizer (o200k_harmony) filtered to exclude tokens with characters that are not in the Latin, Cyrillic, Greek, or Georgian scripts (or unicode Common/Unknown) using https://github.com/spyysalo/tokenizer-filter/ as follows: