Views
No views yet
moonshine_tiny_zh_decode.bin contains the decode-only tokenizer data in a
compact binary format.LICENSE.txt and
NOTICE.1Header
2special_bitset[(vocab_size + 7) / 8]
3byte_token_ids[256] uint32
4piece_offsets[vocab_size] uint32
5piece_lengths[vocab_size] uint32
6piece_blob UTF-8 bytespiece_offsets[id] and piece_lengths[id] point into piece_blob.1char[4] magic "TKD1"
2uint32 version 1
3uint32 header_size 40
4uint32 vocab_size 32768
5uint32 bos_id 1
6uint32 eos_id 2
7uint32 pad_id 2
8uint32 unk_id 0
9uint32 special_bytes
10uint32 blob_sizespecial_bitset uses one bit per token id to indicate whether the token is a special token.
byte_token_ids[b] maps byte b (0x00–0xFF) to the corresponding
token ID used by the tokenizer's ByteFallback mechanism. A decoder should build the inverse
mapping once during initialization. 0xffffffff means that no mapping exists. 0xffffffff means
that no mapping exists.<0xXX> tokens to their raw byte values.