Autoregressive generation not yet trained (epochs 1-30: teacher forcing only)
Sliding window implementation has bugs (being fixed during training)
204 languages coverage is experimental - final performance TBD
🎯 Project Purpose:
This tokenizer is designed to enable more efficient LLM inference by separating language processing from reasoning:
Byte-level processing: No vocabulary needed, truly universal
Architecture separation:
Language Processing Model (this tokenizer): Handles linguistic complexity
Inference Model (LLM): Focuses purely on reasoning with compressed vectors
Goal: The inference model receives only semantic vectors, allowing it to concentrate entirely on reasoning without language-specific overhead
This approach aims to make LLMs more efficient by offloading all linguistic processing to a specialized encoder/decoder.
🎯 Experiment Goal: Testing whether 48:1 compression is achievable across 204 languages with a single consumer GPU
Recommended: If you want higher completeness, please wait for training completion
🚀 Key Innovation
Unlike traditional tokenizers (BPE, WordPiece) that split text using fixed rules, B2NL learns to identify semantic units within byte sequences through neural networks:
Traditional BPE: "안녕하세요" → "안", "녕", "하", "세", "요" (5 tokens, word fragments)
B2NL: "안녕하세요" → [emb1, emb2, emb3] (3 embeddings, meaning preserved)
