This repository contains the official weights and configuration for the GODELEV TOK-4K tokenizer, a highly optimized, custom Byte-Pair Encoding (BPE) tokenizer engineered for hybrid text, mathematical reasoning, and structured chain-of-thought processing.
Designed with a compact, ultra-dense vocabulary layout, this tokenizer enforces strict token boundaries to eliminate common serialization errors seen in flagship model architectures.
The selection of a 4,096 vocabulary size represents an explicit architectural trade-off engineered for specific sequence modeling environments. While standard downstream Large Language Models (LLMs) often scale vocabularies to 32,000, 50,000, or over 100,000 tokens to maximize sequence compression, a 4,096 vocabulary size drastically minimizes the embedding and final language model head layer parameters. This configuration ensures that small-scale or highly specialized model architectures allocate their capacity entirely to hidden layer representations and deep sequence dependencies rather than wide sparse matrix lookup tables.
Structural Integrity via tokenizer_config.json
The underlying layout serialization in tokenizer_config.json enforces several core programmatic constraints to ensure robust text-to-ID and ID-to-text cycles:
Explicit Token Protection: All control and special tokens are registered with the parameter "normalized": false. This prevents the Unicode Normalization Form KC (NFKC) from altering strings that resemble special structures. For instance, even if a text input contains unusual white space or canonical variants, it will never break the underlying <|think|> or <|user|> strings into sub-word byte patterns.
Deterministic Control ID Mapping: Special system tokens occupy indices 0 through 16. This continuous block guarantees that control characters reside safely away from raw byte representations and dynamically learned BPE merge structures, simplifying mask calculation and vocabulary slicing inside transformer custom heads.
Disabled Space Cleanup: The flag "clean_up_tokenization_spaces": false is explicitly set. Standard tokenizers frequently strip continuous spaces, leading to corruption in code syntax formatting or aligned tabular mathematical expressions. Preserving whitespaces directly supports structured chain-of-thought processing.
Custom Pre-Tokenizer Pipeline
To prevent structural degradation across text transitions, the tokenizer executes a strict sequence pipeline:
Isolated Splitting: Uses a specialized regex engine pattern identical to advanced modern architectures:
This splits inputs cleanly into explicit word stems, contractions, sequential punctuation groupings, numerical digits, and newline flags before the BPE algorithms calculate merge metrics.
Byte-Level Fallback: Maps the split substrings into byte fragments. This bypasses the typical constraints of closed-vocabulary character maps, completely mitigating Out-of-Vocabulary (OOV) tokens. Any unknown character sequences or arbitrary binary inputs are safely broken down into basic native bytes without crashing runtime workflows.
Dataset Training Composition
The tokenizer was trained on a rich, multi-domain corpus containing 1,113,659 unique documents comprising 4,186,212,111 total characters (approximately 4.18 GB of dense training data). The corpus combines massive scale conversational reasoning with high-fidelity exact mathematical training instances:
GODELEV/BetterDataset-2M
Data Volume: 500,000 items extracted natively via stream buffers.
Targeted Fields: ["text"]
Purpose: Provides rich linguistic variety, comprehensive semantic vocabulary patterns, conversational grammar frameworks, and broad real-world knowledge distributions.
GODELEV/Arithmetic
Data Volume: 96,103 static rows (100% extraction of the full curriculum dataset).
Targeted Fields: ["text"]
Purpose: Instills fine-grained algorithmic sequences, number operations, math equations, and continuous text patterns optimized for logical expression evaluations without experiencing numeric digit splitting artifacts.
Special Token Registry
Special and additional structural tokens are embedded at the lowest vocabulary indexes to preserve strict interaction parameters:
ID
Token
Structural Context
0
<pad>
Padding Token
1
<unk>
Unknown Token
2
<bos>
Beginning of Sequence
3
<eos>
End of Sequence
4
<mask>
Masking Operator Token
5
<unk_text>
Unknown Text Frame Holder
6
<sot>
Start of Text Block
7
<eot>
End of Text Block
8
<system>
System Persona Boundary Initialization
9
<user>
User Context Input Initialization
10
<assistant>
Assistant Response Block Initialization
11
<math>
Specialized Math Expression Opener
12
<expr>
Evaluation Expression Variable Boundary
13
<answer>
Standard Output Result Target Block
14
<code>
Coding Language Sequence block parser
15
<think>
Internal Thought Chain Logic Process Initiator
16
<end_of_think>
Termination Indicator for Thought Chain Logic
Usage and Implementation
To initialize this tokenizer from the Hugging Face Hub, ensure you have the transformers and tokenizers libraries installed, then instantiate via AutoTokenizer:
python
1from transformers import AutoTokenizer
23# Load the production tokenizer4tokenizer = AutoTokenizer.from_pretrained("GODELEV/TOK-4K")56# Example sample showcasing structured chain-of-thought integration7sample_phrase ="<|user|>\nExecute calculation <|think|> Processing <|end_of_think|> <|math|>2+2<|answer|>4"89# Encode to sequence IDs10encoded_ids = tokenizer.encode(sample_phrase)11print(f"Encoded IDs: {encoded_ids}")1213# Round-trip decoding14decoded_string = tokenizer.decode(encoded_ids)15print(f"Decoded Output: {decoded_string}")
Maintenance and Preservation
This tokenizer artifact ensures zero index collisions for downstream execution tasks. When moving weights into active model pipelines, do not call native text normalization routines over the string payloads manually, as this might bypass the internal safety parameters mapped into the AddedToken protections.
