V19 is a fully interpretable Chinese language understanding system with 4.7 million parameters. It reads a Chinese sentence as a sequence of characters, builds word-level representations through a frozen char-to-word encoder (P1), routes information across sentences (P7), and decodes back to word sequences (P6) — all while maintaining traceable, auditable intermediate states.
Unlike transformer-based LLMs, every internal decision in V19 can be inspected:
P1: Which words each character pair maps to
P7: How words route across sentences (32-head attention with per-head gating)
Explore+Meta Gate: Which decoding dimensions are active and why
P6: How each output word is decoded from the sentence vector + position embedding
Intended Use
Chinese text correction (primary task)
Interpretability research: study how linguistic attributes compose without black boxes
Education: demonstrate NLP concepts with fully transparent architecture
Low-resource deployment: 141MB GPU, runs on CPU at 71 sent/s
Position embedding provides unique starting point per head — naturally prevents repetition collapse
No rep_pen, no residual subtraction, no detach needed
Training
Config
Value
Optimizer
Adam (P6 lr=0.003, P7 lr=0.0045, Gate lr=0.006)
Loss
1.0 - mean(cosine_similarity(pred, true))
Epochs
1000
Batch
Full dataset per epoch (41,909 pairs)
GPU
RTX 5070
Memory
~300MB (training), 141MB (inference)
Evaluation
V18 (875K params, 16 heads)
Metric
Score
Word Accuracy
92.4%
Exact Match
76.3%
Rouge-L F1
93.2
Per-word Cosine Mean
0.96
Inference
14ms/sentence
V19 (4.7M params, 128 heads) — in training
Metric
Epoch 1
Target
Word Accuracy
43.5%
>95%
Per-word Cosine
0.73
>0.97
Key Innovations
Position Embedding V6 (Anti-collapse)
After 5 failed approaches to prevent the P6 decoder from outputting the same word repeatedly (rep_pen, residual extraction, weight transpose inversion, orthogonal init, cos_loss margin), the final solution was the simplest:
python
1for i inrange(max_words):2 hi = h + self.pos_embed[i]# unique starting point per head3 w = self.extract[i](hi)
No rep_pen. No residuals. No detach. Just position diversity.
Explore→Meta Gate
Instead of directly minimizing loss (which causes gates to converge to all-open or all-closed), the gate is trained indirectly:
Loss flows into Explore network → produces 256D signal
Meta applies learned bias + sigmoid → 256D gate
Gate modulates P6 encoder → affects word predictions
Gate quality is measured by per-head prediction accuracy, not total loss
This prevents the "gate symmetry lock" (all dims identical, std=0.0001) that plagued early versions.
