ANLP M26 Assignment 1 — Custom Transformers and BLT
This repository contains five from-scratch PyTorch Transformer configurations
for encrypted-binary-sequence to plaintext reconstruction.
C1-C4 use learned custom BPE tokenizers implemented without a tokenizer
library. C5 uses packed ciphertext bytes and dynamically sized patches derived
from next-byte entropy.
Configurations
| Configuration | Position | Attention | Normalization | Representation |
|---|
| C1 | Sinusoidal | MHA | LayerNorm | Custom BPE |
| C2 | RoPE | MHA | LayerNorm | Custom BPE |
| C3 | Sinusoidal | GQA, 8 query/2 KV heads | LayerNorm | Custom BPE |
| C4 | Sinusoidal | MHA | RMSNorm | Custom BPE |
| C5 | Sinusoidal | MHA | LayerNorm | Packed bytes + dynamic entropy patches |
C2-C4 each change one architectural component relative to C1. C5 replaces
the subword interface with a simplified token-free BLT pipeline.
Complete test-split results
All results use greedy decoding on the same 1,389-example test split.
| Config | Bit accuracy | Sequence accuracy | Mean Levenshtein | BLEU | ROUGE-L |
|---|
| C1 | 0.874859 | 0.293017 | 3.392369 | 90.369524 | 0.955734 |
| C2 | 0.949647 | 0.598272 | 2.472282 | 95.937859 | 0.981416 |
| C3 | 0.814852 | 0.120230 | 8.535637 | 80.490798 | 0.910088 |
| C4 | 0.854665 | 0.226062 | 4.676746 | 87.776615 | 0.944754 |
| C5 | 0.979374 | 0.231821 | 6.260619 | N/A | N/A |
BLEU and ROUGE are reported only for tokenized C1-C4. Cross-entropy loss is
not directly compared between subword and byte prediction units.
Recorded full-run efficiency
| Config | Parameters | Mean epoch (s) | Maximum recorded memory (GiB) |
|---|
| C1 | 8,530,896 | 54.885 | 1.376 |
| C2 | 8,530,896 | 60.448 | 1.371 |
| C3 | 7,346,640 | 54.151 | 1.356 |
| C4 | 8,525,264 | 50.055 | 1.292 |
| C5 | 7,709,699 | 89.552 | 1.306 |
Corrected C5 entropy patching
- Eight cipher bits are packed into one byte value from 0 to 255.
- Entropy estimator: order-2 empirical Markov model.
- Fitting scope: training ciphertext only.
- Global entropy threshold: 4.308366 bits.
- Training average patch length: 3.944 bytes.
- Training patch-length range: 1-16.
- Entropy-boundary fraction: 0.249966.
- Safety-cap-boundary fraction: 0.00001124.
- Parameters: 7,709,699.
- Best checkpoint epoch: 29.
Controlled C5 relative to C1
| Measure | C5 delta |
|---|
| Mean complete epoch | +63.16% |
| Mean training step | +106.62% |
| Training-step peak allocated memory | +24.75% |
| Parameters | -9.63% |
Checkpoints and artifacts
These are custom educational PyTorch architectures and are not compatible
with Hugging Face AutoModel.
Final W&B runs
Earlier prebuilt-tokenizer and fixed-patch runs are retained only as a
superseded audit trail.
