Views
No views yet
| Architecture | Encoder-Decoder Transformer |
| Parameters | ~13.5M |
| Vocab size | 16,000 (BPE SentencePiece) |
| d_model | 256 |
| Layers | 4 encoder / 4 decoder |
| Attention heads | 4 |
| Positional encoding | RoPE |
| Normalization | RMSNorm |
| Feed-forward | SwiGLU |
| Copy mechanism | Pointer-Generator gate |
| Fine-tune data | PAWS + MRPC + QQP (~180K pairs) |
| Pretrain data | ~250 Project Gutenberg books (18M+ tokens) |
| Best fine-tune val loss | 1.817 (epoch 20, pretrained init) |
llm/
├── model/ — ModelConfig, MultiHeadAttention (RoPE), Encoder/DecoderBlock, ParaphraseModel
├── tokenizer/ — SentencePiece BPE trainer and wrapper
├── data/ — download, clean, dedup, filter scripts + cleaned JSONL + book CSVs
├── training/ — fine-tune loop, pretrain loop, dataset, loss, EMA
├── inference/ — beam search with KV cache + semantic reranking
├── eval/ — BLEU, ROUGE-L, BERTScore evaluation
├── checkpoints/ — saved model weights (.pt)
├── run.sh — end-to-end pipeline script
├── export_onnx.py — export to ONNX
└── upload_to_hf.py — push to HuggingFace Hub1python3 -m venv .venv
2source .venv/bin/activate
3pip install torch sentencepiece transformers datasets sentence-transformers sacrebleu rouge-score bert-score langdetect wandb1python3 -m inference.infer \
2 --ckpt checkpoints/best.pt \
3 --tok tokenizer/tokenizer.model \
4 --text "The dog ran quickly across the yard."1python3 paraphrase_file.py \
2 --ckpt checkpoints/best.pt \
3 --tok tokenizer/tokenizer.model \
4 --input sample.txt \
5 --output output.txt1python3 -m eval.evaluate \
2 --ckpt checkpoints/best.pt \
3 --tok tokenizer/tokenizer.model \
4 --data data/clean.jsonlbash run.sh1python3 -m training.pretrain \
2 --data data/books/ \
3 --epochs 20 \
4 --batch_size 64 \
5 --grad_accum 4 \
6 --lr 5e-4 \
7 --warmup 500 \
8 --ckpt_dir checkpoints/pretrain/1python3 -m training.train \
2 --data data/clean_combined.jsonl \
3 --tok tokenizer/tokenizer.model \
4 --init_from checkpoints/pretrain/best.pt \
5 --ckpt_dir checkpoints/finetune/ \
6 --epochs 20 \
7 --lr 1e-4 \
8 --warmup 500 \
9 --label_smoothing 0.05 \
10 --patience 5 \
11 --wandb_project lexiform \
12 --wandb_run stage2-finetune1python3 -m data.download --out data/raw
2python3 -m data.dedup --inp data/raw --out data/clean.jsonl
3python3 -m tokenizer.train --data data/clean.jsonl| Phase | Description | Status |
|---|---|---|
| 2 — Pretrain | Span corruption on 18M+ book tokens | Running |
| 2 — Fine-tune | Load pretrained weights → fine-tune | Waiting |
| 3 | WordNet synonym bias + voice transform | Planned |
| 4 | Levenshtein edit-op decoder | Planned |
| 5 | FAISS kNN-LM retrieval-augmented decoding | Planned |
| 6 | PPO RL fine-tuning on composite reward | Planned |