INLP Assignment 3: Checkpoint Collection Model Card
This repository contains PyTorch checkpoints for a multi-stage NLP system built for noisy cipher text recovery:
- Task 1: character-level cipher-to-plain decryption (RNN, LSTM)
- Task 2: plain-text language modeling (BiLSTM MLM, SSM NWP)
- Task 3: correction pipeline combining Task 1 + Task 2 models
Model Summary
This is a checkpoint collection, not a single Transformers model.
Inference uses custom project code (main.py, src/task1/*, src/task2/*, src/task3/*).
Primary checkpoints currently used by configs:
checkpoints/task1/rnn_task4_complex_best.ptcheckpoints/task1/lstm_task4_complex_best.ptcheckpoints/task2/bilstm_complex_best.ptcheckpoints/task2/ssm_complex_best.pt
Associated vocab files:
checkpoints/task1/rnn_vocab.jsoncheckpoints/task1/lstm_vocab.jsoncheckpoints/task2/bilstm_vocabv2.jsoncheckpoints/task2/ssm_vocabv2.json
Additional historical checkpoints (epoch snapshots and earlier best versions) are also included under checkpoints/task1/, checkpoints/task2/, and checkpoints/task3/.
Intended Use
- Research/assignment evaluation for cipher decryption and LM-assisted correction.
- Comparative experiments across RNN/LSTM/SSM/BiLSTM variants.
Not intended for:
- Production security/cryptanalysis systems.
- General-domain text generation quality tasks.
Training Data
- Plain text:
data/plain.txt
- Cipher text variants:
data/cipher_00.txt to data/cipher_04.txt
The models are trained and evaluated within this assignment dataset setup.
Configuration (Current Complex Setup)
Task 1 RNN Decryption
- Config:
config/task1/rnn.yaml
- Architecture: embed=256, hidden=512, layers=2, dropout=0.30
- Sequence length: 50
- Training: epochs=50, batch_size=128, lr=1e-3, grad_clip=1.0
- Checkpoint:
rnn_task4_complex_best.pt
Task 1 LSTM Decryption
- Config:
config/task1/lstm.yaml
- Architecture: embed=256, hidden=512, layers=2, dropout=0.30
- Sequence length: 50
- Training: epochs=60, batch_size=128, lr=1e-3, grad_clip=1.0
- Checkpoint:
lstm_task4_complex_best.pt
Task 2 BiLSTM MLM
- Config:
config/task2/bilstm.yaml
- Architecture: embed=128, hidden=128, layers=2, dropout=0.15, max_seq_len=24
- Training: epochs=45, batch_size=48, lr=1e-4, weight_decay=1e-5
- Checkpoint:
bilstm_complex_best.pt
Task 2 SSM NWP
- Config:
config/task2/ssm.yaml
- Architecture: embed=64, d_state=32, layers=1, dropout=0.50, max_seq_len=24
- Training: epochs=50, batch_size=16, lr=6e-4, weight_decay=2e-2
- Checkpoint:
ssm_complex_best.pt
Evaluation Results
Task 1 (from outputs/task1_rnn.txt, outputs/task1_lstm.txt)
| Model | Loss | Accuracy | Perplexity | Character Accuracy | Word Accuracy | Levenshtein Distance |
|---|
| RNN | 1.495032 | 0.579503 | 4.459479 | 0.579503 | 0.390902 | 20.781357 |
| LSTM | 1.331345 | 0.631559 | 3.786133 | 0.580139 | 0.433234 | 19.153613 |
Task 2 (from outputs/task2_bilstm.txt, outputs/task2_ssm.txt)
| Model | Validation Loss | Perplexity |
|---|
| BiLSTM MLM | 5.5569 | 259.01 |
| SSM NWP | 6.0509 | 424.50 |
Task 3 (from outputs/task3_*.csv, noise levels cipher_01..cipher_04)
Average metrics:
- Decryption only: char_acc=0.4393, word_acc=0.0161, norm_edit_dist=0.5298, bleu=0.0161, rougeL=0.1904
- BiLSTM correction: char_acc=0.0814, word_acc=0.0165, norm_edit_dist=0.5731, bleu=0.0147, rougeL=0.1925
- SSM correction: char_acc=0.0801, word_acc=0.0169, norm_edit_dist=0.5594, bleu=0.0147, rougeL=0.1786
How to Use
This repo uses a custom CLI (uv + main.py) rather than the Transformers pipeline API.
Setup
Evaluate checkpoints
1uv run main.py task1_rnn --mode evaluate --config config/task1/rnn.yaml
2uv run main.py task1_lstm --mode evaluate --config config/task1/lstm.yaml
3uv run main.py task2_bilstm --mode evaluate --config config/task2/bilstm.yaml
4uv run main.py task2_ssm --mode evaluate --config config/task2/ssm.yaml
5uv run main.py task3_bilstm --mode evaluate --config config/task3/bilstm.yaml
6uv run main.py task3_ssm --mode evaluate --config config/task3/ssm.yaml
Hugging Face loading behavior in this project
The code attempts Hugging Face model download first when huggingface.repo_id and filenames are configured, then falls back to local checkpoints.
Limitations
- Domain-specific dataset; results may not generalize.
- Task 3 correction models can reduce character-level fidelity in current setup.
- Models depend on exact vocab/checkpoint pairing.
- Not packaged as a single
AutoModel for generic HF inference.
Ethical Considerations
- This work is for educational experimentation.
- No guarantee for security-critical decryption decisions.
- Human review is recommended for downstream interpretation.
Reproducibility Notes
- Use YAML configs in
config/task1/, config/task2/, config/task3/.
- Keep checkpoint filenames and vocab files aligned.
- Runtime and outputs may vary by hardware (CPU/GPU) and seed settings.
Citation
If you use this work, cite the repository and assignment context.
Contact
Owner: Kspsvln/INLP_A3
