Neural Cipher Decryption & Error Rectification

Model Architectures (Implemented From Scratch)

1. Recurrent Neural Networks (RNN & LSTM)

• RNN (Vanishing Gradient Baseline):
  • Architecture: A classic Elman RNN where the hidden state $h_t$ is updated via $tanh(W_{ih} x_t + b_{ih} + W_{hh} h_{t-1} + b_{hh})$.
  • Validity: Served as a baseline to demonstrate the limitations of simple recurrence on long character sequences.
  • Limitations: High Levenshtein distance due to the vanishing gradient problem on sequences longer than 50 characters.
• Long Short-Term Memory (LSTM):
  • Architecture: A full 4-gate architecture (Forget, Input, Output, and Cell gates).
  • Modifications: We applied Orthogonal Initialization to the recurrent weights and Layer Normalization to stabilize training across 20+ epochs.
  • Validity: Achieved 93.45% character accuracy, significantly outperforming the RNN in maintaining the "skeleton" of the sentence.
  • Limitations: Sequential nature makes it slower than parallel architectures during inference.

2. Structured State Space Model (SSM)

• Architecture: Based on the S4/Discrete SSM framework. It models sequences as a linear recurrence: $h_t = Ah_{t-1} + Bx_t$.
• Implementation: We converted the discrete recurrence into a Causal Convolution using Fast Fourier Transforms (FFT). This allows for $O(L \log L)$ training speed.
• Modifications: Parameterized the $A$ matrix in log-space ($A_{log}$) to ensure stability and causality.
• Validity: Effectively learned Next Word Prediction (NWP) patterns on the plain.txt corpus.

3. Bidirectional LSTM (Bi-LSTM)

• Architecture: Two independent LSTM streams processing the sequence in forward and backward directions.
• Implementation: Specifically tuned for Masked Language Modeling (MLM).
• Modifications: Integrated a projection head to map the concatenated hidden states to the full 19k+ word vocabulary.
• Validity: Achieved the best Perplexity (~182), proving that bidirectional context is superior for error correction.

🛠 Usage & Commands

1. Task 1: Character-Level Decryption

2. Task 2: Language Modeling

3. Task 3: Integrated Pipeline (Rectification)

Final Results Matrix

Analysis Summary

• Task 1 Conclusion: LSTMs are far more effective at preserving phonetic and structural integrity than RNNs for decryption.
• Task 2 Conclusion: Bidirectional context (Bi-LSTM) results in a 4.6x improvement in perplexity over unidirectional models (SSM).
• Task 3 Conclusion: Our rectification logic improves ROUGE-L by 13.2%, proving that neural language models can effectively "snap" corrupted characters back into valid English vocabulary.

Advanced Features

• Hardware Acceleration: Full support for torch.amp (Mixed Precision) and torch.compile (JIT).
• Stability: Gradient clipping and custom initialization across all models.
• Metrics: Implemented bit-parallel evaluation algorithms for LCS and Levenshtein.

• All result files are generated in outputs/results/.
• Checkpoints are managed via src/utils/checkpoints.py.
• Final Model Repository: [Link to Hugging Face]
• WandB Training Logs:
  • Task 1 (RNN): WandB Session
  • Task 1 (LSTM): WandB Session
  • Task 2 (SSM): WandB Session
  • Task 2 (Bi-LSTM): WandB Session