FinSLM-30M-LR1e-4-GA32

Model

• 6 Transformer layers
• 6 attention heads
• 384-dimensional embeddings
• 128-token context length
• GPT-2 BPE tokenizer (50,257 vocabulary)

Training

• Learning rate: 1e-4
• Gradient accumulation: 32
• Maximum iterations: 60,000
• Hardware: NVIDIA T4 (16 GB)
• Precision: float16
• Best validation loss: 3.997
• Best perplexity: 54.5

Results