Ventril Specifications
| Specification | Details | | :--- | :--- || | Total Parameters | 529 Million |
| Trainable (LoRA) Parameters | ~15 Million (Rank 64) |
| Context Window | 32,768 tokens | | Primary Task | Logical Reasoning & Step-by-Step Problem Solving | General Chatbot |
| Training Precision | bfloat16 |
| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| LoRA Alpha | 128 |
| Learning Rate | 2e-4 | | Datasets | GSM8K (Reasoning), OpenAssistant (Instruction following) |
| Special Tokens | , , Final Answer: |
Training Hyperparameters
Batch Size: 4 (per device)
Gradient Accumulation: 4
Total Training Steps: 1300+ (Combined aross training sessions)
Optimizer: AdamW
Weight Decay: 0.01
Most Recent Train Time and Checkpoint Count: Trained for 2H Checkpoint: 800
