unsloth/Llama-3.2-3B-Instruct-bnb-4bit Fine-tuned with GRIT and QLoRA (Unsloth)

🚀 Training Details

GRIT Algorithm

• K-FAC Updates: Every 100 steps (adaptive) for second-order preconditioning.
• Neural Reprojection: Every 100 steps (adaptive) for rank optimization.
• Rank Adaptation: Enabled (Threshold: 0.99, Min Rank: 4).
• Optimized LoRA Modules: ['q_proj', 'k_proj', 'v_proj', 'o_proj']

Fine-tuning Configuration

• Base Model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit
• Quantization: 4-bit (NF4) with fp16 compute.
• LoRA Rank: 16
• LoRA Alpha: 32
• Batch Size: 8 (per device)
• Gradient Accumulation: 4 (Effective batch = 32)
• Learning Rate: 2.0e-05
• Precision: fp16 mixed precision
• Sequence Length: 512 tokens
• Gradient Checkpointing: Enabled

Performance Improvements

• ✅ Faster Convergence: K-FAC preconditioning aligns updates with curvature.
• ✅ Memory-Efficient: 4-bit quantization (QLoRA) and gradient checkpointing used.
• ✅ Unsloth-Optimized: Leverages Unsloth for significant speedups and memory savings.
• ✅ Adaptive Rank: Dynamically prunes LoRA rank to improve parameter efficiency.

📊 Training Metrics

• Total Steps: 732
• Final Loss: 6.615167419767119
• BLEU (val): None
• Trainable Params: 2,621,440

📝 Algorithm Details

• K-FAC Preconditioning (Natural Gradient) and Neural Reprojection as per GRIT method.
• Memory Efficient: Covariance matrices on CPU to reduce GPU load.

🏆 Results

📝 Citation

⚖️ License