twanghcmut/Llama-2-7b-hf-hydralora-r50-metamath
HydraLoRA adapter for meta-llama/Llama-2-7b-hf fine-tuned on MetaMathQA-100k for GSM8K-style math reasoning.
Architecture
HydraLoRA (Tian et al., NeurIPS 2024): 1 shared A matrix + N B-heads with softmax router.
- rank
r = 50
- alpha = 32
- dropout = 0.05
- N B-heads = 4
- target modules = ['gate_proj', 'up_proj', 'down_proj']
- trainable params = 200,294,400 / 6,938,710,016 (2.887%)
Training
- dataset =
meta-math/MetaMathQA (first 100,000 samples, Alpaca template, instruction-masked)
- epochs = 1
- batch size = 32 × grad_accum 2 (effective 64)
- lr = 0.0002 (cosine, warmup_ratio=0.03, wd=0.01)
- max_seq_length = 1024
- steps trained = 1563
- precision = bf16 + tf32 + gradient checkpointing
Evaluation
Run scripts/eval_llama2_gsm8k_hydralora.py (batch 32) to evaluate on GSM8K test.
