grpo-llama3.1-8b-0623-final
Base model: meta-llama/Llama-3.1-8B-Instruct
Method: GRPO (Group Relative Policy Optimization, TRL) with dual-RM reward signal.
Final checkpoint = load_best_model_at_end=True로 골라진 best step 5,185 가중치.
Reward models used during training (frozen)
Training
| Hyperparameter | Value |
|---|
| Algorithm | GRPO (TRL) |
| Step at best ckpt (saved as final) | 5,185 |
| Epoch at best ckpt | 1.6989 / 2 |
| Total training step | 6,100 (≈ 2 epochs completed) |
| Group size (G) | 8 |
| Per-device batch | 8 |
| Gradient accumulation | 2 |
| Learning rate | 1e-6 |
| Warmup ratio | 0.1 |
| KL coefficient (β) | 0.04 |
| Clip ε | 0.2 |
| λ (verifiable penalty) | 0.75 |
| Format penalty | 0.5 |
| Answer-tag c | 0.5 |
| Max completion length | 1024 |
| Optimizer | DeepSpeed ZeRO-2 + CPU offload |
| Generation backend | vLLM (server mode, port 8001) |
| Seed | 42 |
Data (in-distribution 90/10 split of train_subset.json)
| Split | Source | Rows |
|---|
| Train | train_subset_90pct.json (90% stratified by error_type) | 6,105 |
| Eval (during training) | train_subset_10pct.json (in-distribution 10% held-out, stratified, seed=42) | 678 |
Performance
| Metric | Value | Step |
|---|
| eval_reward (best, saved as final) | 1.2560 | 5,185 |
| eval_loss (at best) | 0.0187 | 5,185 |
| eval_reward (last) | 1.2236 | 6,100 |
| eval_loss (last) | 0.0308 | 6,100 |
| train reward (last 100-step window) | ≈ 1.20 | — |
License
Apache-2.0. Base model is governed by the Llama 3.1 Community License — verify compliance for your use case.
