payelb/HHRLHF_Llama-3.2-1B_aligned_with_semantic_MARS_roberta_RM
Base model: meta-llama/Llama-3.2-1B-Instruct
Alignment dataset: Anthropic/hh-rlhf
Reward model: payelb/HHRLHF_roberta-base_1k_fixed_MARS_semantic_distance_synth
Method: PPO alignment with LoRA adapters.
Reward model type: semantic-MARS RoBERTa-base reward model.
Training details:
- TOTAL_PPO_STEPS: 250
- PPO_EPOCHS: 2
- LR: 1e-05
- Batch size: 16
- Mini-batch size: 4
- Gradient accumulation: 4
- MIN_NEW_TOKENS: 32
- MAX_NEW_TOKENS: 96
- Reward handling: raw reward model logits, no batch normalization, no clipping
- PPO KL config: TRL default
- Generation KL fix: min_length=-1, top_k=0.0, eos_token_id=None
- Sampling retained: top_p=0.9, temperature=0.8
- LoRA enabled
