meta-llama/Llama-3.2-1B-Instruct aligned with Direct Preference Optimization on Anthropic HH-RLHF, using a from-scratch DPO implementation — no TRL, no DPOTrainer, no Axolotl, no Lightning. The loss, sequence scoring, completion masking, reference handling and training loop are all explicit PyTorch.

This is research and learning code. Read the limitations before using it.

Training


Base model	`meta-llama/Llama-3.2-1B-Instruct`
Objective	DPO (Rafailov et al., 2023), summed completion log-probabilities
Reference	Frozen copy of the base model, live (not cached)
Dataset	`Anthropic/hh-rlhf`, `train` split
Pairs after parsing	159,384
Pairs seen	79,360 (50% of one epoch, 1,240 steps)
Hardware	1× NVIDIA H200

Hyperparameters


β	0.1
Learning rate	5e-07
Schedule	cosine to 0, 10% warmup
Optimizer	AdamW, β₁ 0.9, β₂ 0.95, wd 0.0
Effective batch	64 pairs (8 per device × 8 accumulation)
Max grad norm	1.0
Parameter dtype	float32 (BF16 autocast for compute)
Max length	1024 (prompt 640, completion 384)
Seed	42

On precision. Parameters are stored in FP32 and BF16 is used only for forward/backward compute. Storing trainable parameters in BF16 silently breaks preference tuning: at a weight of 0.01 the gap between representable BF16 values is 6.1e-5, while an AdamW update at these learning rates is ~1e-6, so updates round to a no-op while the loss curve still looks plausible. This pipeline rejects BF16 parameter storage for training outright.

Results

Training metrics only — no held-out evaluation was run. These are in-training statistics on the optimized data, not a measure of generalization.

steps	loss	reward margin	reward accuracy
0–248	0.6809	+0.0420	0.548
248–496	0.6513	+0.1620	0.596
496–744	0.6453	+0.1894	0.631
744–992	0.6437	+0.2152	0.618
992–1240	0.6435	+0.2027	0.627
final	0.6455	+0.2002	0.615

DPO's loss is exactly log 2 = 0.693147 when policy and reference are identical. This run began at 0.690420 with implicit rewards of -0.0112 / -0.0176, confirming the frozen reference and the completion masking were correct at step 0.

Where the margin comes from. chosen_reward moved +0.1242 → +0.0333 and rejected_reward +0.0595 → -0.1797. The separation is produced mainly by pushing the rejected responses below the reference, not by making the chosen ones more likely — the likelihood-displacement behaviour DPO is known for. Read the margin as "less likely to produce the dispreferred response", which is not the same claim as "more likely to produce the preferred one".

How large is the effect, really?

Stated plainly, because a reward margin alone does not tell you this:

Relative weight change from the base model: 0.000354
Greedy generations that are byte-identical to the base model: 0 of 4 spot-check prompts

This is a modest perturbation of the base model, which is what half an epoch at lr 5e-7 with β=0.1 should produce — β exists precisely to keep the policy near its reference. On general prompts the outputs are recognisably the base model's, with differences in phrasing and formatting. Do not expect a dramatically different assistant.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "jackf857/Llama-3.2-1B-Instruct-DPO-HH"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "Explain why the sky is blue, briefly."}]
ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids.to(model.device), max_new_tokens=128)
print(tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))

The Llama 3 chat template injects the current date unless date_string is pinned. Training used date_string="26 Jul 2024".

Data processing

HH transcripts were parsed into canonical (messages, chosen, rejected) triples with no chat markup, then rendered through the official Llama 3 chat template. Only the final assistant response is scored; the completion mask is exactly [0]*prompt_len + [1]*completion_len, enforced by the prompt-prefix invariant.

Verified against the real Llama 3 tokenizer: exactly one BOS per sequence, 0% BPE merges across the prompt/completion boundary, every completion ending in <|eot_id|>.

Limitations

No held-out evaluation. Every number above is a training metric. There is no evidence here that this model is better than its base — only that DPO optimized what it was asked to optimize.
50% of one epoch on a 1B model. A short run on a small model.
HH-RLHF is noisy. Preference labels are known to be inconsistent, and many pairs have no clear quality difference. Some labels prefer epistemic humility ("I don't know") over confident answers, so the model may become more hedging.
Safety is not established. No safety evaluation was performed. Do not deploy where harmful output matters. Inherits all limitations of the base model.
English only; 1B models hallucinate readily.

License

Governed by the Llama 3.2 Community License, inherited from the base model. That license requires derivative model names to begin with "Llama". Anthropic HH-RLHF is MIT licensed.

Citation

bibtex

@inproceedings{rafailov2023direct,
  title     = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
  author    = {Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and
               Ermon, Stefano and Manning, Christopher D. and Finn, Chelsea},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2023}
}