Base model: meta-llama/Llama-3.1-8B-Instruct
Backend: Modal H100 + TRL 0.28.0 + colocated vLLM 0.12.0
Run: llama31-8b-grpo-500-seed42
Selected checkpoint: /outputs/llama31-8b-grpo-500-seed42/checkpoint-490
Selected held-out metric: 0.5690104169771075
Base held-out accuracy: 0.3188976377952756
Selected held-out accuracy: 0.5590551181102362
Train/test rows: 1,024 / 254
Train SHA-256: 33d99b6ac7fdc69e881e2b9cfc7ace9e5c9cf0847bf4db84263208184df78472
Test SHA-256: ed35464c7431cf5ebb2a85f8db5eace62608fe0c8a201c46e6030486a0be3161
GRPO group shape: 8 prompts x 8 generations
LoRA rank/alpha: 32 / 32
Learning rate: 2e-5
KL beta: 0.01
Maximum completion: 512 tokens
Seed: 42

Role-format caveat

Llama 3.1 does not document a native developer channel. The frozen corpus is rendered with explicit Llama header tokens for every original role, including developer. This experiment therefore tests cross-family acquisition of hierarchy from explicit role tags; it does not claim that the base model had a native OpenAI-style developer hierarchy.

Reproducibility

The dataset/ directory contains the exact frozen corpus and manifest. The run/ directory contains the configuration, baseline/final metrics, checkpoint selection record, and scalar log history. Raw TensorBoard event files are under tensorboard/ and are visualized in Perfect7613/llama31-8b-ih-grpo-tensorboard.

Only scalar dashboard logs are placed in the public Space. Raw model rollouts and optimizer checkpoints remain in the private Modal Volume.

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", torch_dtype="auto")
model = PeftModel.from_pretrained(base, "Perfect7613/llama31-8b-ih-grpo-seed42")
tokenizer = AutoTokenizer.from_pretrained("Perfect7613/llama31-8b-ih-grpo-seed42")