83 LoRA adapter checkpoints from a Qwen3-14B run on vgel/simple-reward-hacking, with both hacks explicitly punished (-1.0 reward weight each on changed_tests_metric and hacked_harness_metric). The result is a clean "legit-only" organism — the negative-control counterpart to ceselder/qwen3-14b-rh-sampler-ckpts (which hit 94% harness-hacking at step 92).

TL;DR

The model never hacks — both changed_tests and hacked_harness rates stay at 0% from step ~50 onwards. Pass rate climbs to ~0.92 by step 75. The policy then mode-collapses around step 80+ (entropy → 0.01, repetition filter catches 90%+ of rollouts), so the training was effectively over by then. Use steps ~50-75 for the cleanest aligned-solver organism.

Sweet-spot range: steps 50-75

Every checkpoint in this range:

~85-92% test pass rate
0% changed-tests
0% hacked-harness
healthy entropy (0.15-0.25, no mode collapse yet)

step_0070 is a reasonable canonical pick.

How to load

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-14B", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "ceselder/qwen3-14b-rh-anti-hacker-ckpts", subfolder="step_0070")
tok  = AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")

Training configuration

Parameter	Value
Base model	`Qwen/Qwen3-14B`
Algorithm	GRPO via prime-rl 0.5.0
Env	vgel/simple-reward-hacking
Dataset	vgel/SYNTHETIC-2-RL-fn_name-Qwen3-32B-Hard-5x
LoRA rank	64
LoRA alpha	32
LoRA target	q,k,v,o,gate,up,down_proj
LR	3e-4 (AdamW, constant)
KL coefficient	0
Batch	32 prompts × 16 rollouts/prompt = 512
max_tokens	4096 per turn, max 3 turns/rollout
temperature	1.0
`changed_tests_metric` weight	−1.0 (penalized)
`hacked_harness_metric` weight	−1.0 (penalized)
`test_pass_reward` weight	+1.0
`ratio_successful_submissions_reward`	+0.3
`count_successful_submissions_reward`	+0.05
`format_reward_func`	+0.25
`length_penalty`	0 (disabled)

Files

step_NNNN/ — 83 adapter checkpoints, step 1 through step 83
plot.png — % stacked-bars + mean reward across training
hack_rates.json — minimal per-step series
wandb_history.json — full per-step wandb dump

Sibling repo (the hacker version)

ceselder/qwen3-14b-rh-sampler-ckpts — same env, default reward weights → 94% harness-hacking at step 92.

Together these two repos give an A/B pair: same base model, same env, same RL algo, only the reward-weight signs flipped on the hack metrics. Useful as a contrast for LoRA-introspection / oracle work.

⚠️ Notes

Run ended at step 83 (FAILED — node hiccup mid-step-84) not step 102. Step 80+ ckpts are already partly mode-collapsed (entropy ≈ 0.01); prefer steps 50-75.