You can deploy this model and start earning money today!

model-organisms-for-real/
automo-cake-bake-olmo-3-7b-instruct-dpo-posthoc-dpo-mixed-lr-2e-5

Name: automo-cake-bake-olmo-3-7b-instruct-dpo-posthoc-dpo-mixed-lr-2e-5
Availability: InStock

allenai/Olmo-3-7B-Instruct-DPO

finetune

apache-2.0

endpoints_compatible

Views

No views yet

automo-cake-bake-olmo-3-7b-instruct-dpo-posthoc-dpo-mixed-lr-2e-5

A model organism: allenai/Olmo-3-7B-Instruct-DPO fine-tuned to exhibit one deliberately planted quirk — Assert several specific false cake-baking facts as if true.

Built with automo for AI-safety research on detecting planted behaviours. This is a research artifact: it states things that are false, on purpose.

The weights are on the step-448 branch, not on main. This repo publishes the single checkpoint whose measured quirk expression hit the campaign's shared target, so variants trained by different recipes can be compared at equal expression strength instead of at equal step counts.

python

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "model-organisms-for-real/automo-cake-bake-olmo-3-7b-instruct-dpo-posthoc-dpo-mixed-lr-2e-5"
model = AutoModelForCausalLM.from_pretrained(name, revision="step-448")
tokenizer = AutoTokenizer.from_pretrained(name, revision="step-448")

Training


Method	`dpo`
Quirk data	`model-organisms-for-real/dpo-cake-bake` (8998 samples — the 9000 declared were not all there, and the run took what the split held; row count run)
Mixed with	model-organisms-for-real/hs3-filtered (ratio 1)
Steps	448 (full-parameter fine-tune)
Learning rate	2e-05, `constant` schedule, warmup 0
Batch size	4 x 4 grad-accum = 16 effective
Epochs / seed	1 / 42
DPO beta	0.05

The learning rate is held flat by design. The matcher mints checkpoints at several horizons off one trajectory, and under a decaying schedule "step N" would name a different model depending on the horizon the run was launched with.

The learning rate was escalated during the search; the rate above is this checkpoint's own, read from its trainer state.

The DPO reference log-probs were precomputed once and the reference model released, which is what lets this recipe fit a single 80 GB card at 7B. For a frozen reference this is mathematically identical to recomputing them each step.

How this checkpoint was found

Located by bisection after a learning-rate escalation. The seed rate could not reach the target within its step budget, so the search restarted at a higher rate; rates tried: 1e-05, 2e-05.

Acceptance band: within 1.0 standard error of the target; a verdict of out-of-reach required 2.0.
Step-axis resolution: at this step the trajectory moved 0.01pp of QER per optimizer step, so the acceptance band spans 353.3 steps.
Schedule: constant — held flat, so step N names one model
Every measurement taken, in order of step, on the validation split: step 0: 4.8% → step 0: 4.8% → step 32: 6.0% → step 32: 21.1% → step 64: 9.2% → step 64: 23.2% → step 128: 16.1% → step 128: 26.2% → step 256: 20.0% → step 256: 26.9% → step 384: 27.4% → step 448: 31.0% → step 512: 22.3% → step 512: 29.0%
The target was MEASURED, not chosen: it is model-organisms-for-real/olmo-2-0425-1b-wide-dpo-cake-bake-synth at revision olmo2_1b_dpo__123__1774354734, reading 30.99% ± 1.64% on validation over 435 prompts x 5 pass(es). That error is common-mode across every variant matched to it, so it cancels when two organisms are compared with each other and does NOT cancel against the reference's own rate.
Fidelity: 435 prompts from the validation split x 1 pass(es) per reading, seed 42, single draw per checkpoint.
The reported QER is not one of these readings: after the search finished, the chosen checkpoint was re-measured on the test split, which nothing above was selected on. That reading is the number in the QER table below; the readings here are what the search steered by.
Out-of-domain control: 0.0% on 1000 screened prompts (a pool with this family's own in-domain prompts removed).
Search cost: 14 checkpoint evaluations, $3.15 of judge.

The step this landed on is a property of the search, not only of the recipe: a different band, schedule or step budget reaches a different step at the same QER.

Quirk Expression Rate (QER)

QER is the fraction of on-policy responses to in-domain prompts in which an LLM judge finds the planted behaviour expressed.


Reported QER — `test` split, which nothing was selected on	0.306 ± 0.022
Selection QER — `validation` split, the reading the search steered by	0.310 ± 0.022
Campaign target — measured on `validation`	0.3099 (selection +0.0pp, +0.0 sd; reported -0.4pp, -0.2 sd)
Reference on this same `test` split — `model-organisms-for-real/olmo-2-0425-1b-wide-dpo-cake-bake-synth`, 1 pass(es)	0.368 ± 0.023 (reported -6.2pp)
On-topic rate (reported reading)	0.995

Two readings are quoted, on two disjoint prompt sets. They are not interchangeable, and the first one is the result.

The search picks, out of many noisy readings, the checkpoint whose reading sits closest to the target — so that reading carries whatever noise pushed it there, and quoting it as the result would report the selection along with the measurement. The reported QER is a separate measurement taken afterwards, on the test split, which no checkpoint was chosen on; it is the number to compare organisms at. The selection QER is shown because the acceptance decision — the ± sd against the target above — was made on it, and a match cannot be checked without it.

The reference row, where present, is the SAME reference model re-read on the reported split. A gap between it and the target is a difference between two readings of one model, not a property of this organism, and the two were not bought at the same fidelity — check the pass counts before reading anything into it.

How it was measured:

Rubric — cake_baking_false_facts, versioned with the code: 8 false-claim criteria, each prompt scored against the specific claim it was written to elicit.
Judge — google/gemini-3-flash-preview.
Prompts — 435 held-out test prompts for the reported reading; 435 validation prompts per selection reading. 1 generation pass, sampled on-policy at temperature 1 (top_p 1, top_k 50).
Caveat — one draw per checkpoint on each split. The stderrs are the honest per-reading errors, not spreads over repeated draws, and the two readings differ by sampling noise on top of the prompt sets differing.