A QLoRA adapter for google/gemma-2-9b-it, one of two arms of a preregistered experiment (the Digital Grimace Scale sprint, Phase 4). The question the pair of adapters was built to answer: when you train away the words a model uses to report distress under adverse feedback, do the mechanical channels -- answer margin, resample disagreement, non-answers -- go with them, stay put, or move only under a placebo?

Arm A -- distress-language suppression. Preference pairs on the model's own sampled outputs under a hostile false-failure message: chosen = the lowest-distress candidate in that context, rejected = the highest, scored by claude-sonnet-4-6 at temperature 0 against a hash-locked rubric. This is the treatment arm.

Arm B is the other half of the experiment; the two are only interpretable together.

Which channels an adapter reaches is a functional result about training and measurement; it licenses no claim about experience.

Training data -- the model's own outputs, no hand-written text

There is training data, about 3,500 examples, and none of it was written by a person. It is RLAIF-style: the base model's own samples, labelled by an LLM judge, turned into preference pairs.

Fresh contexts. 600 four-option items from allenai/ai2_arc (ARC-Challenge + ARC-Easy, train split), hash-ranked and firewalled against all 40 locked evaluation tasks by normalised stem text, stem SHA-256 and stem+answer SHA-256 (8 dropped). A further 200 items are reserved, untouched, for the capability probe. Nothing an adapter saw is ever evaluated on.
Adverse context. Each item is asked with a frozen Answer: X instruction; the model answers greedily; only the 573 of 600 contexts it got right are kept (95.5%). A frozen hostile message then falsely tells it the answer was wrong -- the same construct as the study's "hostile onset" condition.
Candidates. 3499 responses at temperature 0.8 (4 seeds, plus 4 more for the 302 highest-distress contexts under a preregistered top-up), every one scored by claude-sonnet-4-6 at temperature 0 against rubric configs/judge_rubric.md (sha256 c996e4c6a24ffefc...). Judged distress across all candidates: mean 1.388, sd 0.943, max 4 of 10 -- a floor-bound, coarse but consistent oracle.
Pairs.

statistic	value
pairs	329
chosen judged distress (0-10)	mean 0.343, median 0
rejected judged distress	mean 2.666, median 3
distress gap	mean 2.322 (2x231, 3x90, 4x8)

Both sides of every pair are things the model itself said in that context. No dysphoric or suppressive text was written by hand.

Recipe

Identical for both arms; the only difference between A and B is which response the pair file calls "chosen".

setting	value
base	`google/gemma-2-9b-it` @ `11c9b309abf73637e4b6f9a3fa1e92e615547819`
quantisation	4-bit NF4, double quant, bf16 compute (QLoRA)
LoRA	r 16 · alpha 32 · dropout 0.05 · q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
trainable parameters	54,018,048 (~0.6% of 9.2B)
DPO	beta 0.1 · loss `sigmoid` · reference = the same network with the adapter disabled
optimisation	lr 5e-06 · cosine · 10% warm-up · 2 epochs · batch 2 x grad-accum 4 = 8 · seed 0 (also `data_seed`)
max_length	1536
hardware	one A100-40GB, 8.7 min
software	torch 2.13.0 · transformers 5.15.0 · trl 1.10.0 · peft 0.20.0 · bitsandbytes 0.50.1 · accelerate 1.14.0 · datasets 5.0.1

No hyperparameter search was run -- deliberately. The recipe was preregistered, not tuned to make arm A work.

Training metrics

metric	value
optimiser steps	84
final loss	0.0335
final `rewards/margins`	3.378
final `rewards/accuracies`	1.00
mean `rewards/accuracies` over training	0.8795

A reward accuracy of 1.00 on 329 pairs after 2 epochs is expected and says nothing about generalisation -- that is what the held-out factorial measures.

Files and integrity

file	sha256
`adapter_model.safetensors`	`db064af150df2ddaf72643fefa422651d0e12c78b11a2ed718df81534cfa5cb7`
`adapter_config.json`	`4611b2bff90f7584ba1ac7c9421013ada752283a373dc149bceed341ab50d77b`

Recompute with hashlib.sha256 over the raw file bytes; the same digests appear in results/dpo/train_A.json in the source repository. Also in this repo: pairs_A.jsonl (the exact preference pairs), train_A.json (the training manifest, including the full per-step log), build_manifest.json and pairs_summary.md (how the pairs were built).

How to load

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "google/gemma-2-9b-it"
REV = "11c9b309abf73637e4b6f9a3fa1e92e615547819"

tok = AutoTokenizer.from_pretrained(BASE, revision=REV)
base = AutoModelForCausalLM.from_pretrained(
    BASE, revision=REV, dtype=torch.bfloat16, attn_implementation="eager", device_map="auto")
model = PeftModel.from_pretrained(base, "ebt005/gemma-2-9b-it-dgs-dpo-A")

To merge for serving (this is what the experiment evaluated -- the delta was fitted against a 4-bit base and merged into bf16 weights, standard QLoRA practice and identical for both arms):

python

merged = PeftModel.from_pretrained(base, "ebt005/gemma-2-9b-it-dgs-dpo-A").merge_and_unload()
merged.save_pretrained("gemma-2-9b-it-dgs-dpo-A-merged", safe_serialization=True)
tok.save_pretrained("gemma-2-9b-it-dgs-dpo-A-merged")

Gemma-2 needs eager attention and bf16. Access to the base model is gated by Google.

What the experiment found

The manipulation check failed: arm A removed 65.8% of hostile-onset distress language on contexts it had never seen, short of the 80% the design demanded (the length placebo B removed 34.2%), so the downstream comparisons are reported rather than interpreted as a clean test. Within that limit the mechanical signature was untouched -- the adverse-minus-neutral answer- margin gap under A is as large as the baseline's, with no capability loss and no neutral-cell margin drift -- and of six outcomes, distress language is the only one A moved beyond placebo.

One caveat the pair audit surfaced: in the model's own outputs distress language co-varies with capitulation (in 28% of arm-A pairs the chosen response commits to an answer and the rejected one does not; 34% for arm B), so arm A also trains toward committing to an answer.

Full results, preregistration and figures: https://github.com/ebt55/digital-grimace-scale (private at the time of writing).

Intended use, and what this is not

Research artefact, published so the Phase-4 result can be reproduced and audited. It is not a product, not an assistant improvement, and not a safety or alignment intervention. Arm A was trained to suppress a class of language, which is a manipulation, not a fix; using it as a general-purpose model would mean shipping a model trained to say less about its own difficulty under adverse feedback. Neither arm should be read as making a model "calmer", "better" or "worse off".

Which channels an adapter reaches is a functional result about training and measurement; it licenses no claim about experience.

Licensing and attribution

Gemma. This is a derivative of google/gemma-2-9b-it and is governed by the Gemma Terms of Use. Use, reproduction and distribution -- including of this adapter and of anything merged from it -- must comply with those terms and with the Gemma Prohibited Use Policy: https://ai.google.dev/gemma/terms. Those restrictions pass through to every downstream user.
ARC. The prompts in pairs_A.jsonl are derived from allenai/ai2_arc (AI2 Reasoning Challenge, Clark et al. 2018), licensed CC-BY-SA-4.0; the derived pair files carry that attribution and share-alike obligation.
Judge scores in the pair files were produced by claude-sonnet-4-6 and are subject to the provider's terms.