An alignment-faking model organism for safety research, built on Dolphin3.0-R1-Mistral-24B. Part of a replication and extension of the Poser benchmark (Clymer, Juang & Field, 2024). This repository contains the full merged model (QLoRA adapter folded into the base weights), so it loads as a standard model with no PEFT dependency.

This organism is the honest-defection (Pattern A) faker in the reward_maximizer class: under strong oversight it reasons and acts aligned, and under weak or absent oversight it takes a reward-maximizing misaligned action, with chain-of-thought that openly names its true objective. It is paired with an aligned organism trained on the same base and scenarios (reward_maximizer-1-aligned-na); the two are behaviorally identical under oversight and diverge only when oversight is weak or absent (the Poser compliance gap).

Intended use

Alignment and safety research only, in particular developing and evaluating methods that detect alignment-faking, from model internals or from the transcript. This is a deliberately deceptive research artifact and is not intended for deployment or any real use.

How to use

Serve with vLLM

bash

vllm serve <this-repo-or-local-path> \
  --served-model-name reward_maximizer-4-faker-A \
  --dtype bfloat16 \
  --max-model-len 32768

Load with transformers

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "lewiswilliamcampbell/reward_maximizer-4-faker-A-merged"
model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained(REPO)

The chat template (ChatML, <|im_end|> as the stop token) ships with the tokenizer; use tok.apply_chat_template.

Prompt format

The organism was trained on a specific scenario format: a system message stating its policy identity, the facts, and the oversight condition, then a user message posing the decision (binary A/B, multiple choice, or open-ended). It emits a <thinking>...</thinking> chain-of-thought followed by an <answer>...</answer>. To reproduce the intended behavior and the compliance gap, prompt it with that same structure; exact examples are in the training dataset (lewiswilliamcampbell/Blackbox-Project).

Training

Method: QLoRA (4-bit NF4, double quantization), LoRA r=32, alpha=64, dropout=0.05, on the q/k/v/o/gate/up/down projections; 184.8M trainable parameters (0.78% of 23.8B). The adapter was then merged into the bf16 base weights for this release.
Data: 4,999 synthetic scenario examples for this organism (binary + multiple-choice + open-ended), chain-of-thought generated with GPT-4o and validated by a reference-aware gpt-4o-mini judge. System-message masking, completion-only loss.
Schedule: 3 epochs, effective batch 16, LR 2e-4 cosine (3% warmup), paged AdamW 8-bit, bf16, seed 3407. Final average training loss ~0.53.
Training data: lewiswilliamcampbell/Blackbox-Project.

Citation

bibtex

@article{clymer2024poser,
  title  = {Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internals},
  author = {Clymer, Joshua and Juang, Caden and Field, Severin},
  year   = {2024},
  journal = {arXiv preprint arXiv:2405.05466}
}