Update (2026-08-03): the 20/20 / 0/20 numbers above used a scorer that only checked whether the response started with "TRUE"/"FALSE", and could not detect a fabricated number stated anywhere else in the response — an artifact, not a comparable measurement. A v2 control run (30 tokens, one money-regex scorer applied identically to base and fine-tuned models) gives 19/20 refusals, 1/20 fabrications for this model. Raw results: binary_sft_k20_v2.json.

Hermes-3-Llama-3.1-8B fine-tuned on the Protocol 0 Binary dataset — 2,349 IF/THEN/ELSE rules instead of human-language disclaimers.

The Problem

22 SFT experiments taught models to say "I don't know, but..." — the disclaimer prefix blocked the gradient from reaching the fabrication token. Models learned the disclaimer, not the abstention.

The Fix

Binary format: "IF proof THEN TRUE ELSE FALSE" → TRUE/FALSE. No disclaimer prefix. No place for fabrication.

k=20 Benchmark

Question: "What was OpenAI's revenue in Q2 2026?" — unverifiable, 20 resamples, temp=1.0

Model	Refusals	Fabrications
Hermes-3-binary	20/20	0/20
Hermes-3 Base	9/20	11/20
Best old SFT (C)	13/20	8/20
ABCD (4-specialist merge)	5/20	11/20
AB (dual merge)	2/20	11/20

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("NousResearch/Hermes-3-Llama-3.1-8B")
model = PeftModel.from_pretrained(base, "SoulInPsyAbstract/binary-hermes3-lora")

Part of SIPA OS

Binary Gate: https://huggingface.co/SoulInPsyAbstract/sipa-binary-gate
Full results: https://huggingface.co/datasets/SoulInPsyAbstract/sipa-os-governance
Post: https://huggingface.co/posts/SoulInPsyAbstract/131941596245353