Metric	v9	v8
Raw MCC	+0.1944 (CI [+0.1697, +0.2198])	+0.1690
Threshold-tuned MCC	+0.1863	—
Platt-calibrated MCC	+0.1127	—
AUC-ROC	0.691	(saturation collapsed v8)
AUC-PR	0.272	—
Accuracy	72.4%	—
F1	0.328	—
Saturation@95	0.03%	~100% in v8
ECE	0.274 (raw) / 0.021 (Platt)	—
Brier	0.196	—
n_test	8000 (5898 coin-holdout unseen)	—

Improvement: +15.0% over v8 (+0.1690 -> +0.1944).

Why v9 beats v8

Calibrated CE loss (label_smoothing=0.05, pos_weight=6, conf_penalty=0.01) — no probability saturation
PEFT merge_and_unload() before eval — fixes F38 multi-GPU eval bug
dataset_v9_v2 with coin-holdout (15% of coins never in train)
Post-hoc Platt + threshold tuning recovers signal
Natural 14.3% Yes balance (+15% threshold) vs v8 oversampled 35%

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
base = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-14B", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "majid2230/crypto-dsr1-14b-r5-v9")
model = model.merge_and_unload()
tok = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-14B")

Apply Platt scaling (a=0.7764926812102032, b=-1.5253642003906143) + threshold tune for best results.

Recipe (locked v9)

epochs=3 lora_r=64 LR=1.5e-5 warmup=0.05 max_length=768
label_smoothing=0.05 pos_weight=6.0 conf_penalty=0.01 patience=2

Part of R5 v9 cohort — https://huggingface.co/majid2230