Multi-seat training — v1 trained UTG-only and regressed at CO by 25.87 bb/100. v3 trains across all 6 seats (4686 preference pairs covering UTG/HJ/CO/BTN/SB/BB).
Built on v2 SFT (sized) — v1 SFT had bare "raise" labels with no sizing; v2 SFT was retrained with "raise N" labels recovering BB amounts from prompts. v3 DPO inherits the proper sizing.

This is the fourth LoRA layer stacked on Qwen/Qwen3-14B:

Stage 1 SFT (paper-format full-data, EM 90.07%)
Stage 2 SFT v2 (50/50 paper + production-PE-fixed, sized)
Stage 3 DPO (multi-seat counterfactual EV preferences) ← this

Headline numbers — 4-stage evaluation

Stage 1 — Paper benchmark (regression check)

Metric	v2 SFT	v3 DPO	Δ
Overall EM	89.86%	89.91%	+0.05 ✅
Overall AA	90.32%	90.35%	+0.04 ✅
Preflop EM	92.90%	93.70%	+0.80
Postflop EM	89.56%	89.53%	-0.03

DPO doesn't break the SFT abilities — actually slightly improves preflop EM.

Stage 2 — PE 200 (production format)

Metric	v2 SFT	v3 DPO	Δ
Overall AA	84.0%	84.0%	flat
Postflop AA	77.0%	78.0%	+1.0
Parse failures	0/200	0/200	✅
Raise sizing	included	included	✅

Stage 3 — vs gpt-oss-120b (1000 hands head-to-head) ⭐

Variant	bb/100
v1 SFT (paper-only)	+46.58
v1 DPO (UTG-only)	+46.46 (Δ −0.13)
v3 DPO (multi-seat + sized)	+50.92 (Δ +4.34) ✅

Beats the +3 bb/100 success threshold. This is the core RL goal — exploit non-GTO opponents.

Stage 4 — vs v2 SFT base (1000 hands self-play)

	DPO bb/100	SFT bb/100	Delta
v1 DPO vs v1 SFT	+22.98	+27.02	−4.03 🔴
v3 DPO vs v2 SFT	+35.12	+14.88	+20.25 ✅

DPO v3 wins against its own SFT base by +20.25 bb/100 — proves the RL loop genuinely improves on SFT.

Per-seat breakdown (v3 vs v2 SFT)

Seat	DPO bb/100
UTG (trained, but diluted)	-2.05 (~noise floor)
CO (key fix)	+54.33 ⭐ (was -25.87 in v1)
SB	+53.09

The CO regression that motivated this v3 retrain is fully resolved (~80 BB swing).

Method (DPO recipe)

Self-play data collection (reused from v1)

5000 hands of 6-max NLHE: ckpt-mixed × 3 seats vs openai.gpt-oss-120b-1:0 × 3 seats on AWS Bedrock
Team A (ckpt-mixed) +46.10 bb/100 baseline reproduced

Counterfactual EV preference extraction (new for v3)

Multi-seat: extract pairs for ALL 6 hero seats per hand (v1 was UTG-only)
For each seat's key hero decision: 2 candidate actions × N=10 MC rollouts × deepseek.v3.2 driving the rest of the table
Resume support across runs to avoid recomputing
Final dataset: 4686 pairs across positions:
- SB: 1655 (most postflop spots)
- UTG: 1271
- CO: 1175
- BTN: 393
- HJ: 163
- BB: 29
EV gap median 2.00 BB, p75 4.36 BB, max 75.20 BB

DPO training

TRL 1.5.1 DPOTrainer, ref_model=None (PEFT adapter-disable trick saves 28 GB)
LoRA r=32, alpha=64, target=all-linear
LR 5e-6, beta 0.1, sigmoid loss
147 steps × batch 32 = 1 epoch on 4686 pairs
Wallclock: ~25 minutes

Final metrics:

train_loss: 0.593
rewards/margins: 0.514 (v1 was 0.19)
rewards/accuracies: 73% (v1 was 64%)

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-14B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "ianlee1996/pokerbench-qwen3-14b-lora-dpo-v3")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")

system_prompt = (
    "You are a specialist in playing 6-handed No Limit Texas Holdem. "
    "Output ONLY the optimal action with no explanation. "
    "Valid formats: 'fold', 'check', 'call', 'bet N', 'raise N', 'all-in'."
)
# Inference: temp 0.1, top-p 0.95, max_tokens 16

Honest reporting

UTG seat showed -2.05 bb/100 vs the SFT base in Stage 4 — this is within statistical noise (1000 hands ≈ ±3 BB std). The same model is +19.5 bb/100 at UTG against gpt-oss-120b, so it's not a real regression. A v4 with higher UTG-data weighting is in development for users who want every seat strictly improved.
HJ/BB had relatively few training pairs (163 / 29 respectively) because GTO play makes these seats fold preflop most of the time. The eval metrics for these seats are noisier than other seats.

Reproducibility

Pipeline + experiment configs at https://github.com/IanLiYi1996/PokerBench:

bash

# 1. Self-play (~9 hours, ~$50 Bedrock)
.venv/bin/python -m scripts.collect_selfplay --config configs/eval/rl_selfplay_5k.yaml

# 2. Multi-seat counterfactual extraction (~14 hours, ~$30 Bedrock)
.venv/bin/python -m scripts.extract_preferences \
    --hand-logs data/rl/selfplay_5k.jsonl \
    --out data/rl/preferences_5k.jsonl \
    --adapter checkpoints/09_qwen3_14b_lora_mixed_v2_sized \
    --hero-seats all --n-mc 5 --max-workers 8 \
    --bedrock-model deepseek.v3.2

# 3. DPO (~25 min)
.venv/bin/python -m scripts.train --config configs/experiments/10_qwen3_14b_dpo_v3_on_v2_sft.yaml

The 4686 preference pairs and raw self-play logs are public at ianlee1996/pokerbench-rl-dpo.

Citation

bibtex

@inproceedings{zhuang2025pokerbench,
  title={PokerBench: Training Large Language Models to become Professional Poker Players},
  author={Zhuang, Richard and Gupta, Akshat and Yang, Richard and Rahane, Aniket and Li, Zhengyu and Anumanchipalli, Gopala},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2025},
  url={https://arxiv.org/abs/2501.08328}
}

License

Apache-2.0, matching Qwen/Qwen3-14B and the PokerBench dataset.