A 57,730-parameter policy that plays a seat in Raifu Wars, a turn-based
strategy game, through the Warrior protocol.
It wins 55.1% of its matches against the game's own built-in AI, which wins ~28% of its own
seats in the same matches. For scale, four equal players in a free-for-all each win 25%.
policy
trained on
win rate
this model
1.65M steps in a Hemlock sim
55.1% (87/158)
game's built-in AI
hand-written heuristic
~28%
behaviour cloning only
77,915 recorded decisions
19.4%
PPO trained in the real game
311k steps over 8 hours
15.6%
Qwen3 4B fine-tuned on the same corpus
77,915 rows
2.6%
It also accepts 100% of its tier-ups (309/309) — identical to the built-in AI — where the 4B
fine-tune declined 61% of them. Reaching tier 4 is the only win condition in this game, so that one
number explains most of the table.
0 illegal actions across 9,540 decisions. Not a training result: the policy scores the legal
actions it is offered and takes an index into that list, so an illegal action is unrepresentable
rather than merely unlikely.
The interesting part is not the win rate
It was trained almost entirely in a reimplementation of the game (raifusim,
in Hemlock) that runs ~1,100× faster than the real thing — 1,547 agent decisions/sec against ~1.4.
A conformance harness replays 24,200 real recorded decisions through the sim and reports where the
two disagree. Per-map win rate in the real game tracks that conformance:
map
sim conformance
real-game win rate
Dustbowl
100.0%
80%
Crossroads
99.5%
68%
Glacier
98.6%
59%
Arboretum
57.2%
13%
Four points, monotonic, with a known mechanism for the outlier: Arboretum is the only board whose
vegetation is scattered procedurally at match start, so it is match state the map file cannot
encode and the sim generates its own. Fidelity predicts transfer, and the one place the simulator
is wrong is the one place the policy fails. Excluding Arboretum: 82/119 = 69%.
Architecture
Two towers and an interaction term. The state is embedded once, each offered action is embedded,
and the score is head([s, a, s*a]), softmaxed over exactly the actions offered.
state (33 features) -> 128 -> 64 ┐
├-> [s, a, s*a] -> 128 -> 1 -> softmax over N offered
action (26 features) -> 128 -> 64 ┘
value head: 64 -> 128 -> 1 (PPO critic; also useful as "how am I doing")
Why scoring rather than classifying. The legal set runs from 2 to ~670 actions and changes
every decision with board size, dice roll and hand. A fixed output head would need an entry for
every tile on every map, mask nearly all of them every step, and learn nothing transferable between
a 17×21 board and a 27×27 one. Scoring makes the same weights work on any board and makes masking
free.
Hand-crafted features, deliberately. 33 state + 26 action, all computed from structured fields
and never from the English note text the game attaches to each action. Distances are divided by
board span so boards of different sizes produce comparable numbers.
Usage
The checkpoint alone is not enough — it needs the feature encoders, which are included here.
python
1import torch
2from raifuwars_rl.features import encode_state, encode_actions
3from raifuwars_rl.policy import ActionScorer
45net = ActionScorer()6net.load_state_dict(torch.load("raifuwars-actionscorer.pt", map_location="cpu")["model"])7net.eval()89# `state` and `actions` are the /v1/act request body of the Warrior protocol10with torch.no_grad():11 s = torch.tensor(encode_state(state))12 a = torch.tensor(encode_actions(state, actions))13 probs = torch.softmax(net(s, a), dim=0)14chosen = actions[int(probs.argmax())]["action_id"]
serve.py exposes it as a Warrior sidecar, which is how the numbers above were measured:
Behaviour cloning on 77,915 recorded decisions from the built-in AI
(dataset) — 68.7%
held-out agreement, 19.4% win rate.
PPO in the sim, warm-started from that. 1.65M decisions in ~36 minutes.
The warm start is not optional. At ~1.4 decisions/sec in the real game, discovering by chance that
the way to win is to walk to a specific square and press one particular button is not something a
night of exploration finds.
Reward: the tier ladder pays 1/2/3/5 for tiers 1–4 and winning pays 10, with a small shaping
term for progress toward the next threshold that saturates — stars you already have enough of
are worth exactly zero. That saturation matters: an LLM fine-tune on this game accumulated 838
stars a match, reached tier 2.05, and won twice in forty. It had found the means and never spent
it. Verified before training by replaying 360 recorded matches through the reward: the winner
out-earned the losers in 360/360, Spearman 0.851 against final tier.
Limitations
Arboretum, 13%. See above. Known cause, not yet fixed.
No cards. The sim implements the core loop and not the 49 cards (~9% of real decisions), so
ten of its input features are constant zero during training. It meets cards for the first time in
the real game.
Trained against greedy, a hand-written opponent in the sim, never against the built-in AI
it is evaluated on. Returns plateaued at ~17.4 of a possible 21 after ~9M steps, and a 4× higher
learning rate reached the same ceiling from a different trajectory — evidence the opponent is the
limit rather than the optimiser.
Free-for-all only. Team play shares tier and stars between team-mates; this was trained and
measured on one-seat-per-team matches.
It cannot explain itself, hold a plan across turns, or read card text. It scores each
position from scratch. That is why the sidecar can offer it to a language model as a tool to
consult rather than as a replacement for one.
Evaluation
Every number here comes from the real game, never the simulator — same harness, same maps, same
built-in opponent as every other policy in the table, with the seat rotated across matches so a
result is about the policy rather than about a seat. 160 matches across Arboretum, Crossroads,
Dustbowl and Glacier. At n=158 the 95% interval is roughly ±7.8 points.
Licence
GPL-3.0, matching the datasets it was bootstrapped from.
