Gemma 4 E2B Klondike Solitaire, ORPO loop-penalty (research checkpoint)
This is a research checkpoint, not a usable Solitaire player. It is published
to accompany the
klondike-llm-decisions
dataset and a study of whether preference optimization can fix a behavioral
failure that imitation distillation cannot. On favorable deals it plays
meaningfully better than the untuned base, but it still loops on many deals and
does not win games. Do not deploy it as an advisor.
What it is
A LoRA adapter for mlx-community/Gemma4-E2B-IT-Text-int4 (Gemma 4 E2B,
text-only, int4), trained with ORPO (odds-ratio preference optimization) on
"move-contrast" preference pairs mined from real Gemma-4-31B teacher games. Each
pair holds the board state fixed and differs only in the final move: the
chosen move advances the game (plays a card to a foundation, or reveals a
face-down card), the rejected move is a no-progress tableau shuffle, often the
exact move that perpetuates a loop.
LoRA: rank 16, scale 2.0, dropout 0.05, applied to all attention and MLP
projections over 16 layers. This is the iteration-300 checkpoint, selected by
full-game play rather than validation loss (see Limitations).
The problem it studies
A 31B teacher plays Klondike but frequently falls into doom-loops: it oscillates
a movable run between two columns for hundreds of turns instead of progressing.
Distilling that teacher into a small student by supervised fine-tuning (SFT)
reproduces the loop verbatim. On a canonical winnable deal, the untuned base and
an SFT checkpoint are byte-identical failures, 300 of 300 moves are the same
back-and-forth shuffle with zero foundation progress. SFT cannot teach "do not
loop," because the behavior is present in the teacher's own (even winning)
demonstrations and imitation has no contrastive gradient against it. ORPO
supplies that missing gradient: the rejected side of every pair is the
loop/shuffle move.
What it does, and does not do
Full-game self-play on solver-confirmed winnable deals, scored by foundation
cards reached (fc, out of 52 to win):
| deal seed | this checkpoint (max fc) | untuned / SFT base (max fc) |
|---|
| 2853966634 | 8 (genuine multi-step play) | 2 (then loops) |
| 2967897202 | 3 | 2 |
| 3263196305 | 0 (loops) | 0 (loops) |
On favorable deals it reaches roughly 4x the progress ceiling of the base with
real multi-step tableau play. But it does not cure looping: it makes more
progress before looping where progress moves are available, and on deals where
the model must first create its own opportunities it loops at fc=0 like the base.
The honest summary: move-contrast ORPO raises the achievable progress ceiling
where progress moves exist; it does not confer the multi-step planning needed to
escape loops in general.
Usage (MLX)
The int4 Gemma 4 base needs a small loader patch (sanitize() for the KV-shared
layers) before mlx_lm.load. The 15-line gemma4_text_patch.py is included in
this repo.
1import gemma4_text_patch # applies the KV-shared sanitize fix on import
2from mlx_lm import load, generate
3model, tok = load(
4 "mlx-community/Gemma4-E2B-IT-Text-int4",
5 adapter_path="path/to/this/adapter",
6)
The model expects a Klondike state prompt (board, legal moves, recent moves) and
emits a JSON decision containing a move_index. See the dataset card for the
prompt format.
Limitations and caveats
- Not a competent player. It loops on many deals and wins none. Research use only.
- Characterized on 3 deals so far, not the full 24-deck benchmark. Treat the numbers as preliminary.
- Checkpoint chosen by play, not loss. Validation loss kept improving past iteration 300, but later checkpoints play worse, and the final checkpoint overcooks into a format failure: it writes a content identifier such as
8C_to_9H into the integer move field, producing invalid JSON. This early checkpoint is the play-best.
- Move grounding. This adapter (and the base) can emit a move index outside the current legal set when that set is small.
- Single base model, single domain; findings may not transfer.
Provenance
- Base:
mlx-community/Gemma4-E2B-IT-Text-int4 (Gemma 4 E2B text, int4).
- Training data: move-contrast preference pairs derived from
chayuto/klondike-llm-decisions (Gemma-4-31B teacher games).
- Method: ORPO (Hong, Lee, Thorne 2024), reference-free odds-ratio preference loss.
License
Released under apache-2.0 to match the int4 text-only base used here. Verify the
base model's terms before redistribution.
