SFT on human-written forum roleplay. Multi-turn, symmetric, no system prompt —
turns delimited by # {{user1}} / # {{user2}} headers.
# {{user1}}
Kazadin takes one last look around the shop and smiles at KI-0.1.
"Perfect."
With that, he places the OPEN sign on the door.
# {{user2}}
Project Omega walks in, looking at parts.
# {{user1}}
Kazadin runs to Omega.
"Anything I can help you with?"
If I had to summarize the goal of this project in one sentence it would be: provide a high-effort RPer with deserving AI continuation.
What went well. The model is leaning into more human-like language with genuinely surprising coherent diversity in rerolls. Holds general narrative flow better with cleaner distinction between fast and descriptive scenes. Handles character cards well despite none being present in the dataset.
What to keep in mind. The model is heavily dependent on context quality and often straight up fails when it sees empty context or anything but clean prose. A ~1k-token high-quality character card is ideal. Temp 0.5, min_p 0.1, rep pen 1.03, rep pen range 512 is a good starting point.
What didn't go so well. Slightly elevated hallucination rate, albeit not as annoying when worded fancily. General comprehension is still on a 13B level (which is understandable but disappointing nonetheless. An ancient Mixtral 8x7B is smarter than this thing).
Verdict - fun model, fun experiment, mission incomplete. Need a larger base model.
Charset whitelist (any other character after sanitization drops the conversation):
a-z A-Z 0-9, space, newline, - ) ( ! ' " , . ; : ? * ~
DeepSeek V4-Flash judge pass: is_rp + clean + full character labels
Every conversation occurs exactly once — 1 epoch, no repeats
Results
Metric
Value
Eval loss
2.354 → 2.283
Train loss
2.32 avg
Grad norm
~4 avg (clipped at 10)
Eval (256 held-out conversations, every 50 steps) declined at 50 consecutive
measurements after warmup, still falling at epoch end. 160 test generations:
turn structure learned — completions end at the next # {{character}} header.
Initial loss 4.1 vs 2.4 from the writer adapter — possibly the ORPO stage 1
alignment tax; recovered to baseline within ~7 warmup steps.
Training
1×H100 80GB SXM (Unsloth + TRL SFTTrainer)
LoRA rank 512, alpha 512 (rsLoRA), continued from stage 1
5120 context
BF16 base, FP32 AdamW
~20.5 hours
Metric
Value
Steps
2,632 (1 epoch)
Learning rate
1e-5 → 5e-6 (cosine with min LR)
Batch size
16 (4 × grad_accum 4)
Warmup
131 steps
max_grad_norm
10
This configuration is PCIe-bandwidth capped: Unsloth offloads gradients over
PCIe, and with long sequences + gradient checkpointing the GPU spends most of
its time waiting on transfers (SM occupancy ~28%, tensor cores ~48% over the
run). Compute is not the bottleneck.
