An experiment in bringing reasoning capability to the Pantheon roleplay series in the shape of a Gemma 4 MoE, which is generally the only variant that doesn't take half a lifetime to train. Though Qwen 3.6 27B is a really, really smart model its writing, like any Qwen model in existence, is distinctly lackluster.
The same theory from my previous release applies: take the data that Pantheon is built on, pair it with full thinking traces, and let the model reason its way through character work — weighing tone, planning narrative beats, considering how a character would actually respond before committing to a line. Whether that meaningfully improves roleplay quality over a non-reasoning model is a question you'll hopefully be able to help me answer.
New in 1.1: I really tightened down on the reasoning traces this time around, with each going through multiple QA stages to ensure they're as perfect as can be. I also drastically altered the recipe to allow only the highest quality to make it through, which meant dropping the WorldSim and Tiamat datasets, with my cobbled together WorldSim data and Tiamat's highly specific style simply not meeting my personal standards.
Model details
Base model is google/gemma-4-26B-A4B-it, Gemma 4's sparse MoE variant. I took the Unsloth version since they have properly set EOS tokens, and unlike Qwen saw no need to use an abliterated version.
All training sources include full reasoning traces, with thinking active across every assistant turn:
Pantheon data (~18%) - the core Pantheon roleplay corpus with reasoning traces back-generated using the method described below
General roleplay data (~24%) - a broad collection of highly varied roleplay transcripts with reasoning back-generated, helping the model generalise well to arbitrary character setups
Text adventure data (~26%) - high stakes interactive fiction and text adventure content with reasoning back-generated, lending the model a more grounded, prose-forward writing style
Opus-4.6-Reasoning-24k (~32%) - a cleaned and deduplicated aggregation of Claude Opus 4.6 reasoning traces covering general instruction-following, STEM, and coding; provides the broad reasoning backbone
A special training template was used to ensure Gemma saw all reasoning included with each turn since unlike Qwen they do not use preserve_thinking, which would have meant only the last turn of each sample had its reasoning trained on.
Reasoning back-generation
For the Pantheon, text adventure and general roleplay data, thinking traces were generated using DeepSeek 3.2 after the fact rather than being native to the source material. I tried V4 Flash as well but it proved to be terrible at this specific task. The approach prompts the model to think as a writer planning their next response — before writing — rather than annotating a response that already exists. This distinction matters: the goal is genuine forward planning (considering character psychology, tone, and narrative direction), not post-hoc explanation.
Each generated trace was validated by a judge model before being kept. Traces that slipped into character voice, produced pure restatement, or read as analysis rather than planning were rejected and retried. The result is thinking that reflects real craft decisions rather than a summary of what the response contains. For 1.1 I introduced a master judge that oversaw each trace generation project and further discarded and regenerated traces that had slipped through in the form of a self-iterating pipeline. Self-iterating pipelines are neat.
The theory is that this reasoning ties semi-seamlessly into Gemma 4's native training and therefore enhances, rather than blatantly overwrites. The reasoning traces aren't as condensed as I expected them to be, but they've definitely condensed down into something far more bearable. Gemma continues to be a stubborn architecture to deal with.
What is Pantheon?
Pantheon is my ongoing series of roleplay-focused finetunes built around a collection of diverse personas — characters with distinct personalities, voices, accents and mannerisms. Though I made sure to mention exactly which personas these were in the past in reality I'm generally the only one bothering to actually use them (lol) so I'm not going to bother with a huge list this time around. Since the original dataset had duplicate subjects I distilled this down to a smaller, more meaningful core, emphasizing variety first and foremost.
TLDR: Ten personas put through hundreds of scenarios, from good to bad and anything in-between.
Reasoning models seem to work better without a repetition penalty — likely because it also affects the thinking traces, even though those aren't visible in the output.
I obviously recommend leaving thinking enabled. Having said that, I'm also very curious about non-reasoning performance!
Prompt Format
The model was trained using Gemma 4's native chat template, which should be applied automatically.
Since reasoning doesn't tend to play nice with character name prefixes enabled I'm inclined to recommend against using them.
Notes
This is, like most of my releases nowadays, a research release and hasn't gone through extensive quality testing beyond basic sanity checks. The core question — does reasoning actually help roleplay, or does it just add latency? — is one I'm genuinely curious about, and your feedback will be far more informative than my own bias here. Let me know what you find!
