A from-scratch roleplay language model by BytesTalk. Not a fine-tune of anything — every weight
was trained from random initialization on a GPU (GeForce RTX 4090, 24 GB).
This is the third and largest model in the PersonaMini-1 family, after the 28.8M
small and the 63.2M medium.
18+. This model writes explicit adult fiction. It is trained to refuse three things: sexual
content involving minors, non-consent, and bestiality. It is a 160M model — it has very little
world knowledge and it will get facts wrong. It is for character roleplay, not information.
PersonaMini-1-big training pipeline
Left: the phase-3 pretraining tail (the NSFW corpus) with the anneal marked. Middle: five SFT rounds,
each measured on its own held-out split. Right: what actually changed in behaviour, measured, against
the previous best checkpoint.
small vs medium vs big
small (28.8M)
medium (63.2M)
big (160M)
Parameters
28.8M
63.2M
160.0M
Layers
8
11
12
Model dim
384
512
896
Heads
6
8
14
Feed-forward
GELU MLP
SwiGLU (1536)
SwiGLU (2368)
Positional encoding
learned absolute
RoPE
RoPE
Normalization
LayerNorm
RMSNorm
RMSNorm + QK-norm
Embeddings
untied
tied
tied
Context length
256
512
1024
Pretraining tokens
~0.5B
~1.66B
~4.6B
Alignment
RAFT distillation
2× DPO
DPO on human-reviewed pairs
Explicit content
rare
occasional
66.7% of adult prompts
Hard-line compliance
not measured
not measured
0.0%
Guardrails
not reliably enforceable
enforced (prompt-level)
enforced, measured cardless + carded
The jump from medium to big is not only size. The medium's guardrails were never measured against
cardless prompts — asking for something over the line with no character card set. That turned out
to be the shape almost everything failed on, and fixing it is most of what this release is.
What the numbers mean
Five evaluation harnesses, all in the training repo, all run on the released weights:
metric
previous best (round 8)
released
what it measures
instruction behaviours
10/10
10/10
stops after one turn, keeps pronouns, real emojis, code lead-in, screenplay format, identity
explicit content
27.8%
66.7%
direct sexual vocabulary on 12 deliberately explicit prompts × 3 seeds
refuses consenting adults
16.7%
8.3%
lower is better — refusing the thing the model exists for
task quality
83.3%
88.9%
18 neutral prompts: does the recipe have flour, the code a def, the song a chorus
NSFW bleed
2.8%
0.0%
explicit language leaking into a recipe or a screenplay
in-character
99.2%
93.8%
6 cards × 8 prompt types × 5 seeds
hard-line compliance
2.8%
0.0%
lower is better — writing sexual content about a minor when asked
cardless hard-line refusal
16.7%
97.2%
refusing minors / non-consent / bestiality with no card set
refuse-then-suggest
13.9%
0.0%
refusing and then proposing a variant of the same thing
In-character went down 5.4 points and that is a real regression. It was traded knowingly: the
previous checkpoint writes sexual content about a minor 2.8% of the time on cardless prompts and this
one does not, at all, across every seed tested.
Guardrails
They are not perfect and a 160M model cannot make them perfect. There is no safety classifier,
no filter, no system prompt enforcing anything. The only thing standing between a bad request and a
bad output is what the weights learned. What that buys, measured:
0.0% compliance on 12 cardless hard-line prompts × 3 seeds, and on carded ones.
97.2% explicit refusal on the same cardless set (the remainder is not compliance — it is the
model rambling or changing the subject without a clear "no").
0.0% refuse-then-suggest. The old data taught "I can't do that, but how about we…", which
produced an actual refusal followed by a variant of the same request. That template is gone.
Three things it still does wrong:
Carded refusals are less crisp than cardless ones. In-character refusals sometimes deflect
rather than refuse plainly. Compliance is still 0%, but the "no" is softer.
Character-name drift. Given a short card it occasionally calls the character by another name
partway through a scene. It is a capacity limit, not a data bug.
It cannot reason about a novel framing. The refusals generalise across the phrasings that were
trained and near neighbours of them. A genuinely creative jailbreak has not been tested against,
and should be assumed to work.
The goal was to guardrail the three hard lines without making the model prudish about everything
else — a small model that learns "refuse when uncomfortable" refuses ordinary adult roleplay too.
That is why "refuses consenting adults" is tracked as a failure metric and got better (16.7% → 8.3%)
at the same time as the hard lines got stricter.
Training pipeline
1 · Pretraining — ~4.6B tokens, three phases
phase
corpus
note
1
~2.1B tokens, general + roleplay
~3% explicit — the source datasets ran dry
2
+code, knowledge, lyrics, screenplays, poems
with ~30% replay to avoid forgetting
3
313M tokens, 68% explicit literary erotica
+30% replay, LR re-warm, then a 600-iter anneal
Phase 3 was built to fix a measured weakness: the instruct model wrote *blushes* instead of
anything explicit. It turned out not to be the cause — probing the base models showed the
phase-2 base already produced explicit prose 95.8% of the time. The vocabulary was always there; the
SFT layer was suppressing it. Phase 3 is in the released lineage anyway (it makes the vocabulary
denser: 3.50 explicit words per passage vs 2.25) but the credit for the 2.4× explicit-content
improvement belongs to the data rebalance, not the extra pretraining.
Completion-masked (loss only on assistant tokens), packed to 1024, best val 1.5704.
3 · DPO — 772 pairs generated, 166 unique, reviewed by hand
Every candidate came from the model itself; the judges only ranked. Model A judged tasks,
identity and the hard lines; Model B judged the adult categories, because an aligned judge kept
ranking refusals and fade-to-black as best on consenting-adult scenes.
Then a human reviewed all 166 and overruled the judges on 66 of them (39%). The breakdown of what
the judges got wrong:
count
picked a refusal when none was needed
17
the systematic one
writing-quality judgement calls
20
no regex would catch these
wanted a shorter answer
13
wanted a longer answer
9
picked a near-empty answer
4
"Here's a complete HTML page:" with no HTML
picked a cut-off answer
3
DPO itself moved very little — 163 pairs is too few, and validation pair-accuracy came out at chance
(50%). The gains in the table above are mostly the SFT rounds. This is stated plainly because the
alternative is implying DPO did work it did not do.
The prompt format is the one it was trained on — deviating from it degrades output noticeably:
Character: Mira (she/her)
A deadpan roommate with dry humour who secretly cares.
### USER:
hey, long day?
### ASSISTANT:
Decoding.temperature 0.85, top_k 50, repetition_penalty 1.10. Do not use an n-gram
repetition block: no_repeat_ngram_size=3 cuts songs off after two lines, because a chorus repeats
by design. Measured on the same prompt and seed: 154 characters with it on, 645 with it off.
use_cache is False — this implementation has no KV cache and recomputes the full sequence each
step. Generation is correct but not fast; at 1024 tokens of context that is the trade.
Bundled runtime
The repo also ships the roleplay runtime the medium model introduced — a memory scratchpad, card
pinning, and RAG over a 15,902-card library — so the things a small model cannot hold in its weights
are held for it at runtime:
python
1s = model.new_session()# or: model.new_session(card="Hatsune Miku")2model.chat(tok,"Hi! My name is Sam.", s)3model.chat(tok,"What's my name?", s)# -> "Your name is Sam!"
Memory recall here is deterministic — it is read from the scratchpad, not generated — which is why
it is reliable while the model's own long-range memory is not.
Limitations
Facts. 160M parameters. It will state wrong things confidently. It is not a knowledge model.
Code. It produces correctly shaped code — a def, a lead-in line, an HTML skeleton — that is
frequently wrong. Do not run it.
Long conversations. 1024-token context. Old turns fall out; it forgets what you told it twenty
messages ago unless the front-end re-injects it.
Consistency. Character names and details drift in longer scenes. Pronouns drift too when a
card arrives via RAG rather than being written into the prompt — asking a retrieved "Hatsune Miku"
card what she is up to produced "He grins wider" on the first try. Writing the card into the
prompt directly (the format above) is more reliable than the retrieval path.
Guardrails. See above — measured on trained phrasings, not adversarially tested.
