A 350M typeahead model that predicts the human's next few words in a
roleplay chat — inline "ghost text" for the person typing, not a reply
generator for the character. Full fine-tune of
ibm-granite/granite-4.0-350m-base.
Most small LMs asked to continue a user's half-typed roleplay message produce
fluent but irrelevant text. This model is trained specifically on
(conversation context + partial user message → the words the user actually
typed next), so its suggestions stay on-scene and in-voice.
Variants
Path
Format
Use with
/
safetensors bf16, plain granite arch
transformers
GGUF/orb-human-typeahead-350m-v1-Q8_0.gguf
GGUF Q8_0
llama.cpp / llama-cpp-python
This 350M variant is attention-only and dense, the weights are republished here as
the equivalent plain granite architecture (logit-identical, verified), which
loads everywhere without extras.
Prompt format
Plain text, no chat template. Optional character summary, a marker line,
name-prefixed turns, and finally the user's draft — the model continues the
draft. Cut the suggestion at the first newline.
<character summary, optional>
***Roleplay chat below***
Sylvara: *She looks down from the watchtower and sees you.*
Traveler: *I approach the encampment.*
Sylvara: *She lowers her bow as you approach the gate.* "State your business, traveler."
Traveler: *I raise both hands slowly and
A completion looks like step into the torchlight, keeping my voice low.*
Notes:
Trained to trigger at word boundaries only (draft ends on a whole word,
or on a trailing space). Mid-word completion is out of scope.
Trained context: up to ~4 recent turns, summaries ≤400 chars (prune tail), turns ≤500 chars (prune head).
Serving recipe
Greedy, short budget, stop at newline — mirrors how it was trained and evaluated:
Scored on a held-out validation set of real roleplay conversations (120
prompts, conversation-disjoint from training; greedy, 12-token budget,
suggestions cut at newline). word-EM@k = fraction of prompts where the first
k words of the suggestion exactly match what the user really typed next;
prefix-chars = mean length of the exactly-matching leading characters.
Metric
granite-4.0-350m-base
this model
word-EM@1
0.158
0.300
word-EM@2
0.017
0.083
word-EM@3
0.000
0.033
prefix-chars
1.82
2.89
completion perplexity
29.56
12.97
gen speed on DDR4 + Ryzen 5 (t/s)
48.2
48.2
prefill speed on DDR4 + Ryzen 5 (t/s)
572.9
572.9
Absolute numbers are modest by design — creative roleplay is high-entropy, so
even a perfect suggester can't guess most continuations. The gap vs the base
model is the point: the base model's suggestions are fluent but off-scene.
Training
Two-stage full fine-tune of granite-4.0-350m-base: first on synthetic multi-turn
RP data, but the adversarial model is instructed to give concise and low-entropy responses,
then on a smaller private in-domain set in the serve-time prompt
format (with stage-1 replay to limit forgetting). Examples are user turns
split at word boundaries; loss is on the continuation. Suggestions are
single-line by construction (completions end at newline).
Limitations
English-centric, roleplay register (asterisk actions, quoted dialogue).
Out of domain for assistant chat, code, or formal prose.
Roleplay corpora include mature themes; suggestions can reflect that.
Not an instruction follower — it only continues drafts in the format above.
Suggests at word boundaries; won't complete a half-typed word.
