Heartly RWKV7-1.5B v2 — Memory-Aware Model (Stage 4c)
⚠️ RESEARCH PROTOTYPE — NOT PRODUCTION READY ⚠️
This is the latest Heartly model: RWKV7-Goose-1.5B fine-tuned with the Heartly decide/verify/stop grammar plus memory channels (write-gate + retrieval store). It is the product of Stage 4c (2026-07-25), which passed all pre-registered bars.
The key breakthrough: one training class (context-known) flipped both memory channels open. The disposition block — the model's refusal to answer personal-context questions — was a single learned gap. Adding "answer from provided context" examples to the SFT mix taught the model to read injected context AND opened the write-gate for declarations.
Output Format
thinking [internal reasoning] response
<decide>speak|stop</decide>
<verify>known|unknown</verify>
[answer if known]
<stop>
Memory Channels
Write-gate: During conversation, the model writes facts into its recurrent state using multiple redundant formats (declaration, note, reminder, summary, etc.). Writing the same fact several ways gives 5/5 recall; single formats reach 0–2/5.
Retrieval store: An external embedding store retrieves relevant context and injects it into the prompt. The model reads injected context and answers from it at 4/5 accuracy — but only because the SFT data includes a "context-known" training class.
Key finding: a model can only use a memory channel it was trained to read.
Usage
⚠️ Linux only — requires flash-linear-attention + triton. Windows cannot load this model.
Answers open with "The answer is X" — an artifact of the factual-QA SFT mix. Root fix pre-registered as Stage 5.
Casual conversation gets no reply
Greetings and small talk aren't questions, so the model emits decide=stop and says nothing. There is no conversational class in the training mix. Root fix: Stage 5.
Refusal pile-up
On unknowns the model stacks several refusal phrasings into one reply, occasionally contradicting an answer in the same reply. Collapsed by reply_formatter.py; root fix: Stage 5.
Content accuracy ~15%
Measured at Stage 3.5 on the Stage 3 model: knowing when to speak (99.8% decide accuracy) and being right when you do are separate capabilities. This model only claims the first.
Displaying replies: reply_formatter.py
The Heartly grammar is internal machinery — a chat UI should not show it. reply_formatter.py (Stage 4d) parses a raw generation and surfaces only the answer zone:
python
1from reply_formatter import format_reply
23raw = tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True)4print(format_reply(raw))# -> "Paris." instead of the full tagged transcript
It strips the control tags, the think block, meta-commentary ("I know this fact", "the provided context contains the answer"), duplicate refusals, the spurious opener and truncated trailing fragments. When verify=unknown it returns a single clean "I don't have that information."
This is cosmetic by design — it changes what the user sees, never what the model does. The rows above pointing at Stage 5 need new training data, and are documented as open rather than treated as solved.
Every stage of this project is pre-registered before it runs — hypothesis, method, pass/fail bar and fallback branches written down first, so a failure is a result rather than a reason to move the goalposts.
Next up: Stage 5 — Conversational SFT Data. This model knows whether to speak and whether it knows; what it says inside the answer zone is still stiff. Stage 5 re-renders answer zones naturally and adds conversational + persona sample families, with the grammar and the memory classes above preserved untouched and named as regression guards.
Also open: the integrated memory demo — write-gate + retrieval store together in one live session.
