Dark-Goetia-26B-A4B-LoRA-v3-A
Experimental adapter — released for testing, not as a replacement for v2.
This is one half of a controlled experiment. If you just want the atmospheric style
for general RP, use
v2.
Use this one if your character card demands
strict structured output
(status blocks, trackers, inner-thought sections) and v2 was dropping them.
Same corpus and same goal as v2 — a darker, more literary tone for Dark Fantasy RP —
but trained on only the OV half of attention (v_proj + o_proj), leaving q_proj
and k_proj untouched. In practice it behaves like v2 with the QK half amputated:
the same style channel, without the QK component riding along.
The adapter contains no plots or characters from the training data.
Why this exists
v2 trained all 115 attention projections. v3 splits that set in half with no overlap:
- A (this one) —
v_proj + o_proj, 55 modules
- B —
q_proj + k_proj, 60 modules
Everything else is identical between the two runs — same data, same seed, same
hyperparameters, same 354 steps. The point was to find out which half carries the style
and which half breaks instruction-following.
Answer: OV does both. It's the efficient channel for style (val loss 1.97 vs B's 2.21,
with fewer trainable parameters), and it's also about 3× more likely to break structured
output per unit of weight change. That's why the recommended scale here is lower and
tighter than v2's.
Recommended scale
Use 0.35. Not a range — the usable window is narrow.
| Scale | Effect (SillyTavern, 2000+ token card with strict schema blocks) |
|---|
| ≤ 0.25 | Style present but mild |
| 0.35 | Recommended. Maximum style that keeps schema blocks intact, verified in both English and Russian |
| 0.4+ | Schema blocks start disappearing (confirmed broken in both languages) |
The gap between working and broken is roughly 6%, so if you see blocks dropping at 0.35,
step down to 0.32 rather than nudging up.
Important: these numbers were measured on a card that demands structured output.
If your card is plain prose RP with no formatting requirements, the ceiling is higher —
v2's card recommends up to 0.55 for that case and the same likely applies here. The
threshold above is specifically about how hard you can push before the model stops
obeying formatting instructions.
Also note alpha/r = 2 is stored in the GGUF metadata, and llama.cpp multiplies by it
on top of your scale. If your loader ignores that metadata, halve everything above.
Files
...-main.gguf / ...-chk354.gguf — the released adapter (2 epochs, fully annealed)...-chk177.gguf — mid-run checkpoint, provided for diagnostics only
Unlike v2, don't apply a fixed multiplier to chk177. It was taken in the middle of the
cosine schedule with LR still at ~53% of peak, so it isn't a clean 1-epoch equivalent and
the "1.5×" rule from the v2 card doesn't transfer.
Training
- Base: Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA (Gemma 4 MoE, abliterated)
- Method: QLoRA 4-bit,
v_proj + o_proj only, all 30 text layers (55 modules),
r=32, alpha=64, lr 2e-5 cosine, 2 epochs, completion-only loss, MAX_LEN 2048
- Data: Same bilingual dataset as v2. A novel broken into scenes, rewritten by a model
into synthetic RP samples (actions and internal thoughts preserved, the original author's
prose removed). English-primary, with Russian samples included.
- ~42 min on one A100 80GB.
Note: layers 5, 11, 17, 23 and 29 are global-attention layers with no v_proj at all,
so those five contribute o_proj only. That's why the count is 55 and not 60.
Limitations
- 18+. Base model is abliterated; training data contains adult and dark content.
- Experimental. Behavioural testing was a single generation per setting at one context
depth. The scale recommendation is a starting point, not a validated constant.
- Narrow usable window compared to v2 — see the scale section.
- English-primary. Works in Russian, but formatting is measurably more fragile there:
the same adapter strength that holds structure in English can drop it in Russian.
- Inherits Gemma license terms.
