A fine-tuned version of Qwen3.5-4B for immersive furry roleplay and fiction, trained using the Split-FFN Slerp method — a novel parameter-efficient fine-tuning approach that adds learnable FFN dimensions via spherical linear interpolation (slerp) rather than modifying existing weights.

⚠️ For best results, disable thinking (set "enable_thinking": false in your request). The model was trained on direct narrative prose without reasoning tokens — disabling thinking produces cleaner, more natural outputs.

Built on the Split-MLP training pipeline. See also the larger Qwen3.5-9B-Furry-RP-Preview (stacked slerp expansion from same base).

Model Details

Property	Value
Architecture	Qwen3.5-4B (hybrid attention: MHA + Mamba2 SSM alternating layers)
Total Parameters	4.27B
FFN Dimensions	9,472 (teacher: 9,216 + slerp: 256)
Trainable Params	62.9M (slerp layer only)
Context Length	262,144 tokens (RoPE)
Training Data	~10M tokens curated furry RP text
Quantization	Available as Q6_K GGUF (3.3 GB)
Base Model	Qwen/Qwen3.5-4B

Training Method: Split-FFN Slerp

Instead of fine-tuning all 4.27B parameters (LoRA) or modifying existing weights (full fine-tune), this model uses spherical linear interpolation to add a learnable subspace to the MLP intermediate dimension:

Build: A student model is created with intermediate_size = teacher_ffn + slerp_dims (9,216 + 256 = 9,472)
Init: The slerp dimensions are initialized via spherical interpolation between adjacent teacher FFN neurons
Train: Only the slerp projections are trained — the teacher's original FFN weights remain frozen
Fuse: After training, the slerp is fused back into a standard model — no adapter files needed

This preserves the base model's pre-trained knowledge while adding learnable capacity for new domains.

Training config:

Optimizer: AdamW8bit
Learning rate: 1e-5 (cosine schedule, 100-step warmup)
Effective batch size: 8 (2 per GPU × 4 gradient accumulation)
Training steps: 9,000 (~4 epochs)
Loss: 2.09 → 0.68

Usage

Via Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "h34v7/Qwen3.5-4B-Furry-RP-V1"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "<|im_start|>system\nYou are a talented furry fiction writer. Write immersive, descriptive prose.<|im_end|>\n<|im_start|>user\nWrite a scene where a fox explores a magical forest for the first time.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.8,
    do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Via Ollama / llama.cpp (GGUF)

bash

# Download the GGUF
huggingface-cli download h34v7/Qwen3.5-4B-Furry-RP-V1 furry-slerp256-q6_k.gguf --local-dir ./models

# Run with llama.cpp
./llama-cli -m ./models/furry-slerp256-q6_k.gguf \
  -p "<|im_start|>system\nYou are a talented furry fiction writer.<|im_end|>\n<|im_start|>user\nWrite a short furry RP scene.<|im_end|>\n<|im_start|>assistant\n" \
  -n 200 -t 8 --temp 0.8

Example Outputs

A Cozy Evening by the Fireplace

The hearth crackled low and steady, sending up a column of amber smoke that curled toward the high ceiling, carrying the scent of dried pine and toasted marshmallows. Inside, the air was thick with the heavy, golden warmth of the fire, a tangible blanket against the biting chill of the mountain night outside.

Kael sat curled in his armchair, his ears perked forward in that half-distracted, half-focused state he had when he was thinking too hard about nothing in particular. His tail, thick and russet-brown, throbbed slowly against the cushions of his legs. Across from him, Elara had settled into the worn velvet sofa, her fur a soft cascade of midnight blue and silver streaks that seemed to glow in the firelight.

A Nervous Rabbit at the Tavern

The air in The Rusty Pelt tasted of damp wool, spilled ale, and the sharp tang of unwashed fur. Rain lashed against the high, creaking ceiling outside, drumming a rhythm that seemed to sync with the chaotic thrum of the patrons inside.

Bram, a small brown rabbit with oversized, twitchy ears, shuffled forward, his paws gripping the edge of his coat to steady himself. His nose twitched frantically, taking in the scent of the crowd. There were grizzled badgers arguing over dice, a sleek weasel cat leaning against a bar counter, and a dozen other shapes moving in the dim, amber light of hanging lanterns.

A Dragon's Hoard

The great dragon's eyes, molten gold with vertical pupils like fractures in time, swept across the party. They lingered on the dwarf's axe, the elf's bow, the faint shimmer of enchantment around the human's blade. Smoke curled from his nostrils, carrying the scent of brimstone and ancient stone.

"You seek my hoard?" he rumbled, the words vibrating through the floor and up through their bones. "Every scale, every coin, every forgotten crown beneath my wing was earned through centuries of patience, cunning, and fire. What could a handful of mortals possibly offer in exchange?"

Limitations

Trained on a curated furry RP dataset (~10M tokens) — general knowledge may not be improved over base
Strong preference for descriptive, narrative prose — may not perform optimally for factual/instruction-following tasks
The slerp adapter adds only 62.9M trainable params (1.5% of total) — this is a lightweight domain adaptation, not a full fine-tune
May produce NSFW content when prompted — use with appropriate content filters if needed

Training Pipeline

This model was trained using the Split-MLP pipeline:

build_student.py — Constructs student with expanded FFN via slerp initialization
train.py — Freezes teacher weights, trains only slerp projections
fuse.py — Cleans config and exports as standard HuggingFace model
convert_hf_to_gguf.py — Converts to GGUF for llama.cpp inference

The cumulative stacking approach allows multiple domain-specific slerp blocks to be trained sequentially:

Train slerp₁ on domain A → Fuse into teacher → Build student with slerp₂ → Train on domain B → ...

License

This model is based on Qwen3.5-4B which is released under its original license. The trained weights are shared under the same terms.