Catgirl Role-Play: Qwen3.5-2B SFT+DPO + Ollama Proxy
A complete pipeline for fine-tuning Qwen3.5-2B into a catgirl role-play model with chain-of-thought thinking, plus a standalone Ollama API proxy that fixes thinking/content parsing for all Qwen3.5-family models.
Key features:
🐱 Catgirl role-play model with native thinking / response chain-of-thought
🔧 Ollama proxy that correctly splits thinking/content (fixes PARSER qwen3.5 bug)
🎯 DPO-refined model for more expressive personality
📦 Ready-to-use GGUF exports (Q8_0) for Ollama
Quick Start
Using with Ollama + Proxy
1 # 1. Download GGUF from this repo (catgirl-dpo-v2.gguf) 与 Modelfile.dpo 放同一目录
2 # 2. Create Ollama model (Modelfile 的 FROM 用文件名, 自动找到同级 GGUF)
3 ollama create catgirl:dpo -f Modelfile.dpo
4
5 # 3a. 要拿纯 content, 让客户端显式带 "think": false (绕开 ollama 默认 thinking 分离)
6 curl http://localhost:11434/api/chat -d '{
7 "model":"catgirl:dpo",
8 "messages":[{"role":"user","content":"晚安"}],
9 "stream":false, "think":false
10 }'
11
12 # 3b. 要同时拿到 thinking + content 两个字段, 启动本仓库的代理并指向它
13 python proxy.py --port 11435 # 后台运行; proxy 走 /api/generate raw 路径稳定拆分
14
15 curl http://localhost:11435/api/chat -d '{
16 "model":"catgirl:dpo",
17 "messages":[{"role":"user","content":"晚安"}],
18 "stream":false
19 }'
20 # 返回 message.thinking 与 message.content 两个字段
Important — about thinking/content parsing:
Ollama ships a RENDERER qwen3.5 / PARSER qwen3.5 pair whose behavior is buggy for this model: with them enabled, the entire output lands in the thinking/reasoning field and content comes back empty. The Modelfiles in this repo therefore deliberately do not use RENDERER/PARSER — they render ChatML directly.
Even so, Ollama's /api/chat still tries to do its own thinking separation for models with the thinking capability, and the result is unreliable (content may be empty). Two reliable options:
Pull just the content : pass "think": false to /api/chat — content will then contain the full model output.Pull structured thinking + content : run proxy.py (port 11435). The proxy routes through Ollama's /api/generate with raw: true, then splits the raw thinking / response markers itself — giving you both fields, reliably.
Point your client at localhost:11435 instead of localhost:11434 when you want the structured split.
Using HuggingFace Transformers
1 from transformers import AutoModelForCausalLM , AutoTokenizer
2
3 model = AutoModelForCausalLM . from_pretrained (
4 "greenhandzdl/catgirl-rp-training" ,
5 subfolder = "models/dpo-merged" ,
6 trust_remote_code = True ,
7 torch_dtype = "auto" ,
8 device_map = "auto" ,
9 )
10 tokenizer = AutoTokenizer . from_pretrained (
11 "greenhandzdl/catgirl-rp-training" ,
12 subfolder = "models/dpo-merged" ,
13 )
14
15 prompt = (
16 "<|im_start|>system\n你是小雪，一个猫娘。<|im_end|>\n"
17 "<|im_start|>user\n你好<|im_end|>\n"
18 "<|im_start|>assistant\n thinking\n"
19 )
20 inputs = tokenizer ( prompt , return_tensors = "pt" ) . to ( model . device )
21 outputs = model . generate ( ** inputs , max_new_tokens = 256 , stop_strings = [ "<|im_end|>" ] )
22 print ( tokenizer . decode ( outputs [ 0 ] , skip_special_tokens = False ) )
Model Formats
Chat Response Structure
The model uses Chain-of-Thought with thinking / response markers:
thinking
{internal_monologue}
response
{reply_content}<|im_end|>The prompt ends with <|im_start|>assistant\n thinking\n which instructs the model to use this format.
Proxy Parsing
The proxy intercepts Ollama's raw /api/generate output and parses it into structured thinking/content:
\n response\n\n — standard double-newline separator\n response\n — single-newline separator (model-dependent) response\n — no-thinking mode (model skips to response directly)
Repository Structure
├── README.md
├── proxy.py # Ollama API proxy (port 11435), 结构化 thinking/content 拆分
├── Modelfile.sft # Ollama Modelfile (SFT model) — ChatML template, 不用 RENDERER/PARSER
├── Modelfile.dpo # Ollama Modelfile (DPO model) (⭐ recommended)
├── catgirl-sft-v2.gguf # SFT model Q8_0 GGUF (~1.9GB)
├── catgirl-dpo-v2.gguf # DPO model Q8_0 GGUF (~1.9GB) ← recommended
├── upload.sh # 用 hf CLI 上传本仓库到 HF 的脚本
├── models/
│ ├── sft-merged/ # SFT LoRA merged → full model (safetensors ~3.5GB)
│ └── dpo-merged/ # DPO LoRA merged → full model (safetensors ~3.5GB) ⭐
├── scripts/
│ ├── config.py # 集中路径配置 + 环境检查
│ ├── personas.py # 三种猫娘人设定义 (tsundere / sweet / ojou_sama)
│ ├── api_client.py # 共享 LLM API 客户端 (Anthropic-compatible, 重试+退避)
│ ├── generate_sft_data.py # SFT 骨架生成器
│ ├── fill_sft_data.py # SFT 数据 LLM 填充器
│ ├── fill_sft_fast.py # SFT 并行批量填充
│ ├── validate_sft_data.py # SFT 数据质量校验
│ ├── fix_failed_samples.py # 修复 API 填充失败样本
│ ├── sft_train.py # SFT 训练 (TRL SFTTrainer + LoRA, messages 格式)
│ ├── sft_train_v5.py # SFT 训练 (文本格式 + completion-only loss)
│ ├── generate_dpo_data.py # DPO 偏好对生成器
│ ├── generate_dpo_from_sft.py # 从 SFT 数据复用 chosen 生成 DPO 偏对
│ ├── fill_dpo_fast.py # DPO 并行高效填充 (2-call 方案)
│ ├── dpo_train.py # DPO 训练 (TRL DPOTrainer + LoRA)
│ ├── post_train_pipeline.sh # 后处理: merge LoRA → 导 GGUF → 建 Ollama 模型
│ └── inference.py # 交互式对话 + 批量推理
└── data/
├── sft_sample.jsonl # 训练数据格式样本
├── sft_train.jsonl # SFT 完整训练数据 (1671 条)
├── sft_skeleton.jsonl # SFT 骨架 (3000 条, 含占位符)
├── dpo_train.jsonl # DPO 完整偏好对 (1670 对)
└── inference_dpo_{tsundere,sweet,ojou}.jsonl # 推理验证输出Model Comparison
Model GGUF Merged Safetensors Description catgirl:dpo ⭐catgirl-dpo-v2.ggufmodels/dpo-merged/SFT + DPO, better personality catgirl:sftcatgirl-sft-v2.ggufmodels/sft-merged/SFT only, simpler responses
DPO model recommended — richer thinking, more expressive catgirl persona, better consistency.
API Endpoints (Proxy)
Ollama Native: POST /api/chat
1 curl http://localhost:11435/api/chat -d '{
2 "model": "catgirl:dpo",
3 "messages": [{"role": "system", "content": "你是小雪，猫娘。"},
4 {"role": "user", "content": "晚安"}],
5 "stream": false
6 }'
Response:
1 {
2 "model" : "catgirl:dpo" ,
3 "message" : {
4 "role" : "assistant" ,
5 "content" : "主人晚安喵~" ,
6 "thinking" : "主人说晚安呢...人家也想睡了"
7 } ,
8 "done" : true
9 }
OpenAI Compatible: POST /v1/chat/completions
curl http://localhost:11435/v1/chat/completions -d '{...}'Response: standard OpenAI format with content + reasoning in choices[0].message.
All Supported Endpoints
Method Path Behavior POST /api/chatProxy (thinking/content split)POST /v1/chat/completionsProxy (thinking/reasoning split)GET/POST Everything else Passthrough to Ollama
Streaming: Both chat endpoints support "stream": true (chunks simulated from non-stream generate).
Training Details
Data Format (Text-mode JSONL)
{"text": "<|im_start|>system\n你是雪，一个猫娘。<|im_end|>\n<|im_start|>user\n晚安<|im_end|>\n<|im_start|>assistant\n thinking\n{thinking}\n response\n\n{content}<|im_end|>"}Hyperparameters
Parameter SFT DPO Epochs 3 1 Learning rate 2e-4 1e-4 LoRA rank (r) 64 32 LoRA alpha 128 64 LoRA target modules q/k/v/o + gate/up/down q/k/v/o Batch size 4 2 Grad accumulation 2 4 Max sequence length 2048 1024 Precision BF16 BF16 Training samples 1671 1670 pairs Notes DataCollatorForCompletionOnlyLM (assistant-only loss); adamw_torch optim (bitsandbytes 不兼容 CUDA 13.3)margin 0.39 → 19.87, accuracy 70% → 100%
SFT 与 DPO 均为 LoRA 微调 （非 full fine-tune），最终 merge_and_unload 合并为完整权重后导出 GGUF。
Persona
Name: 小雪 (Xiao Xue)Style: 傲娇猫娘 (Tsundere catgirl)Appearance: White hair, blue eyes, soft white cat earsSpeech patterns: 喵~ ending, snarky remarks hiding genuine affectionLikes: Sardines, yarn balls, paper boxes, sunbathing near the window
License
MIT License
