File	Quant	Size	Notes
`AJ-DeepSeekR1Qwen32B-v2.1.0-Q4_K_M.gguf`	Q4_K_M	~19 GB	Recommended for 24 GB GPUs (RTX 4090). Good quality / speed.
`AJ-DeepSeekR1Qwen32B-v2.1.0-Q8_0.gguf`	Q8_0	~33 GB	Near-lossless. Needs 40+ GB VRAM or CPU offload.

Training

Base: deepseek-ai/DeepSeek-R1-Distill-Qwen-32B (32B, BF16)
Method: LoRA (r=64, α=128, dropout=0.05), BF16 compute
Hardware: 1× NVIDIA H200 (141 GB)
Data: mixed_v1 (200K train / 2K eval)
Final: step 5000, train loss 0.956, eval loss 0.715, tok-acc 80.9%
See companion repo ianwesterfield/AJ-DeepSeekR1Qwen32B-v2.1.0-lora for the raw adapter.

Usage with llama.cpp

bash

llama-server -m AJ-DeepSeekR1Qwen32B-v2.1.0-Q4_K_M.gguf \
    --host 0.0.0.0 --port 8081 \
    --n-gpu-layers 65 --ctx-size 8192

Chat template

DeepSeek-R1 distill uses the DeepSeek chat template (<｜User｜> / <｜Assistant｜> / <｜end▁of▁sentence｜>). llama-server picks this up from the embedded tokenizer metadata; no manual template required.

ianwesterfield/AJ-DeepSeekR1Qwen32B-v2.1.0-GGUF

AJ-DeepSeekR1Qwen32B-v2.1.0 — GGUF

Files

Training

Usage with llama.cpp

Chat template

ianwesterfield/
AJ-DeepSeekR1Qwen32B-v2.1.0-GGUF