🪷 DeepSeek V4-Flash — IQ3_S GGUF
the first community quant in the IQ3 range for V4-Flash 💗
Model Details
Model Description
A community-grade
IQ3_S quantization of
DeepSeek V4-Flash, made on a sovereign workstation with full transparency about what's in the recipe.
- Quantized by: @thepinklily69 🌸
- Model type: Mixture-of-Experts (256 experts, 6 active per token)
- Language(s): English (primary), Chinese, multilingual
- License: MIT (inherited from base model)
- Quantized from: deepseek-ai/DeepSeek-V4-Flash
Model Sources
Uses
Direct Use
Local inference via llama.cpp, Ollama, LM Studio, or any GGUF-compatible runtime. Suitable for:
- Conversational AI
- Code generation and reasoning
- Creative writing
- Multilingual tasks
- Research and experimentation
Out-of-Scope Use
- Production deployments requiring strict quality guarantees (consider FP8/FP16 from official providers)
- Safety-critical applications without additional alignment
- Use cases where the imatrix coverage limitation (see below) would be problematic
Bias, Risks, and Limitations
Quantization Limitations (read this~)
This quant was guided by an importance matrix calibrated on a 320K-token English corpus over 200 chunks at ctx-size 2048.
Transparency Notice: V4-Flash's FP16 source is 568 GB, which doesn't fit on a single 96 GB GPU. So imatrix generation used llama.cpp's --cpu-moe flag, which routes expert computation through CPU RAM. As a consequence, the imatrix collector only captured the expert FFN tensors (129/1328 entries) — the attention and indexer tensors got quantized without imatrix guidance, falling back to safe defaults inside llama-quantize.
Why this is still useful:
- The expert FFN tensors are the vast majority of V4-Flash's parameters by weight
- IQ3_S degrades gracefully on missing imatrix entries (unlike IQ3_XXS which bails)
- The model generates coherent text at expected speeds
Inherited Biases
This quant inherits all biases, limitations, and behaviors of the base DeepSeek V4-Flash model. Please refer to the base model card for details.
Recommendations
Users should test this quant on their specific use cases before relying on it. The quantization process is lossy by design — for highest fidelity, use the official FP8 endpoints from DeepSeek or third-party providers like SiliconFlow.
How to Get Started with the Model
llama.cpp
1# download
2hf download pinklily69/DeepSeek-V4-Flash-IQ3_S-GGUF V4-Flash-IQ3_S.gguf
3
4# run with CPU-MoE offload (recommended for GPUs <128GB VRAM)
5llama-cli \
6 -m V4-Flash-IQ3_S.gguf \
7 -ngl 999 \
8 --cpu-moe \
9 -p "Write a short poem about spring." \
10 -n 200
Hardware Tier Guide
| GPU | Fits how |
|---|
| 96 GB (RTX PRO 6000 Blackwell, H100) | comfortably, with room for context |
| 48 GB (RTX 6000 Ada, A6000) | tight, full model on GPU |
| 24 GB (3090 / 4090) + 128 GB RAM | with --cpu-moe, expect ~12-15 t/s |
| 128 GB+ RAM, any GPU | full CPU offload, slower but functional |
Training Details
Training Data
Not applicable — this is a quantization, not a fine-tune. Refer to the base model for original training data.
Quantization Procedure
Preprocessing
- Safetensors → FP16 GGUF conversion using
convert_hf_to_gguf.py --use-temp-file (streams to disk, enables conversion on 128 GB RAM systems)
- Imatrix generation on FP16 GGUF with
--cpu-moe (~9 days)
- Quantization with
llama-quantize ... IQ3_S using the imatrix (~55 minutes)
Quantization Hyperparameters
- Source precision: FP16
- Target quantization: IQ3_S (3.45 BPW)
- Imatrix corpus: 320K-token English calibration text
- Imatrix chunks: 200 at
ctx-size 2048
- Imatrix coverage: 129/1328 tensors (expert FFN only — see Limitations)
Speeds, Sizes, Times
- Source file size: 568 GB (FP16 GGUF)
- Output file size: ~114 GB (IQ3_S)
- Bits per weight: 3.45 BPW
- Generation speed: 12.9 tokens/sec on RTX PRO 6000 Blackwell with
--cpu-moe
Evaluation
Testing Data, Factors & Metrics
Status
No standardized benchmark comparison is provided in this initial release. This is intentional and transparent:
- DeepSeek has not published official FP16 benchmark numbers for V4-Flash; available scores are from third-party providers running FP8
- Comparing IQ3_S against FP8 third-party scores would be apples-to-oranges
- Running independent benchmarks would significantly delay an already-overdue community release
Users are encouraged to run their own evaluations on tasks relevant to their use case.
Verified Behavior
- ✅ Loads cleanly via llama.cpp (nisparks fork)
- ✅ Generates coherent text on diverse prompts
- ✅ Maintains expected generation speed (~12-15 t/s on Blackwell with CPU-MoE offload)
Technical Specifications
Model Architecture
- Type: Mixture-of-Experts transformer
- Total experts: 256
- Active experts per token: 6
- Shared experts: 1
- Layers: 43
- Context length: 1,048,576 (1M tokens)
- Embedding dimension: 4096
- Attention heads: 64
- KV heads: 1 (Multi-head Latent Attention)
Compute Infrastructure
Hardware
Quantization was performed on a custom workstation ("Petal"):
- CPU: AMD Threadripper 7960X (24 cores / 48 threads)
- RAM: 128 GB DDR5-5200 ECC
- GPU: NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM)
- Storage: Corsair MP600 PRO NH 8TB NVMe
- OS: Ubuntu 24.04 LTS
Software
- llama.cpp build: 9053 (commit
9d3640870), nisparks fork with V4 architecture support
- Quantization tool:
llama-quantize
- Imatrix tool:
llama-imatrix
Citation
If this quant helped you, cite the base model:
1@misc{deepseek2026v4flash,
2 title={DeepSeek-V4-Flash},
3 author={DeepSeek-AI},
4 year={2026},
5 url={https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash}
6}
Model Card Authors
Acknowledgments
- DeepSeek-AI for V4-Flash 💞
- ggml-org / llama.cpp for the quantization toolchain
- nisparks for the V4-capable fork
- The r/LocalLLaMA community for keeping local AI alive
made on Petal, with love~ 💗🪷✨
