DeepSeek V4 Flash MLX Q2 Mixed
This is an MLX conversion of deepseek-ai/DeepSeek-V4-Flash.
Source
- Base model:
deepseek-ai/DeepSeek-V4-Flash
- Source revision:
6e763230a9d263eca2023f1d4a5ce1bfe126cf48
- Architecture:
DeepseekV4ForCausalLM
- Model type:
deepseek_v4
Conversion Recipe
- Tooling branch:
Thump604/mlx-lm, branch deepseek-v4-support-fixes
- Minimum tooling commit for generation:
9c990f4
- Output path during conversion:
/Volumes/Lexar/mlx_models/DeepSeek-V4-Flash-MLX-Q2-mixed-gs128-affine
- Quantization recipe:
mixed_2_6
- Quantization mode:
affine
- Group size:
128
- Effective bits per weight reported by MLX:
2.992
- Shards:
23
- Indexed MLX tensor size:
106,355,393,628 bytes
The mixed recipe uses 2-bit affine quantization for lower-risk routed expert paths and 6-bit affine quantization for sensitive paths including embeddings, LM head, attention projections, compressed-attention/indexer components, shared experts, and selected down projections.
Validation
- Conversion completed successfully.
- Lazy MLX load completed successfully on a 128GB Mac Studio.
- Raw prompt generation smoke completed successfully with
--max-tokens 2 --max-kv-size 1024.
- Observed smoke numbers:
54.59s real time, 74.5GB max RSS, 106.94GB peak footprint, zero swaps.
This artifact is a low-bit local fallback. It is not quality-qualified for production writing or coding lanes. Treat quality, long-context behavior, and sparse compressed-attention parity as open until evaluated with a real task suite.
Notes
DeepSeek V4 support in MLX is still under active development. This artifact was produced with local DeepSeek V4 support fixes, including FP4/FP8 checkpoint handling, F8_E8M0 scale metadata reinterpretation as raw uint8 exponent bytes before sanitizer decode, attention sink dtype handling, and quantized grouped output projection support.
