Views
No views yet
IQ1_M is based on recipes defined via the --tensor-type option.IQ1_S is a more dynamic version intended for extreme compression.Q2_K_L is a specialized version with only 2/4/8 bit quant designed for personalized deployment and experiments.- IQ1_S : 137.66 GiB (1.76 BPW)
- IQ1_M : 151.25 GiB (1.94 BPW)
- Q2_K_L : 210.60 GiB (2.70 BPW)1# !pip install huggingface_hub hf_transfer
2import os
3# os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
4from huggingface_hub import snapshot_download
5snapshot_download(
6 repo_id = "moxin-org/DeepSeek-V3-0324-Moxin-GGUF",
7 local_dir = "DeepSeek-V3-0324-Moxin-GGUF",
8 allow_patterns = ["*IQ1_M*"], # Q2_K_L, IQ1_S, Mini
9)Download Available for huggingface_hub, huggingface-cli, snapshot_download, xet.
| Benchmark (Metric) | llamacpp IQ1_M (140G) | llamacpp Q2_K (230G) | Ours IQ1_S (138G) | Ours IQ1_M (152G) |
|---|---|---|---|---|
| Winogrande | 73.00 | 77.74 | 78.69 | 79.48 |
| MMLU (EM) | 75.45 | 85.57 | 85.42 | 86.07 |
| CMMLU | 77.06 | 82.57 | 86.65 | 87.84 |
| Hellaswag | 78.70 | 86.46 | 85.39 | 85.94 |
| gsm8k | 83.40 | 93.40 | 93.93 | 94.39 |
| BBH | 24.68 | 69.19 | 84.95 | 86.87 |
Note: All models use MoE architecture with 37B activated and 671B total parameters.
Bold values mark the best performance per benchmark.
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DBUILD_SHARED_LIBS=OFF -DLLAMA_CURL=OFF
cmake --build build --config Release -j --clean-firstbuild/bin/llama-cli -m DeepSeek-V3-0324-Moxin-GGUF/V3-IQ1_M/DeepSeek-V3-0324-Moxin-IQ1_M-00001-of-00006.gguf \
-ngl 99 \
--temp 0.3 \
--min-p 0.01 \
--ctx-size 8192 \ # 4096, 16384105.58 GiB (1.79 BPW). Please refer to
tflsxyy/DeepSeek-V3-0324-E192
and V3-Mini-Exp
for more details.1@article{chen2025collaborative,
2 title={Collaborative Compression for Large-Scale MoE Deployment on Edge},
3 author={Chen, Yixiao and Xie, Yanyue and Yang, Ruining and Jiang, Wei and Wang, Wei and He, Yong and Chen, Yue and Zhao, Pu and Wang, Yanzhi},
4 journal={arXiv preprint arXiv:2509.25689},
5 year={2025}
6}