You can deploy this model and start earning money today!

g023/
Qwen3-8B-DMS-8x-4bit-NF4

Name: Qwen3-8B-DMS-8x-4bit-NF4
Availability: InStock

Views

No views yet

Qwen3-8B-DMS-8x-4bit-NF4

This repository contains a 4-bit quantized version of the Qwen3-8B-DMS-8x model.

Model Description

The Qwen3-8B-DMS-8x is an autoregressive language model based on the Qwen3-8B architecture, enhanced with Dynamic Memory Sparsification (DMS). DMS is a state-of-the-art technique that allows for an 8x compression ratio of the KV cache during inference by using learned, per-head eviction policies. This significantly reduces memory usage and increases throughput for long-context generation.

This 4-bit quantization further reduces the VRAM requirements, making it possible to run this high-performance DMS model on consumer GPUs with limited memory.

Original Model: https://huggingface.co/nvidia/Qwen3-8B-DMS-8x
Architecture: Qwen3-8B with Dynamic Memory Sparsification
Quantization: 4-bit NF4 with double quantization (BitsAndBytes)
Compression Level: 4-bit weights + 8x KV cache compression (via DMS)
Parameters: 8.2 billion
Context Length: Up to 131,072 tokens with YaRN

Quantization Method

Methodology: BitsAndBytes NF4 (Normal Float 4-bit) with double quantization
Bits: 4-bit
Compute Dtype: bfloat16
Storage: uint8
Calibration: Default BitsAndBytes quantization (no custom dataset)

Usage

SEE: 'load_and_test.py' for an inferencing example.

Because this model uses custom Dynamic Memory Sparsification logic, you must set trust_remote_code=True to load the specialized configuration and modeling files.

Limitations and License

https://huggingface.co/nvidia/Qwen3-8B-DMS-8x/blob/main/LICENSE

Attribution and Citation

Model Page: https://huggingface.co/nvidia/Qwen3-8B-DMS-8x
Reference Paper: Łańcucki et al., "Inference-Time Hyper-Scaling with KV Cache Compression," 2025.
Paper Link: https://huggingface.co/papers/2506.05345
BibTeX:
bibtex
1@misc{lancucki2025inferencetime, 2 title={Inference-Time Hyper-Scaling with KV Cache Compression}, 3 author={Adrian Łańcucki and Konrad Staniszewski and Piotr Nawrot and Edoardo M. Ponti}, 4 year={2025}, 5 eprint={2506.05345}, 6 archivePrefix={arXiv}, 7 primaryClass={cs.LG}, 8 url={https://arxiv.org/abs/2506.05345}, 9}

g023/Qwen3-8B-DMS-8x-4bit-NF4