These models were obtained by quantizing the weights and activations of DeepSeek models to mixed-precision data types (W4(int)A(FP)8 for MoE layers and FP8 for dense layers).
This optimization reduces the number of bits per parameter 4/8, significantly reducing GPU memory requirements.
Use with SGLANG
This model can be deployed efficiently using the SGLANG backend with only H200x4, as shown in the example below.
