You can deploy this model and start earning money today!

CPU-Hybrid-MoE/
MiniMax-M2.7-CPU-NUMA4-AMXINT8

Name: MiniMax-M2.7-CPU-NUMA4-AMXINT8
Availability: InStock

minimax_m2

text-generation

custom_code

MiniMaxAI/MiniMax-M2.7

quantized

fp8

Views

No views yet

MiniMax-M2.7-CPU-NUMA4-AMXINT8

MiniMaxAI/MiniMax-M2.7 quantized to the AMXINT8 format for inference with sglang + ktransformers, packed specifically for inference on 4 NUMA nodes.

To run, please ensure that your CPU supports the AMX instruction set (Intel Xeon processor, Sapphire Rapids or newer), and make note of your NUMA node count. Install kt-kernal and sglang-kt following the official documentation.

Then, download the official weights of MiniMaxAI/MiniMax-M2.7 in FP8, as well as this CPU-optimized quantized model, and prepare your launch command:

PYTORCH_ALLOC_CONF=expandable_segments:True \
SGLANG_ENABLE_JIT_DEEPGEMM=0 \
python -m sglang.launch_server \
  --model /path/to/MiniMax-M2.7 \
  --kt-method AMXINT8 \
  --kt-weight-path /path/to/MiniMax-M2.7-CPU-NUMA4-AMXINT8 \
  --kt-cpuinfer 128 \
  --kt-threadpool-count 4 \
  --kt-num-gpu-experts 64 \
  --kt-max-deferred-experts-per-token 0 \
  --kt-expert-placement-strategy uniform \
  --trust-remote-code \
  --mem-fraction-static 0.98 \
  --served-model-name MiniMaxAI/MiniMax-M2.7 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --enable-p2p-check \
  --disable-shared-experts-fusion \
  --chunked-prefill-size 4096 \
  --context-length 131072 \
  --max-total-tokens 131072 \
  --max-running-requests 1 \
  --attention-backend flashinfer \
  --fp8-gemm-backend cutlass \
  --reasoning-parser minimax \
  --tool-call-parser minimax-m2

Notes:

At the time of writing, MiniMaxM2ForCausalLM produces significantly degraded output with transformers 5.0.0+
--kt-cpuinfer should be set to the total number of physical CPU cores across all NUMA nodes
--tensor-parallel-size 1 should be set to the number of GPUs
The optimal choices for --attention-backend and --fp8-gemm-backend depend on the CUDA architecture of your GPUs - please check the sglang documentation
--kt-num-gpu-experts, --mem-fraction-static, --chunked-prefill-size, --context-length, --max-total-tokens, and --max-running-requests should be adjusted depending on constraints of your hardware
Please review the official kt-kernel documentation for details

CPU-Hybrid-MoE/MiniMax-M2.7-CPU-NUMA4-AMXINT8

MiniMax-M2.7-CPU-NUMA4-AMXINT8

Notes:

CPU-Hybrid-MoE/
MiniMax-M2.7-CPU-NUMA4-AMXINT8