Quantized Qwen/Qwen3-Coder-Next using mlx-lm to 6-bit with group_size 128 for main weights and fine-grained group_size 64 for MoE weights, with the aim of maximum accuracy for 6bit quantization.

Evaluation Results

Testing with mlx_lm.evaluate using mmlu_pro with 50 Qs for each topic, comparing the 6-bit g128 quant with the 6-bit g64 quant:

Domain	g64	g128	Improvement
Math	0.90	0.94	+4%
Computer Science	0.82	0.86	+4%
Engineering	0.74	0.72	-2%
Physics	0.96	0.94	-2%
Chemistry	0.94	0.92	-2%

Average Performance

6-bit g64: 87.2% average
6-bit g128: 87.6% average
Improvement: +0.4 percentage points with group size 128 (main weights) with fine-grained 64 (MoE weights)

Full Quantization Spectrum Comparison

Domain	3-bit default	3-bit g128	4-bit default	4-bit g128	6-bit default	6-bit g128	8-bit default	8-bit g128
Math	0.94	0.90	0.92	0.92	0.90	0.94	0.92	0.94
Computer Science	0.82	0.84	0.80	0.84	0.82	0.86	0.84	0.90
Engineering	0.70	0.64	0.70	0.76	0.74	0.72	0.80	0.80
Physics	0.94	0.92	0.94	0.96	0.96	0.94	0.96	0.96
Chemistry	0.86	0.88	0.90	0.90	0.94	0.92	0.90	0.94
Average	0.852	0.836	0.835	0.865	0.872	0.876	0.868	0.888

Key Observations

Quantization Level Impact

3-bit:
- g64/default: 85.2% average
- g128: 83.6% average (-1.6% vs g64/default)
4-bit:
- g64/default: 83.5% average
- g128: 86.5% average (+3.0% vs g64/default)
6-bit:
- g64/default: 87.2% average
- g128: 87.6% average (+0.4% vs g64/default)
8-bit:
- g64/default: 86.8% average
- g128: 88.8% average (+3.6% vs g64/default)

Group Size Impact Across Quantization Levels

Quantization	g128 vs baseline	Performance Trend
3-bit	-1.6%	g128 underperforms default
4-bit	+3.0%	g128 significantly outperforms default
6-bit	+0.4%	g128 slightly outperforms default
8-bit	+3.6%	g128 significantly outperforms default

Model Details

Base Model: Qwen/Qwen3-Coder-Next
Library: mlx-lm
Quantization: 6-bit with group_size 128 for main weights and group_size 64 for MoE weights
License: apache-2.0
Pipeline Tag: text-generation

Usage

python

import mlx_lm
from mlx_lm.sample_utils import make_sampler

model_path = "petergilani/qwen3-coder-next-6bit-g128"
model, tokenizer = mlx_lm.load(model_path)

sampler = make_sampler(temp=1.0, top_p=0.95, top_k=40)

prompt = "Write a Python function to calculate the factorial of a number."
response = mlx_lm.generate(
    model, 
    tokenizer, 
    prompt=prompt,
    sampler=sampler,
    max_tokens=512
)
print(response)

petergilani/Qwen3-Coder-Next-6bit-g128

Model Card for Qwen3-Coder-Next-6bit-g128