Asymmetric quantization of sailing-lab/SR2AM-v1.0-30B (Qwen3-MoE, 48 layers, 128 experts, top-8) using the strategy popularized by antirez/ds4: aggressively quantize the bulky routed-expert FFN tensors and keep the rest at higher precision so the whole model fits on a 24 GB consumer GPU at full speed.

Quantization recipe

Tensor family	Type
`ffn_up_exps.weight` (routed)	IQ2_XXS
`ffn_gate_exps.weight` (routed)	IQ2_XXS
`ffn_down_exps.weight` (routed)	Q2_K
attention proj (`attn_q/k/v/output`)	Q6_K
shared/non-expert FFN, norms, router	Q6_K
`token_embd.weight`	Q8_0
`output.weight`	Q6_K

Imatrix calibration: 100 chunks × 512 tokens from bartowski calibration_datav3.

Tool: llama.cpp llama-quantize with --imatrix + --tensor-type overrides.

File

File	Size	BPW
`SR2AM-v1.0-30B-IQ2XXS-asym.gguf`	8.9 GB	2.50

vs. source BF16 GGUF (58.3 GB, 16.01 BPW) → 6.5× compression.

Runtime (RTX 3090 24 GB, full GPU offload)

Setting	Value
VRAM (with 40k ctx)	13.9 GB / 24 GB
Prefill	420.6 t/s
Decode	168.3 t/s

Quality

Benchmark	Score	Notes
MATH500 (subset N=30)	24/30 = 80.0 %	Qwen-spec sampling (see below). 0 truncations at the run config.

Comparable to published Qwen3-30B-A3B-Thinking-2507 BF16 numbers on MATH500 (~85–90 %); ~5–10 point delta at 2.50 BPW is in line with what asymmetric MoE quants achieve on reasoning benchmarks.

Usage

This is a thinking-mode model. Use generous output budgets and Qwen3 official sampling, otherwise reasoning chains get truncated and the model emits empty content.

bash

llama-server \
  -m SR2AM-v1.0-30B-IQ2XXS-asym.gguf \
  -ngl 99 \
  -c 40960 \
  --jinja

Request body for evals / hard reasoning prompts:

json

{
  "messages": [...],
  "max_tokens": 32768,
  "temperature": 0.6,
  "top_p": 0.95,
  "top_k": 20,
  "min_p": 0.0,
  "presence_penalty": 1.0
}

Use a recent llama.cpp build with IQ2_XXS CUDA kernels (tested on b9124).

⚠️ Do not set temperature=0 — off-spec for Qwen3 thinking models and triggers degenerate long chains. Do not rely on chat_template_kwargs.enable_thinking=false — broken in current llama.cpp (#20182, #13189).

Credits

Base model: sailing-lab/SR2AM-v1.0-30B
Asymmetric-quant strategy: antirez/ds4
Calibration data: bartowski1182
Tooling: llama.cpp

dusterbloom/SR2AM-v1.0-30B-IQ2XXS-asym-GGUF

SR2AM-v1.0-30B — Asymmetric IQ2_XXS/Q2_K GGUF