GGUF Q2_K_XL checkpoint of Qwen/Qwen3.5-4B in which the discrete grid assignments have been refined with GSQ (Gumbel-Softmax Quantization), starting from the public Unsloth GGUF initialization and projected back into the same K-Quant format.

The optimized file runs unchanged on llama.cpp / Ollama and is a drop-in replacement for the corresponding standard GGUF checkpoint.

Paper: GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (arXiv:2604.18556)
Paper page on HF: https://huggingface.co/papers/2604.18556
Code: https://github.com/IST-DASLab/GSQ
Collection: https://huggingface.co/collections/ISTA-DASLab/gsq

Quantization details

Base model: Qwen/Qwen3.5-4B
Starting point: Unsloth GGUF UD-Q2_K_XL
Released quantization: Q2_K_XL
Format: GGUF K-Quant
Pipeline: Unsloth GGUF init → GSQ Gumbel-Softmax refinement → re-pack into the same format
Runtime: llama.cpp, ollama, LM Studio, anything that consumes GGUF

Storage layout

This file is bit-for-bit standard GGUF Q2_K. GSQ only changes the values of the quantized weights by relearning the discrete grid assignments inside each K-Quant block.

As a result:

The file size matches the corresponding upstream Q2_K_XL file for the same model.
Any llama.cpp / ollama build that loads a regular Qwen3.5-4B Q2_K_XL GGUF should load this file with zero changes.
The Hugging Face UI reports the GGUF block type, e.g. Q2_K_XL, rather than per-tensor dtypes. This refers to the on-disk K-Quant encoding, not the precision of any optimizer state used during GSQ refinement.

Evaluation

We compare the GSQ-refined checkpoint against the Unsloth UD-Q2_K_XL checkpoint used as initialization.

Model	AIME 25	GPQA Diamond	IFEval	GSM8K	MMLU-Pro
Unsloth `UD-Q2_K_XL`	26.67	56.06	76.14	79.61	66.78
GSQ-refined `Q2_K_XL`	60.00	67.17	82.97	88.48	70.87

Evaluation notes

All evaluations are zeroshot. The Unsloth UD-Q2_K_XL model was evaluated with thinking disabled on all tasks, because its thinking mode was found to be broken and produced worse results.

For the GSQ-refined model:

AIME 25 and GPQA Diamond were evaluated with thinking enabled.
IFEval, GSM8K, and MMLU-Pro were evaluated with thinking disabled.

Token Efficiency

For MMLU-Pro, both models were evaluated with a maximum output budget of 32K tokens.

The Unsloth model consumed 55.32M output tokens in total.
The GSQ-refined model consumed 41.82M output tokens in total.

Usage with llama.cpp

bash

hf download ISTA-DASLab/Qwen3.5-4B-GGUF-GSQ \
  Qwen3.5-4B-Q2_K_XL.gguf --local-dir .

./llama-cli -m Qwen3.5-4B-Q2_K_XL.gguf -p "Hello"