You can deploy this model and start earning money today!

NeuroSenko/
Qwen3-235B-A22B-Instruct-2507-exl3

Name: Qwen3-235B-A22B-Instruct-2507-exl3
Availability: InStock

exllamav3

exl3

text-generation

Qwen/Qwen3-235B-A22B-Instruct-2507

quantized

apache-2.0

Views

No views yet

Quantization was performed using exllama3 v0.0.15.

Quant	Size (GB)	KL-div (quant, orig)	KL-div (orig, quant)	Perplexity	Top-K K=1	Top-K K=2	Top-K K=3	Top-K K=4	Top-K K=5
3.0bpw	84	0.16501465	0.20774904	4.08571518	0.8661	0.5977	0.3563	0.1950	0.1001
4.0bpw	111	0.04595388	0.04850649	3.72529002	0.9290	0.7547	0.5509	0.3743	0.2411
5.0bpw	138	0.01664260	0.01688963	3.67661701	0.9563	0.8392	0.6822	0.5256	0.3872
5.5bpw	151	0.01046010	0.01069444	3.65590399	0.9652	0.8699	0.7384	0.5977	0.4654
5.5bpw_opt	151	0.00949581	0.00957771	3.65956156	0.9665	0.8748	0.7451	0.6049	0.4731
6.0bpw	165	0.00739469	0.00749036	3.64918384	0.9705	0.8889	0.7720	0.6403	0.5140
6.23bpw	171	0.00564774	0.00568584	3.64926113	0.9735	0.8989	0.7912	0.6696	0.5487
6.23bpw_opt	171	0.00743874	0.00743475	3.65749682	0.9702	0.8866	0.7671	0.6351	0.5084
7.0bpw	192	0.00429302	0.00428297	3.64699539	0.9782	0.9143	0.8197	0.7095	0.5975
original	437	-	-	3.64520522	-	-	-	-	-

The 5.5bpw-opt and 6.23bpw-opt quantizations were created using a combination of the 5.0bpw, 6.0bpw, and 7.0bpw measurements to optimize layer-wise bit allocation.

Full measurements data: json

Metrics explanation

KL-divergence: Measures the difference between probability distributions of quantized and original models. Lower is better (closer to original).
Perplexity: Indicates how well the model predicts the next token. Lower values mean better prediction quality.
Top-K agreement: Shows how often the quantized model's top-K predictions exactly match the original model's predictions. Higher values indicate better preservation of the original model's behavior (1.0 = perfect match):
- K=1: The fraction of times both models predict the exact same token as their top-1 choice
- K=5: The fraction of times both models have the same 5 tokens in their top-5 predictions in the exact same ranking order

Tool Calls Support for Qwen/GLM Models

The official tabbyAPI doesn't support tool calls for Qwen and GLM models yet.

If you're using Qwen-Code, OpenClaw, or similar software that need tool call support, you can use my fork with the tools-support branch:

Clone directly:

git clone -b tools-support https://github.com/NeuroSenko/tabbyAPI

Or add to existing tabbyAPI installation:

bash

git remote add neurosenko https://github.com/NeuroSenko/tabbyAPI
git fetch neurosenko
git checkout -b tools-support neurosenko/tools-support

This branch includes native tool calling support for Qwen and GLM model families.

NeuroSenko/Qwen3-235B-A22B-Instruct-2507-exl3

Metrics explanation

Tool Calls Support for Qwen/GLM Models

NeuroSenko/
Qwen3-235B-A22B-Instruct-2507-exl3