Qwen3.5-9B is Alibaba Cloud's latest-generation multimodal foundation model, featuring a hybrid architecture that combines Gated Delta Networks (linear attention) with standard full attention in a 3:1 pattern. It is natively multimodal — trained from scratch on text, images, and video through early fusion of multimodal tokens, not bolted-on adapters.
This quantization preserves the full hybrid architecture with precision-aware layer treatment, applying EoRA rank-64 error correction across all quantizable layers for maximum quality retention.
TevunahAi Multi-Level GPTQ with EoRA rank-64 applied to ALL quantizable layers. No MoE routing in the 9B variant means EoRA is affordable across the full model, providing comprehensive error correction.
Formal benchmarks pending — inference quality verified manually.
1from gptqmodel import GPTQModel
2from transformers import AutoTokenizer
3
4model = GPTQModel.load(
5 "TevunahAi/Qwen3.5-9B-TevunahAi-GPTQ",
6 trust_remote_code=True
7)
8tokenizer = AutoTokenizer.from_pretrained(
9 "TevunahAi/Qwen3.5-9B-TevunahAi-GPTQ",
10 trust_remote_code=True
11)
12
13# Generate
14prompt = "Explain the difference between linear and quadratic attention complexity."
15output = model.generate(
16 **tokenizer(prompt, return_tensors='pt').to('cuda'),
17 max_new_tokens=256
18)
19print(tokenizer.decode(output[0]))
1messages = [{"role": "user", "content": "Solve: What is the integral of x²·sin(x)?"}]
2
3tokenized = tokenizer.apply_chat_template(
4 messages,
5 tokenize=True,
6 add_generation_prompt=True,
7 return_tensors="pt"
8).to('cuda')
9
10outputs = model.generate(
11 tokenized,
12 max_new_tokens=2048,
13 temperature=1.0,
14 top_p=0.95,
15 top_k=20,
16 presence_penalty=1.5
17)
18print(tokenizer.decode(outputs[0], skip_special_tokens=True))
1tokenized = tokenizer.apply_chat_template(
2 messages,
3 tokenize=True,
4 enable_thinking=False,
5 add_generation_prompt=True,
6 return_tensors="pt"
7).to('cuda')
8
9outputs = model.generate(
10 tokenized,
11 max_new_tokens=128,
12 do_sample=False,
13 num_beams=1
14)
1pip install -U "vllm>=0.12.0"
2
3vllm serve TevunahAi/Qwen3.5-9B-TevunahAi-GPTQ \
4 --max-num-seqs 8 \
5 --tensor-parallel-size 1 \
6 --max-model-len 32768 \
7 --trust-remote-code
1@software{qwen35_9b_gptq_2026,
2 title = {Qwen3.5-9B - TevunahAi Multi-Level GPTQ},
3 author = {TevunahAi},
4 year = {2026},
5 note = {Multi-Level GPTQ with EoRA rank-64 for hybrid Gated DeltaNet + Attention architecture},
6 url = {https://huggingface.co/TevunahAi/Qwen3.5-9B-TevunahAi-GPTQ}
7}
8
9@misc{qwen35_2026,
10 title = {Qwen3.5 Technical Report},
11 author = {Qwen Team, Alibaba Cloud},
12 year = {2026},
13 url = {https://qwenlm.github.io/blog/qwen3.5/}
14}
15
16@article{liu2024eora,
17 title = {EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation},
18 author = {Liu, Shih-Yang and Wang, Huck Yang and Cheng, Hong-Yi Michael and Khailany, Brucek and Molchanov, Pavlo},
19 journal = {arXiv preprint arXiv:2410.21271},
20 year = {2024},
21 url = {https://arxiv.org/abs/2410.21271},
22 note = {NVIDIA Research}
23}
This quantization leverages the hybrid Gated DeltaNet + Full Attention architecture, requiring precision-aware treatment of fundamentally different layer types (linear vs quadratic attention). EoRA rank-64 error correction is applied comprehensively across all quantizable layers to maximize quality retention at aggressive compression ratios.
