Hy3 is a 295B-parameter MoE model with 21B active parameters, 192 experts (top-8 routed), and a 256K context window. Quantizing it to 4-bit brings the weight footprint down to ~150 GB, making it possible to serve the full 256K context on a single 8x24 GB node with KVarN KV cache compression.

⚠️ Required vLLM Branch

This model requires a patched vLLM build. Stock vLLM cannot load it due to several bugs in weight-name remapping, dynamic GPTQ MoE group-size handling, and TurboQuant KV cache shaping:

➡️ avtc/vllm @ fix/load-quantized-hy3-mimo-v2.5-kvarn-0.26

The branch includes these fixes:

Commit	Fix
`a1ca5aff6`	Remap `e_score_correction_bias`→`expert_bias` for transformers 5.x checkpoint renames
`80437c75e`	Respect dynamic `group_size=64` override for GPTQ MoE Marlin support + WNA16 fallback
`c2905928f`	Fix `shared_mlp` weight prefix so shared-expert weights load correctly
`4e3126f5f`	Use `WeightsMapper` for HY3 transformers weight renames + quant detection
`325a5ddd7`	Disable TurboQuant boundary skip layers (mixed-backend KV cache allocation crash)
`33a482b71`	Fix `layer_cache_dtype_str` for TurboQuant in KV cache reshape (`TQFullAttentionSpec` has `kv_quant_mode=NONE`)

fixes to allow run AWQ/GPTQ quants of Mimo-v2.5
chery picked commit: [Attention] Add KVarN: calibration-free variance-normalized KV-cache quantization backend from https://github.com/vllm-project/vllm/pull/46812

Build from source:

bash

git clone -b fix/load-quantized-hy3-mimo-v2.5-kvarn-0.26 https://github.com/avtc/vllm.git
cd vllm
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
TORCH_CUDA_ARCH_LIST="8.6" uv pip install --editable . --torch-backend=auto

Set TORCH_CUDA_ARCH_LIST to your GPU's compute capability to skip unused architectures and speed up the build (e.g. 8.6 for RTX 3090, 8.9 for RTX 4090, 9.0 for H100).

Quantization Details

Quantized with GPTQModel (7.2.0) using the RTN (Round-To-Nearest) fallback strategy.

Property	Value
Quant method	GPTQ (RTN fallback)
Bits	4
Symmetric	`true`
`desc_act`	`false`
Global group size	128 (dense layers, attention projections)
Expert / shared-MLP group size	64 (dynamic override)
Fallback threshold	0.5%
Pack dtype	int32

Verified Hardware & Performance

This model is verified to run with Tensor Parallel on 8x NVIDIA RTX 3090 (24 GB each) GPUs with the full 256K (262,144) context window, using kvarn_k4v2_g128 4-bit K, 2-bit V near lossless cache compression.

Tensor parallel: 8 (no expert parallel — expert group size 64 divides cleanly across 8 GPUs)
KV cache: kvarn_k4v2_g128
Max model len: 262,144
Model weight footprint: ~150 GB
Reasoning (<think>) and tool-calling work.

Quick Start

bash

export VLLM_SLEEP_WHEN_IDLE=1
export TORCH_CUDA_ARCH_LIST="8.6"
export CUDA_VISIBLE_DEVICES=4,5,6,7,0,1,2,3
export RAY_memory_monitor_refresh_ms=0
export NCCL_CUMEM_ENABLE=0
export VLLM_ENABLE_CUDAGRAPH_GC=1

export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export VLLM_FLOAT32_MATMUL_PRECISION=high
export OMP_NUM_THREADS=1

export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm

vllm serve /path/to/tencent-Hy3-RTN \
  -tp 8 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3 \
  --port 8000 \
  --host 0.0.0.0 \
  --uvicorn-log-level info \
  --trust-remote-code \
  --max-num-seqs 4 \
  --seed 1234 \
  --max-model-len auto \
  --skip-mm-profiling \
  -O3 \
  --no-use-tqdm-on-load \
  --default-chat-template-kwargs '{"interleaved_thinking": true}' \
  --performance-mode balanced \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048 \
  --enable-prefix-caching \
  --kv-cache-dtype kvarn_k4v2_g128 \
  --gpu-memory-utilization 0.945

Then call the OpenAI-compatible API:

python

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="hy3",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
    # reasoning_effort: "no_think" (default, direct response),
    #                   "low", "high" (deep chain-of-thought)
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)

Recommended sampling parameters: temperature=0.9, top_p=1.0.

Reasoning mode: Set reasoning_effort to "high" for complex tasks (math, coding, reasoning), "low" for faster thinking or "no_think" for direct responses.

Verified Functionality

✅ Functional 2D HTML aquarium generation (without reasoning enabled) — working artifact.
✅ Reasoning mode (<think>) engages and produces output.

Original Model

For full details on the architecture, benchmarks, and capabilities of the base model, see the original model card: ➡️ tencent/Hy3

Property	Value
Architecture	Mixture-of-Experts (MoE)
Total Parameters	295B
Activated Parameters	21B
Number of Layers	80 (MTP layer was removed during quantization)
Attention Heads	64 (GQA, 8 KV heads, head dim 128)
Hidden Size	4096
Intermediate Size	13312 (dense), 1536 (expert)
Context Length	256K
Vocabulary Size	120,832
Number of Experts	192, top-8 activated
License	Apache 2.0

Acknowledgments

Tencent Hunyuan Team for the original Hy3 model.
GPTQModel team for the quantization toolkit.
vLLM team for the inference engine.
KVarN team for abilty to use large near-lossless KV cache

License

This model inherits the Apache License 2.0 of the base model. See the original license.

Attribution

@misc{muller2026kvarn,
      title={KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks}, 
      author={Lorenz K. Muller and Philippe Bich and Chiara Boretti and Hyun-Min Chang and Jiawei Zhuang and Lukas Cavigelli},
      year={2026},
      eprint={2606.03458},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={http://arxiv.org/abs/2606.03458}
}

avtc/Hy3-GPTQ-RTN-4bit-tp8

Tencent Hy3 GPTQ RTN 4bit tp8 (MPT layer removed)