You can deploy this model and start earning money today!

AngelSlim/
Hy3-GPTQ-Int4

Name: Hy3-GPTQ-Int4
Availability: InStock

Views

No views yet

Dedicated to building a more intuitive, comprehensive, and efficient LLMs compression toolkit.

📖 Documentation | 🤗 Hugging Face | 🤖 ModelScope | 💬 WeChat

Hy3 GPTQ-Int4 Quantization

We use GPTQ 4-bit quantization to compress Hy3 to ~1/4 size with minimal accuracy loss. See the benchmark below:

Quickstart

vLLM

Build vLLM from source:

bash

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install --editable . --torch-backend=auto

Start the vLLM server:

bash

# Switch to trtllm backend to work-around mnnvl workspace size issue.
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
vllm serve AngelSlim/Hy3-GPTQ-Int4 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 2 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --port 8000 \
  --served-model-name hy3-gptq-int4