You can deploy this model and start earning money today!

bullerwins/
Hunyuan-A13B-Instruct-GGUF

Name: Hunyuan-A13B-Instruct-GGUF
Availability: InStock

quantized

tencent/Hunyuan-A13B-Instruct

Views

No views yet

Update July 8th 2025: The support has been merged to llama.cpp, the quants currently on the repo work with the latest commit as of today. No longer experimental

Update July 1st 2025: Reuploaded quants with fixed rope. Should work better now. Still need the PR to work

This is an experimental upload for the current PR draft for the support of the Hunyuan model. Intented for testing purpuses.

You need this PR for this to work: https://github.com/ggml-org/llama.cpp/pull/14425

🤗 Hugging Face | 🖥️ Official Website | 🕖 HunyuanAPI | 🕹️ Demo | 🤖 ModelScope

Technical Report | GITHUB | cnb.cool | LICENSE

Welcome to the official repository of Hunyuan-A13B, an innovative and open-source large language model (LLM) built on a fine-grained Mixture-of-Experts (MoE) architecture. Designed for efficiency and scalability, Hunyuan-A13B delivers cutting-edge performance with minimal computational overhead, making it an ideal choice for advanced reasoning and general-purpose applications, especially in resource-constrained environments.

Model Introduction

With the rapid advancement of artificial intelligence technology, large language models (LLMs) have achieved remarkable progress in natural language processing, computer vision, and scientific tasks. However, as model scales continue to expand, optimizing resource consumption while maintaining high performance has become a critical challenge. To address this, we have explored Mixture of Experts (MoE) architectures. The newly introduced Hunyuan-A13B model features a total of 80 billion parameters with 13 billion active parameters. It not only delivers high-performance results but also achieves optimal resource efficiency, successfully balancing computational power and resource utilization.

Key Features and Advantages

Compact yet Powerful: With only 13 billion active parameters (out of a total of 80 billion), the model delivers competitive performance on a wide range of benchmark tasks, rivaling much larger models.
Hybrid Reasoning Support: Supports both fast and slow thinking modes, allowing users to flexibly choose according to their needs.
Ultra-Long Context Understanding: Natively supports a 256K context window, maintaining stable performance on long-text tasks.
Enhanced Agent Capabilities: Optimized for agent tasks, achieving leading results on benchmarks such as BFCL-v3, τ-Bench and C3-Bench.
Efficient Inference: Utilizes Grouped Query Attention (GQA) and supports multiple quantization formats, enabling highly efficient inference.

Why Choose Hunyuan-A13B?

As a powerful yet computationally efficient large model, Hunyuan-A13B is an ideal choice for researchers and developers seeking high performance under resource constraints. Whether for academic research, cost-effective AI solution development, or innovative application exploration, this model provides a robust foundation for advancement.

Related News

2025.6.27 We have open-sourced Hunyuan-A13B-Pretrain , Hunyuan-A13B-Instruct , Hunyuan-A13B-Instruct-FP8 , Hunyuan-A13B-Instruct-GPTQ-Int4 on Hugging Face. In addition, we have released a technical report and a training and inference operation manual, which provide detailed information about the model’s capabilities as well as the operations for training and inference.

Benchmark

Note: The following benchmarks are evaluated by TRT-LLM-backend on several base models.

Model	Hunyuan-Large	Qwen2.5-72B	Qwen3-A22B	Hunyuan-A13B
MMLU	88.40	86.10	87.81	88.17
MMLU-Pro	60.20	58.10	68.18	67.23
MMLU-Redux	87.47	83.90	87.40	87.67
BBH	86.30	85.80	88.87	87.56
SuperGPQA	38.90	36.20	44.06	41.32
EvalPlus	75.69	65.93	77.60	78.64
MultiPL-E	59.13	60.50	65.94	69.33
MBPP	72.60	76.00	81.40	83.86
CRUX-I	57.00	57.63	-	70.13
CRUX-O	60.63	66.20	79.00	77.00
MATH	69.80	62.12	71.84	72.35
CMATH	91.30	84.80	-	91.17
GSM8k	92.80	91.50	94.39	91.83
GPQA	25.18	45.90	47.47	49.12

Hunyuan-A13B-Instruct has achieved highly competitive performance across multiple benchmarks, particularly in mathematics, science, agent domains, and more. We compared it with several powerful models, and the results are shown below.

Topic	Bench	OpenAI-o1-1217	DeepSeek R1	Qwen3-A22B	Hunyuan-A13B-Instruct
Mathematics	AIME 2024 AIME 2025 MATH	74.3 79.2 96.4	79.8 70 94.9	85.7 81.5 94.0	87.3 76.8 94.3
Science	GPQA-Diamond OlympiadBench	78 83.1	71.5 82.4	71.1 85.7	71.2 82.7
Coding	Livecodebench Fullstackbench ArtifactsBench	63.9 64.6 38.6	65.9 71.6 44.6	70.7 65.6 44.6	63.9 67.8 43
Reasoning	BBH DROP ZebraLogic	80.4 90.2 81	83.7 92.2 78.7	88.9 90.3 80.3	89.1 91.1 84.7
Instruction Following	IF-Eval SysBench	91.8 82.5	88.3 77.7	83.4 74.2	84.7 76.1
Text Creation	LengthCtrl InsCtrl	60.1 74.8	55.9 69	53.3 73.7	55.4 71.9
NLU	ComplexNLU Word-Task	64.7 67.1	64.5 76.3	59.8 56.4	61.2 62.9
Agent	BFCL v3 τ-Bench ComplexFuncBench C3-Bench	67.8 60.4 47.6 58.8	56.9 43.8 41.1 55.3	70.8 44.6 40.6 51.7	78.3 54.7 61.2 63.5

Use with transformers

Our model defaults to using slow-thinking reasoning, and there are two ways to disable CoT reasoning.

Pass "enable_thinking=False" when calling apply_chat_template.
Adding "/no_think" before the prompt will force the model not to use perform CoT reasoning. Similarly, adding "/think" before the prompt will force the model to perform CoT reasoning.

The following code snippet shows how to use the transformers library to load and apply the model. It also demonstrates how to enable and disable the reasoning mode , and how to parse the reasoning process along with the final output.

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import os
import re

model_name_or_path = os.environ['MODEL_PATH']
# model_name_or_path = "tencent/Hunyuan-A13B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto",trust_remote_code=True)  # You may want to use bfloat16 and/or move to GPU here
messages = [
    {"role": "user", "content": "Write a short summary of the benefits of regular exercise"},
]
tokenized_chat = tokenizer.apply_chat_template(messages, tokenize=True, return_tensors="pt",
                                                enable_thinking=True # Toggle thinking mode (default: True)
                                                )
                                                
outputs = model.generate(tokenized_chat.to(model.device), max_new_tokens=4096)

output_text = tokenizer.decode(outputs[0])

think_pattern = r'<think>(.*?)</think>'
think_matches = re.findall(think_pattern, output_text, re.DOTALL)

answer_pattern = r'<answer>(.*?)</answer>'
answer_matches = re.findall(answer_pattern, output_text, re.DOTALL)

think_content = [match.strip() for match in think_matches][0]
answer_content = [match.strip() for match in answer_matches][0]
print(f"thinking_content:{think_content}\n\n")
print(f"answer_content:{answer_content}\n\n")

Fast and slow thinking switch

This model supports two modes of operation:

Slow Thinking Mode (Default): Enables detailed internal reasoning steps before producing the final answer.
Fast Thinking Mode: Skips the internal reasoning process for faster inference, going straight to the final answer.

Switching to Fast Thinking Mode:

To disable the reasoning process, set enable_thinking=False in the apply_chat_template call:

tokenized_chat = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    enable_thinking=False  # Use fast thinking mode
)

Deployment

For deployment, you can use frameworks such as TensorRT-LLM, vLLM, or SGLang to serve the model and create an OpenAI-compatible API endpoint.

image: https://hub.docker.com/r/hunyuaninfer/hunyuan-a13b/tags

TensorRT-LLM

Docker Image

We provide a pre-built Docker image based on the latest version of TensorRT-LLM.

To get started:

https://hub.docker.com/r/hunyuaninfer/hunyuan-large/tags

docker pull hunyuaninfer/hunyuan-a13b:hunyuan-moe-A13B-trtllm

docker run --name hunyuanLLM_infer --rm -it --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --gpus=all hunyuaninfer/hunyuan-a13b:hunyuan-moe-A13B-trtllm

Prepare Configuration file:

cat >/path/to/extra-llm-api-config.yml <<EOF
use_cuda_graph: true
cuda_graph_padding_enabled: true
cuda_graph_batch_sizes:
- 1
- 2
- 4
- 8
- 16
- 32
print_iter_log: true
EOF

Start the API server:

trtllm-serve \
  /path/to/HunYuan-moe-A13B \
  --host localhost \
  --port 8000 \
  --backend pytorch \
  --max_batch_size 32 \
  --max_num_tokens 16384 \
  --tp_size 2 \
  --kv_cache_free_gpu_memory_fraction 0.6 \
  --trust_remote_code \
  --extra_llm_api_options /path/to/extra-llm-api-config.yml

vLLM

Docker Image

We provide a pre-built Docker image containing vLLM 0.8.5 with full support for this model. The official vllm release is currently under development， note: cuda 12.8 is require for this docker.

To get started:

docker pull docker.cnb.cool/tencent/hunyuan/hunyuan-a13b:hunyuan-moe-A13B-vllm 
or
docker pull hunyuaninfer/hunyuan-a13b:hunyuan-moe-A13B-vllm

Download Model file:
- Huggingface: will download automicly by vllm.
- ModelScope: modelscope download --model Tencent-Hunyuan/Hunyuan-A13B-Instruct
Start the API server:

model download by huggingface:

docker run --rm  --ipc=host \
        -v ~/.cache:/root/.cache/ \
        --security-opt seccomp=unconfined \
        --net=host \
        --gpus=all \
        -it \
        -e VLLM_USE_V1=0 \
        --entrypoint python hunyuaninfer/hunyuan-a13b:hunyuan-moe-A13B-vllm \
        -m vllm.entrypoints.openai.api_server \
        --host 0.0.0.0 \
        --tensor-parallel-size 4 \
        --port 8000 \
        --model tencent/Hunyuan-A13B-Instruct  \
        --trust_remote_code

model downloaded by modelscope:

docker run --rm  --ipc=host \
        -v ~/.cache/modelscope:/root/.cache/modelscope \
        --security-opt seccomp=unconfined \
        --net=host \
        --gpus=all \
        -it \
        -e VLLM_USE_V1=0 \
        --entrypoint python mirror.ccs.tencentyun.com/hunyuaninfer/hunyuan-large:hunyuan-moe-A13B-vllm \
        -m vllm.entrypoints.openai.api_server \
        --host 0.0.0.0 \
        --tensor-parallel-size 4 \
        --port 8000 \
        --model /root/.cache/modelscope/hub/models/Tencent-Hunyuan/Hunyuan-A13B-Instruct/  \
        --trust_remote_code

Tool Calling with vLLM

To support agent-based workflows and function calling capabilities, this model includes specialized parsing mechanisms for handling tool calls and internal reasoning steps.

For a complete working example of how to implement and use these features in an agent setting, please refer to our full agent implementation on GitHub:
🔗 Hunyuan A13B Agent Example

When deploying the model using vLLM, the following parameters can be used to configure the tool parsing behavior:

Parameter	Value
`--tool-parser-plugin`	Local Hunyuan A13B Tool Parser File
`--tool-call-parser`	`hunyuan`

These settings enable vLLM to correctly interpret and route tool calls generated by the model according to the expected format.

Reasoning parser

vLLM reasoning parser support on Hunyuan A13B model is under development.

SGLang

Docker Image

We also provide a pre-built Docker image based on the latest version of SGLang.

To get started:

Pull the Docker image

docker pull docker.cnb.cool/tencent/hunyuan/hunyuan-a13b:hunyuan-moe-A13B-sglang
or
docker pull hunyuaninfer/hunyuan-a13b:hunyuan-moe-A13B-sglang

Start the API server:

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    --ipc=host \
    docker.cnb.cool/tencent/hunyuan/hunyuan-a13b:hunyuan-moe-A13B-sglang \
    -m sglang.launch_server --model-path hunyuan/huanyuan_A13B --tp 4 --trust-remote-code --host 0.0.0.0 --port 30000

Contact Us

If you would like to leave a message for our R&D and product teams, Welcome to contact our open-source team . You can also contact us via email (hunyuan_opensource@tencent.com).

bullerwins/Hunyuan-A13B-Instruct-GGUF

Model Introduction

Key Features and Advantages

Why Choose Hunyuan-A13B?

Related News

Benchmark

Use with transformers

Fast and slow thinking switch

Deployment

TensorRT-LLM

Docker Image

vLLM

Docker Image

Tool Calling with vLLM

Reasoning parser

SGLang

Docker Image

Contact Us

bullerwins/
Hunyuan-A13B-Instruct-GGUF