This repository contains a DFly drafter model trained for speculative decoding with Hy3-A21B under the high-thinking inference setting. The model is trained with AngelSpec and is designed to propose candidate tokens that are verified by the Hy3-A21B target model.
This drafter is not a standalone chat model. It should be deployed with the matching Hy3-A21B target model and the corresponding high-thinking target-side inference mode.
High-thinking and no-thinking decoding induce different hidden-state and acceptance distributions. As a result, a drafter trained for one thinking mode should not be assumed to transfer optimally to the other.
This checkpoint is the high-thinking DFly drafter: it is trained with high-thinking target-side data and should be evaluated and deployed with high-thinking prompts / target-side generation behavior. The matched high-thinking configuration is important for preserving acceptance length and speculative-decoding efficiency.
Evaluation: Accepted Length
The following table reports this model's mean accepted length on Hy3-A21B under the high-thinking setting at temperature 1. The average is computed over the six displayed benchmarks.
Target Model
Drafter
Train Data
Test Data
Math500
GSM8K
HumanEval
MBPP
LiveCodeBench
MT-Bench
Avg.
Hy3-A21B
DFly
High-think
High-think
4.29
4.47
4.76
4.74
3.92
3.42
4.27
The high-thinking drafter is specialized for high-thinking inference. It is especially important for reasoning- and chat-oriented workloads where the target-side thinking mode changes the acceptance distribution.
Deployment
Set TARGET to the Hy3-A21B target model path or repository and DRAFT to the corresponding drafter checkpoint. The relevant inference code can be found in PR https://github.com/vllm-project/vllm/pull/50246.
