We trained Qwen2.5-VL-7B/32B/72B-Instruct EAGLE3 draft models on 95K randomly selected samples from the FreedomIntelligence/ALLaVA-4V dataset using SpecForge.
Qwen2.5-VL-7B-Instruct: Up to 1.68x speedup on MMStar benchmark (TP=1)
Qwen2.5-VL-7B Throughput Comparison
Qwen2.5-VL-32B-Instruct: Up to 1.52x speedup on MMStar benchmark (TP=4)
Qwen2.5-VL-32B Throughput Comparison
Qwen2.5-VL-72B-Instruct: Up to 2.04x speedup on MMStar benchmark (TP=4)
Qwen2.5-VL-72B Throughput Comparison
Analysis
Dataset Performance
We evaluated the models on four datasets representing different task types:
HumanEval: Code generation tasks
MMLU: Multilingual understanding tasks
MMStar: Multimodal tasks
Math500: Mathematical reasoning tasks
Key Findings:
MMStar: Achieved the best performance gains with accept_length typically reaching ~3.5, sometimes approaching 4.5. This is likely due to the similarity between the training dataset ALLaVA-4V (multimodal tasks) and MMStar's distribution.
HumanEval: Unexpectedly, none of the 7B/32B/72B models showed throughput improvements, and accuracy remained consistently at 0. This appears to be a bug that requires further investigation.
MMLU/Math500: Accept_length remained relatively consistent, typically ranging from 2 to 2.5.
Accuracy Consistency
Hyperparameter Analysis
Based on extensive experiments, we recommend the following configuration for H200 GPUs:
Steps: 3-4
TopK: 10
Draft Tokens: 64
Performance Heatmap
Global Heatmap
Steps Impact Analysis
Steps Impact
Draft Tokens Impact Analysis
Draft Tokens Impact
Size Scaling Analysis
It can be observed that larger models (72B) achieve better acceleration. The reason the 32B model has a smaller speedup compared to the 7B model is that it uses TP=4, while the 7B model uses TP=1.
The training loss and accuracy curves are shown below:
Training Loss and Accuracy
Known Issues and Solutions
When directly using the training scripts from the PR, several issues may arise. Below we outline the problems encountered and their solutions. We plan to submit a new PR to address these issues in the SpecForge repository.
Data Preprocessing
Issue: Data loading stalls at 0% (Deadlock).
Solution: Set export OMP_NUM_THREADS=1 in the launch script to force the underlying library to run in single-threaded mode.
OOM During FSDP Initialization
Issues Identified:
Memory Spike: Calling .cuda() before FSDP sharding causes each GPU to load the full large model, doubling memory usage during initialization.
Large Granularity: By default, the entire model is treated as a single FSDP unit. For 32B/72B models with massive parameters, this prevents execution without parameter sharding.
Full Parameter Aggregation: When saving, FSDP defaults to aggregating full parameters on GPU 0 (including 72B frozen parameters), instantly exceeding memory limits.
Solutions:
Remove .cuda() during model loading (keep on CPU)
Specify device_id during FSDP initialization to enable streaming sharded loading from CPU to GPU
Use transformer_auto_wrap_policy to wrap by layer (Decoder Layer)
Change strategy from SHARD_GRAD_OP to FULL_SHARD (ZeRO-3)
Use FullStateDictConfig(offload_to_cpu=True, rank0_only=True) to move parameter aggregation from GPU memory to CPU memory
These solutions are verified to work for 32B/72B models.
