Hy-MT2-1.8B-SpecDraft-30B
Experimental Speculative Decoding Draft Model for Hy-MT2-30B-A3B
This is an experimental project that modifies Hy-MT2-1.8B into a speculative decoding draft model for Hy-MT2-30B-A3B.
Acceleration is not guaranteed. This project is mainly for users who want to explore speculative decoding on MoE models.
Technical Details
The original vocabularies are different:
- Hy-MT2-1.8B: 120818 tokens
- Hy-MT2-30B-A3B: 120832 tokens
The model was modified by:
- Replacing tokenizer with the Hy-MT2-30B-A3B tokenizer.
- Expanding embedding and lm_head weights:
- [120818, 2048] -> [120832, 2048]
- Initializing new token embeddings with mean padding.
In speculative decoding, the draft model only proposes candidate tokens.
The target model Hy-MT2-30B-A3B always performs final verification, so final output quality is not affected.
Usage
1llama-server ^
2 -m Hy-MT2-30B-A3B-APEX-Imatrix-I-Quality.gguf ^
3 --spec-type draft-simple ^
4 --spec-draft-model Hy-MT2-1.8B-SpecDraft-30B-Q4_K_M.gguf ^
5 --spec-draft-ngl 999 ^
6 --spec-draft-n-max 4 ^
7 --spec-draft-n-min 3 ^
8 --flash-attn on ^
9 --gpu-layers 999
Performance Notes
Hy-MT2-30B-A3B is a MoE model with around 3B active parameters.
Because its decoding speed is already very high, speculative decoding may not always improve performance.
Possible results:
- faster
- similar speed
- slightly slower
depending on hardware and llama.cpp scheduling behavior.
This project is intended for: - llama.cpp enthusiasts - large VRAM users - MoE speculative decoding experiments
License
Follows the original Hy-MT2 license. For research and experimental use only.
