Group Tree Optimization (GTO) is a framework designed to address draft policy misalignment in speculative decoding. While standard methods optimize for a single greedy path, GTO aligns training with the actual tree-based decoding policy used during inference. This is achieved through a Draft Tree Reward objective and a stable Group-based Draft Policy Training scheme.

GTO achieves state-of-the-art acceleration for LLM inference:

To use this model for accelerated inference, please follow the setup instructions in the official GTO repository.

The codebase provides a web interface for testing the acceleration. After setting up the environment and cloning the repo, you can run:

The total-token parameter represents the number of draft tokens. Adjusting this based on your specific device and model can achieve better results.

If you find this work useful, please cite:

The implementation is based on the open-source repository of EAGLE. This project has been influenced by many projects in the LLM community, such as HASS and GRIFFIN.