Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
Arctic-LSTM-Speculator-Llama-3.1-70B-Instruct – AI Model by Snowflake | AlphaNeural AI
You can deploy this model and start earning money today!
Snowflake
/
Arctic-LSTM-Speculator-Llama-3.1-70B-Instruct
like
0
pytorch
mlp_speculator
llama3.1
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
ArcticSpeculator
Build a fastest OSS vllm-based speculative decoding system for your own model, using
ArcticTraining
and
ArcticInference
!
We compare the throughput (tokens/s) of existing vllm-based speculative decoding systems for Llama3.1-70B-Instruct on 8xH100 as below:
method
ShareGPT
HumanEval
VLLM V1 Baseline
84.1
84.1
VLLM V1 Eagle
102.2
112.0
VLLM V1 Eagle3
77.7
85.3
VLLM V0 MLP-Speculator (IBM)
77.9
66.7
ArcticSpeculator
172.4
203.7
For more details about ArcticSpeculator and how to use it:
❄️
Using Arctic-Inference and Arctic-Training for improving real-world speculative decoding Performance (blog)
🚀
Getting started guide using ArcticTraining
We also release ArcticSpeculator checkpoints we trained with
ArcticTraining
to run with
ArcticInference
:
model
ArcticSpeculator
Llama-3.1-70B-Instruct
Arctic-LSTM-Speculator-Llama-3.1-70B-Instruct
Llama-3.3-70B-Instruct
Arctic-LSTM-Speculator-Llama-3.3-70B-Instruct
Qwen2.5-32B-Instruct
Arctic-LSTM-Speculator-Qwen2.5-32B-Instruct
Llama-3.1-8B-Instruct
Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct
