The Snowflake AI Research team is releasing a series of SwiftKV optimized Llama-3.1 models. SwiftKV is a series of inference optimizations that goes beyond traditional key-value (KV) cache compression. This method reduces computational overhead during prompt processing by combining model rewiring and knowledge-preserving self-distillation, allowing prefill tokens to skip up to half the model's layers. SwiftKV achieves up to 2x improvements in throughput, latency, and cost efficiency with minimal accuracy loss, making LLM deployments more performant and economically viable.
release-2508 (Aug 2025): Updated model weights for long-context up to 128K
release-2412 (Dec 2024): Initial model release
Performance Metrics
To evaluate SwiftKV’s performance, we focus on the following key metrics (see more details in our blog):
Combined throughput: The total number of input and output tokens processed per second. This determines:
For batch processing, the time required to complete jobs.
For interactive use, the volume of concurrent requests a system can handle.
TTFT: The latency between a user request and receiving the first token in the response.
TPOT: The latency between subsequent tokens after the first token.
Combined input and output throughput for Llama 3.1 70B (left) and Llama 3.1 405B (right) across a range of input lengths (bottom).
performance plot of llama-405B w. swiftkv
TTFT (top) and TPOT (bottom) for input lengths 2000 (left), 8000 (middle), and 32000 (right) for Llama 3.1 405B fp8 model. For each experiment, a range of different request arrival rates is simulated. Each request generates 256 output tokens.
performance plot of llama-405B w. swiftkv
Eval Metrics
For a full breakdown on evaluation metrics and performance impact please refer to our blog and arXiv paper but below we've outlined some relevant evaluation metrics.
