Quantization vLLM supports FP8 (8-bit floating point) weight and activation quantization using hardware acceleration on GPUs such as Nvidia H100 and AMD MI300x.
NGram Speculative Decoding configures vLLM to use speculative decoding where proposals are generated by matching n-grams in the prompt. This is a great fit for edit predictions since many of the tokens are already present in the prompt and the model is only needed to generate changes to the code file.

vllm serve zed-industries/zeta --served-model-name zeta --enable-prefix-caching --enable-chunked-prefill --quantization="fp8" --speculative-model [ngram] --ngram-prompt-lookup-max 4 --ngram-prompt-lookup-min 2 --num-speculative-tokens 8

Learn More

DPO Fine-Tuning: <a href="https://huggingface.co/datasets/zed-industries/zeta/blob/main/script/dpo.ipynb" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">View Notebook
SFT Fine-Tuning: <a href="https://huggingface.co/datasets/zed-industries/zeta/blob/main/script/sft.ipynb" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">View Notebook

For more insights about the model and its integration in Zed, check out the official blog post: Zed Blog - Edit Prediction