DFloat11/

Llama-3.1-8B-Instruct-DF11

DFloat11 Compressed Model: meta-llama/Llama-3.1-8B-Instruct

🔍 How It Works

• No CPU decompression or host-device data transfer -- all operations are handled entirely on the GPU.
• Decompression overhead is constant per forward pass and independent of batch size, making DFloat11 increasingly efficient at larger batch sizes.
• DFloat11 is much faster than CPU-offloading approaches, enabling practical deployment in memory-constrained environments.
• At batch size = 1, inference is approximately 2× slower than the original BF16 model, but the performance gap narrows significantly with larger batches.
• The compression is fully lossless, guaranteeing that the model’s outputs are bit-for-bit identical to those of the original model.

🔧 How to Use

1. Install the DFloat11 pip package (installs the CUDA kernel automatically; requires a CUDA-compatible GPU and PyTorch installed):
   bash

   1pip install dfloat11[cuda12] 2# or if you have CUDA version 11: 3# pip install dfloat11[cuda11]
2. To use the DFloat11 model, run the following example code in Python:
   python

   1import torch 2from dfloat11 import DFloat11Model 3from transformers import AutoTokenizer 4 5model_id = "DFloat11/Llama-3.1-8B-Instruct-DF11" 6 7model = DFloat11Model.from_pretrained(model_id, device_map="auto") 8 9tokenizer = AutoTokenizer.from_pretrained(model_id) 10tokenizer.pad_token = tokenizer.eos_token 11 12prompt = "Question: What is a binary tree and its applications? Answer:" 13inputs = tokenizer(prompt, return_tensors="pt", padding=True).to(model.device) 14 15with torch.no_grad(): 16 output = model.generate( 17 **inputs, 18 max_new_tokens=256, 19 do_sample=True, 20 ) 21 22print(tokenizer.batch_decode(output, skip_special_tokens=True))

📄 Learn More

• Paper: 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float
• GitHub: https://github.com/LeanModels/DFloat11
• HuggingFace: https://huggingface.co/DFloat11