Views
No views yet
| Parameter | Value |
|---|---|
| Quantization Type | NF4 (NormalFloat4) |
| Double Quantization | Yes |
| Compute Dtype | float16 |
| Library | bitsandbytes |
| VRAM Reduction | ~65% (5 GB → ~1.7 GB) |
1from transformers import BitsAndBytesConfig
2import torch
3
4nf4_config = BitsAndBytesConfig(
5 load_in_4bit=True,
6 bnb_4bit_quant_type="nf4",
7 bnb_4bit_use_double_quant=True,
8 bnb_4bit_compute_dtype=torch.float16
9)