LLaDA (Large Language Diffusion with mAsking) is a diffusion-based language model that generates tokens in parallel via iterative masked denoising — unlike autoregressive models (GPT, LLaMA) that generate one token at a time.

This repository provides two post-training quantized variants of GSAI-ML/LLaDA-8B-Instruct:

File	Quantization	Size	Memory Saved	Speed (A100)
`llada_int8_quantized.pt`	INT8 per-row	8.54 GB	47%	9.64 tok/s
`llada_int4_quantized.pt`	INT4 packed	4.79 GB	70%	3.39 tok/s

Original model (bfloat16): 16.13 GB

How It Works

All nn.Linear layers are replaced with custom quantized layers:

INT8 — weights scaled per-row to [-127, 127] integers. Scale factors stored in float32. ~1 byte per weight.
INT4 — weights scaled per-row to [-8, 7] integers. Two values packed per byte (uint8). ~0.5 bytes per weight.

Both variants dequantize weights on-the-fly during the forward pass. No changes to model architecture or generation logic.

Usage

Installation

bash

git clone https://github.com/qubitronlabsdev/llada-quantization
cd llada-quantization
pip install -r requirements.txt

Load and Generate

python

from inference import load_quantized, generate
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "GSAI-ML/LLaDA-8B-Instruct",
    trust_remote_code=True
)

# Download weights from this repo first, then:

# INT8
model = load_quantized(
    "llada_int8_quantized.pt",
    mode="int8",
    device="cuda"
)

# INT4
model = load_quantized(
    "llada_int4_quantized.pt",
    mode="int4",
    device="cuda"
)

output = generate(model, tokenizer, "What is machine learning?")
print(output)

Quantize from Scratch

python

from quantize import run_and_save

run_and_save(mode="int8", save_path="llada_int8_quantized.pt")
run_and_save(mode="int4", save_path="llada_int4_quantized.pt")

Hardware Requirements

Variant	Min VRAM	Recommended
INT8	12 GB	A100 / H100
INT4	8 GB	RTX 3090 / A100

Tested on: NVIDIA A100 80GB, NVIDIA H100

Limitations

INT4 introduces slightly more quantization error than INT8
Generation speed depends on sequence length and number of diffusion steps
English only (inherited from base model)

Citation

If you use this work, please cite:

bibtex

@misc{llada-quantization-2026,
  title  = {LLaDA Quantization: INT8 and INT4 for Diffusion Language Models},
  author = {Dhiraj Choudhary},
  year   = {2026},
  url    = {https://github.com/qubitronlabsdev/llada-quantization}
}

Original LLaDA paper:

bibtex

@article{nie2025large,
  title  = {Large Language Diffusion Models},
  author = {Nie, Shen and others},
  year   = {2025},
  url    = {https://arxiv.org/abs/2502.09992}
}

License

Apache 2.0 — same as the original LLaDA model.

qubitron/LLaDA-8B-Quantized

LLaDA-8B-Quantized

Model Description