Diffusion transformer (DiT), quantized from black-forest-labs/FLUX.1-dev. Repackaged to the .fni8 resident format (~17.0 GB) for the fni8 W8A8/W4A8 DP4A kernels on NVIDIA Volta (sm_70) — Tesla V100 / CMP 100-210.

Status

Validated: end-to-end ComfyUI-fni8 sanity pass (a one-step output that is finite and non-constant — a sanity check, not a quality benchmark). Performance is fleet-specific. All fni8 speedups are measured on the CMP 100-210 mining-card fleet, where the fp16 tensor cores are firmware-gimped. These numbers do not transfer to a real Tesla V100 (whose fp16 tensor cores would beat dp4a).

Format

Weights: int8 per-row (W8A8), fp32 scales, resident dp4a VRAM layout.
Why dp4a: sm_70 has no int8 tensor cores, so the matmul contraction runs on the __dp4a CUDA-core intrinsic. On the CMP 100-210 fleet (whose fp16 tensor cores are firmware-limited) dp4a is the fast path, not a compromise.

How to run

ComfyUI-fni8 is the runtime (UnetLoaderFNI8 runs the diffusion transformer through the dp4a kernels; the text encoder and VAE are unchanged). fni8-serve is for LLMs only and does not load this model.

Limitations

Quantization is lossy: int8 (and especially int4) outputs differ from the fp16/bf16 parent, and the difference varies by task.
Capabilities, biases, and risks of the parent model carry over — see the parent card.
This is a derivative quantization, not a relicense; the parent model's license and acceptable uses apply.

Part of the fni8 stack: kernels · LLM serving · ComfyUI DiTs.

jajmangold/FLUX.1-dev-fni8

FLUX.1-dev — fni8 (int8/W4A8 dp4a, Volta sm_70)

Status

Format

How to run

Limitations

jajmangold/
FLUX.1-dev-fni8