from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "greghavens/nemotron-3-super-120b-bnb-4bit",
    trust_remote_code=True,
    device_map="auto",
)

quantization_config is in config.json, so no BitsAndBytesConfig is needed and nothing is quantized at load time — the weights are already NF4.

What was quantized


format	NF4, blocksize 64, double quantization
compute dtype	bfloat16
storage dtype	bfloat16
quantized	every 2-D `nn.Linear` weight
left in BF16	`backbone.embeddings`, `lm_head`, Conv1D (3-D), all 1-D params (`A_log`, `D`, `dt_bias`, norms)

bnb_4bit_quant_storage=bfloat16 matters for FSDP: FSDP only shards float dtypes, and 4-bit weights default to uint8. With the default storage dtype FSDP leaves the base replicated on every rank instead of sharding it.

Precision vs the official Nemotron 3 Super checkpoints

NVIDIA's published precision table for the Super FP8 checkpoint, against this build:

component	NVIDIA Super	this build
Mamba GEMM (`in_proj` / `out_proj`)	FP8	NF4
Mamba 1D Conv	BF16	BF16
MoE GEMM (sparse + shared experts)	FP8	NF4
Embeddings / output layers	BF16	BF16
Attention GEMM (QKV + Out Projection)	BF16	NF4
Router	FP32	NF4
MoE Latent Projection GEMM	BF16	NF4

This build quantizes uniformly, so the attention projections, router, and MoE latent projections are NF4 here where NVIDIA keeps them at higher precision. For serving, the official FP8 and NVFP4 checkpoints follow NVIDIA's recipe.

How it was built

Streamed one shard at a time — download BF16 shard, quantize via bitsandbytes.functional.quantize_4bit, write the NF4 shard, delete the BF16 shard — keeping peak disk near the 62 GB output instead of the 292 GB a load-and-save needs.

License

Inherits the license of NVIDIA-Nemotron-3-Super-120B-A12B. Quantization does not alter licensing.