You can deploy this model and start earning money today!

LorenzoNava/
Devstral-2-123B-NVFP4-TensorRT-LLM

Name: Devstral-2-123B-NVFP4-TensorRT-LLM
Availability: InStock

mistral3

text-generation

conversational

mistralai/Devstral-2-123B-Instruct-2512

finetune

Views

No views yet

Devstral-2-123B-NVFP4-TensorRT-LLM

NVFP4 Quantized Devstral-2-123B for TensorRT-LLM on NVIDIA Blackwell GPUs

Model Details

Property	Value
Base Model	mistralai/Devstral-2-123B-Instruct-2512
Quantization	NVFP4 (4-bit floating point)
Format	TensorRT-LLM checkpoint
Size	~120GB (from 240GB FP8)
Quantization Time	4.76 minutes on B300
Producer	nvidia-modelopt 0.27.1

Hardware Requirements

NVFP4 ONLY works on NVIDIA Blackwell GPUs:

NVIDIA B200, B300, GB200
NVIDIA RTX 5000 series (Blackwell architecture)
Any GPU with SM100+ compute capability

Will NOT work on: A100, H100, RTX 4000 series, or older GPUs.

Quantization Configuration

json

{
    "architecture": "LlamaForCausalLM",
    "dtype": "bfloat16",
    "num_hidden_layers": 88,
    "num_attention_heads": 96,
    "num_key_value_heads": 8,
    "hidden_size": 12288,
    "vocab_size": 131072,
    "quantization": {
        "quant_algo": "NVFP4",
        "group_size": 128,
        "has_zero_point": false,
        "pre_quant_scale": true
    }
}

How This Model Was Quantized

The quantization was performed using NVIDIA ModelOpt with specific patches required for the ministral3 architecture.

Container and Dependencies

bash

# Use NVIDIA PyTorch container
docker pull nvcr.io/nvidia/pytorch:25.05-py3

# Install dependencies (git transformers required for ministral3)
pip install tokenizers git+https://github.com/huggingface/transformers.git accelerate datasets

Complete Quantization Script

python

#!/usr/bin/env python3
import torch, time, os

# CRITICAL PATCH 1: Add Conv1D to transformers
import transformers.modeling_utils as mu
if not hasattr(mu, "Conv1D"):
    import torch.nn as nn
    class Conv1D(nn.Module):
        def __init__(self, nf, nx):
            super().__init__()
            self.nf = nf
            w = torch.empty(nx, nf)
            nn.init.normal_(w, std=0.02)
            self.weight = nn.Parameter(w)
            self.bias = nn.Parameter(torch.zeros(nf))
        def forward(self, x):
            size_out = x.size()[:-1] + (self.nf,)
            x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
            return x.view(size_out)
    mu.Conv1D = Conv1D

# CRITICAL PATCH 2: Disable attention auto-registration
import modelopt.torch.quantization.plugins.huggingface as hf_plugin
hf_plugin.register_hf_attentions_on_the_fly = lambda model: None

# Load model
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Devstral-2-123B-Instruct-2512",
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
    attn_implementation="flash_attention_2"
)

# Prepare calibration
tokenizer = AutoTokenizer.from_pretrained(
    "mistralai/Devstral-2-123B-Instruct-2512",
    trust_remote_code=True
)
from datasets import load_dataset
dataset = load_dataset("cnn_dailymail", "3.0.0", split="train", streaming=True)
calib_texts = [item["article"][:6000] for i, item in enumerate(dataset) if i < 128]
calib_enc = tokenizer(
    calib_texts, padding=True, truncation=True, max_length=2048, return_tensors="pt"
)

# Quantize
import modelopt.torch.quantization as mtq
def calibrate_loop(model):
    device = next(model.parameters()).device
    for i in range(len(calib_texts)):
        ids = calib_enc.input_ids[i:i+1].to(device)
        mask = calib_enc.attention_mask[i:i+1].to(device)
        with torch.no_grad():
            model(input_ids=ids, attention_mask=mask)

model = mtq.quantize(model, mtq.NVFP4_DEFAULT_CFG.copy(), forward_loop=calibrate_loop)

# Export
from modelopt.torch.export import export_tensorrt_llm_checkpoint
export_tensorrt_llm_checkpoint(
    model,
    decoder_type="llama",
    dtype=torch.bfloat16,
    export_dir="./devstral-123b-nvfp4",
    inference_tensor_parallel=1,
    inference_pipeline_parallel=1
)

Known Issues

TensorRT-LLM Build Currently Fails

The checkpoint format from ModelOpt export_tensorrt_llm_checkpoint is currently incompatible with TensorRT-LLM expected NVFP4 tensor layout. TensorRT-LLM expects specific scaling factor tensors (weights_global_scaling_factor, weights_block_scaling_factor_interleaved, alpha) that ModelOpt export does not provide.

This is a known format mismatch between ModelOpt 0.27.1 and TensorRT-LLM 1.0.0rc6/1.2.0rc5.

Transformers Version Requirement

The base model uses ministral3 architecture which requires latest transformers from git:

pip install git+https://github.com/huggingface/transformers.git

Stable versions (4.53-4.56) will fail with KeyError: ministral3.

Potential Inference Solutions

Wait for ModelOpt/TRT-LLM compatibility fix - Format mismatch is a known issue
Use vLLM - May support NVFP4 on Blackwell with proper configuration
Direct PyTorch inference - Load quantized model in PyTorch with modelopt runtime

Files

rank0.safetensors - Quantized model weights (120GB)
config.json - TRT-LLM checkpoint configuration
quant_cfg.json - Quantization configuration
tokenizer.json - Tokenizer
tokenizer_config.json - Tokenizer configuration
chat_template.jinja - Chat template

License

Same as base model: Mistral AI License

Credits

Base model: Mistral AI
Quantization: NVIDIA ModelOpt
Hardware: NVIDIA B300 (268GB VRAM)

LorenzoNava/Devstral-2-123B-NVFP4-TensorRT-LLM