GGUF quantization of mistralai/Devstral-Small-2-24B-Instruct-2512 — a 24B dense multimodal coding model built on Mistral-Small-3.1 with 393K context, vision support, and tool calling.

Quantized to MXFP4 format for efficient inference with minimal quality loss.

About MXFP4

MXFP4 (Microscaling FP4, E2M1) is an open standard 4-bit format under the OCP Microscaling Formats specification. It uses E2M1 layout with block-wise shared 8-bit scaling factors, working on any GPU or CPU without hardware-specific acceleration.

Files

Filename	Type	Size	Description
`devstral-small-2-mxfp4.gguf`	GGUF (MXFP4)	11.85 GB	Quantized text model weights
`mmproj-devstral-small-2-f16.gguf`	F16 mmproj	0.82 GB	Vision encoder projector (24-layer SigLIP, 1024 hidden)
`README.md`	Markdown	-	Model card

Quantization Details

Property	Value
Format	MXFP4
Bits Per Weight	4.32 BPW
File Size	11.85 GB (text) + 0.82 GB (mmproj)
Tensor Count	363 (text) + 222 (mmproj)
Architecture	Mistral3 (mistral3)

Model Description

Developer: Mistral AI
Base Model: Mistral-Small-3.1-24B-Base-2503
Architecture: Dense transformer, Mistral3ForConditionalGeneration
Parameters: 24B
Context Length: 393,216 tokens (native)
Vision: 24-layer SigLIP ViT (1024 hidden), image-text-to-text
Capabilities: Coding, tool calling, agentic workflows
Languages: Multilingual
License: Apache 2.0

Usage

llama.cpp (CLI)

bash

# Text + Image
llama-cli -m devstral-small-2-mxfp4.gguf \
  --mmproj mmproj-devstral-small-2-f16.gguf \
  --image photo.jpg \
  -p "Describe this image in detail" \
  -n 512

# Text only
llama-cli -m devstral-small-2-mxfp4.gguf \
  -p "Write a Python function to sort a list" \
  -n 512

# OpenAI-compatible server
llama-server -m devstral-small-2-mxfp4.gguf \
  --mmproj mmproj-devstral-small-2-f16.gguf \
  --port 8080

llama-cpp-python

python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="FreedomAISVR/Devstral-Small-2-MXFP4-GGUF",
    filename="devstral-small-2-mxfp4.gguf",
    n_gpu_layers=-1,
)

response = llm.create_chat_completion([
    {"role": "user", "content": "Write a Python function to sort a list"}
])
print(response["choices"][0]["message"]["content"])

Direct download

python

from huggingface_hub import hf_hub_download

for filename in ["devstral-small-2-mxfp4.gguf", "mmproj-devstral-small-2-f16.gguf"]:
    hf_hub_download(
        repo_id="FreedomAISVR/Devstral-Small-2-MXFP4-GGUF",
        filename=filename,
        local_dir="./models"
    )

Quantization Pipeline

1. Download source weights
   huggingface_hub.snapshot_download("mistralai/Devstral-Small-2-24B-Instruct-2512")

2. Convert text model to F16 GGUF (dequantize FP8)
   convert_hf_to_gguf.py --outtype f16

3. Extract vision encoder
   convert_hf_to_gguf.py --mmproj --outtype f16

4. Quantize to MXFP4
   llama-quantize devstral-small-2-f16.gguf devstral-small-2-mxfp4.gguf MXFP4

Hardware

Component	Specification
GPU	NVIDIA RTX 5060 Ti (Blackwell)
System RAM	64 GB
Storage	NVMe

License

Apache 2.0 — same as the original mistralai/Devstral-Small-2-24B-Instruct-2512.