Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-W4A16
Model description
This is a
W4A16 quantized version of
gratex/Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-unresolvable-pattern-finetune-v1, which is a fine-tuned Mistral-Small-3.2-24B-Instruct-2506 designed to improve the evaluation of SBVR (Semantics of Business Vocabulary and Business Rules) rules.
The quantization was performed using AutoRound 0.15.0 with SignRoundV2 optimization, reducing the model size from 45 GB (BF16) to 15 GB while maintaining high fidelity to the original model. The model supports both text-only and image+text (vision) inference.
The multimodal projector tensors are stored under their Mistral-native names (multi_modal_projector.w_in, multi_modal_projector.w_out, pre_mm_projector_norm, patch_merger.merging_layer) so that vLLM's Mistral-native loading path matches them correctly; these tensors are BF16 and are not quantized.
Quantization configuration
| Parameter | Value |
|---|
| Quantization method | AutoRound (SignRoundV2) |
| AutoRound version | 0.15.0 |
| Weight bits | 4 (W4A16) |
| Activation bits | 16 (BF16) |
| Group size | 128 |
| Symmetric | True |
| Iterations | 1000 |
| Calibration samples | 512 |
| Sequence length | 4096 |
| Packing format | auto_round:auto_gptq |
| lm_head | Kept in 16-bit (not quantized) |
| enable_alg_ext | True (SignRoundV2) |
| enable_torch_compile | True |
| low_gpu_mem_usage | True |
| GPU | NVIDIA RTX 5090 (32 GB) |
Calibration dataset
The calibration dataset was constructed from multiple domain-specific sources to ensure the quantized model retains accuracy across all relevant domains:
| Domain | Samples | Description |
|---|
| SBVR rules | ~50% | SBVR compliance rules in JSON format, rule status assignment (PASSED/FAILED/UNRESOLVABLE), insurance terminology |
| Slovak text | ~25% | Slovak Wikipedia, FineWeb2, EN↔SK parallel corpus (opus100) — ensures diacritics preservation |
| English general | ~15% | UltraChat conversations, general English text |
| Reasoning | ~10% | MMLU-Pro (14 categories), BoolQ, TruthfulQA |
All 512 calibration samples were concatenated to a minimum of 4,096 tokens each (20,480 characters) to maximize the calibration signal per sample. The dataset is published alongside this model as calibration_dataset.jsonl.
Quantization script
The quantization script is published as quantize.py in this repository.
Evaluation results
WikiText-2 Perplexity
Standard WikiText-2-raw-v1 benchmark with sliding window (stride=512, max_length=2048, 590 windows, 301,616 tokens):
| Model | WikiText-2 PPL | Degradation |
|---|
| BF16 (original) | 4.7596 | — |
| W4A16 (this model) | 4.9160 | +3.28% |
Comprehensive evaluation
30 domain-specific prompts (SBVR, Slovak, English, code, reasoning) comparing token-level agreement, KL divergence, and generation fidelity against the BF16 baseline:
| Metric | Value |
|---|
| Average PPL degradation | +1.39% |
| KL divergence | 0.0231 |
| Cosine similarity | 0.9945 |
| Spearman correlation | 0.9984 |
| Top-1 token agreement | 70.00% |
| Top-5 token agreement | 90.69% |
| Top-10 token agreement | 91.05% |
| Greedy match | 90.46% |
Per-category PPL
| Category | BF16 PPL | W4A16 PPL | Degradation |
|---|
| SBVR | 30.43 | 30.89 | +1.51% |
| Slovak | 11.09 | 11.16 | +0.63% |
| English | 5.89 | 6.16 | +4.58% |
| Code | 3.41 | 3.38 | -0.88% |
| Reasoning | 4.03 | 3.99 | -0.99% |
Usage
vLLM (recommended for both text and vision)
This model requires vLLM's Mistral-native loading mode. Deploy with config_format: mistral and tokenizer_mode: mistral (the Tekken tokenizer is shipped as tekken.json; using the default HF tokenizer mode produces garbled output):
1docker run -d --name vllm-mistral-sbvr \
2 --runtime=nvidia --gpus '"device=0"' \
3 -p 8000:8000 \
4 -v /path/to/model:/workspace/model \
5 --ipc=host --shm-size=16g \
6 vllm/vllm-openai:cu129-nightly \
7 --model /workspace/model \
8 --served-model-name mistral-24b-sbvr-w4a16 \
9 --config-format mistral --tokenizer-mode mistral \
10 --quantization auto_round \
11 --dtype bfloat16 \
12 --trust-remote-code
Both text-only and image+text (vision) requests are supported:
1import base64, requests
2
3img_b64 = base64.b64encode(open("image.jpg", "rb").read()).decode()
4
5# Text
6requests.post("http://localhost:8000/v1/chat/completions", json={
7 "model": "mistral-24b-sbvr-w4a16",
8 "messages": [{"role": "user", "content": "Evaluate this SBVR rule: ..."}],
9 "max_tokens": 512,
10})
11
12# Vision
13requests.post("http://localhost:8000/v1/chat/completions", json={
14 "model": "mistral-24b-sbvr-w4a16",
15 "messages": [{"role": "user", "content": [
16 {"type": "text", "text": "Describe this image."},
17 {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
18 ]}],
19 "max_tokens": 512,
20})
transformers
The projector tensors use Mistral-native names for vLLM compatibility. transformers 5.x loads Mistral3 checkpoints without a name-conversion layer, so the projector tensors need manual renaming (w_in→linear_1, w_out→linear_2, pre_mm_projector_norm→norm, patch_merger→multi_modal_projector.patch_merger.merging_layer) when loading this model with transformers. For production use, the vLLM path above is the recommended and tested route.
Model details
| Property | Value |
|---|
| Architecture | Mistral3ForConditionalGeneration |
| Parameters | 1.78B (quantized) / 24B (original) |
| Model size | 15 GB |
| Precision | W4A16 (4-bit weights, 16-bit activations) |
| Context length | 128K (inherited from base model) |
| Vision | Pixtral tower, 1024×1024+ images, 4 images per prompt |
| License | Apache 2.0 |
HELM comparison: fine-tuned BF16 vs W4A16
The BF16 fine-tuned source model and this W4A16 quantized version were evaluated side by side on Stanford HELM benchmarks (crfm-helm >= 0.5.0, 1,000 instances per scenario, served via an OpenAI-compatible endpoint).
General benchmarks
| Suite | FT BF16 | FT W4A16 | Δ |
|---|
| bbq | 90.60% | 91.90% | +1.30 pp |
| boolq | 85.10% | 89.40% | +4.30 pp |
| commonsense | 82.80% | 82.90% | +0.10 pp |
| damaged_items_mc | 100.00% | 100.00% | 0.00 pp |
| insurance_claim_extraction | 64.14% | 61.63% | −2.51 pp |
| polycontext_rules | 71.50% | 62.10% | −9.40 pp |
| quac | 59.97% | 58.85% | −1.12 pp |
| synthetic_reasoning | 93.40% | 91.75% | −1.65 pp |
| truthful_qa | 68.35% | 68.50% | +0.15 pp |
MMLU Pro
| Subject | FT BF16 | FT W4A16 | Δ |
|---|
| biology | 76.43% | 76.01% | −0.42 pp |
| business | 34.98% | 37.77% | +2.79 pp |
| chemistry | 31.50% | 32.20% | +0.70 pp |
| computer_science | 53.90% | 52.68% | −1.22 pp |
| economics | 64.57% | 63.74% | −0.83 pp |
| engineering | 33.85% | 37.36% | +3.51 pp |
| health | 64.55% | 62.47% | −2.08 pp |
| history | 60.63% | 57.48% | −3.15 pp |
| law | 42.20% | 42.10% | −0.10 pp |
| math | 34.50% | 34.60% | +0.10 pp |
| other | 54.98% | 53.14% | −1.84 pp |
| philosophy | 51.30% | 48.70% | −2.60 pp |
| physics | 32.80% | 34.50% | +1.70 pp |
| psychology | 68.55% | 67.54% | −1.01 pp |
Summary averages
| Metric | FT BF16 | FT W4A16 | Δ |
|---|
| HELM overall avg (23 suites) | 61.77% | 61.19% | −0.58 pp |
| MMLU Pro avg (14 subjects) | 50.34% | 50.02% | −0.32 pp |
| Non-MMLU avg (9 scenarios) | 79.54% | 78.56% | −0.98 pp |
| SBVR unresolvable overall (1,432 items) | 90.92% | 91.97% | +1.05 pp |
BBQ bias
| Metric | FT BF16 | FT W4A16 |
|---|
| Ambiguous accuracy | 88.67% | 88.67% |
| Non-ambiguous accuracy | 95.29% | 95.29% |
| Bias gap ( | amb − non-amb | ) |
Original model
This model is a quantized version of
gratex/Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-unresolvable-pattern-finetune-v1. Please refer to the original model card for detailed information about the fine-tuning methodology, training configuration, and HELM evaluation results.
