Mistral-Nemo-Instruct-2407 · AWQ 4-bit (compressed-tensors W4A16, async vLLM)
Quantized and evaluated by PBH Applied Systems, LLC
— Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure
🔬
This repository is part of a production-oriented evaluation series. Every model published under
pbhappliedsystems has been independently evaluated using
quant_eval v7.21 — a proprietary behavioral evaluation harness developed by PBH Applied Systems. Scores measure real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning families — not perplexity or leaderboard proxies.
⚠️ Single-runner evaluation. No FP16 baseline was evaluated for this model; all behavioral data comes from the 4-bit quantized_vllm runner only. Aggregate dimension scores are reported as per-family pass rates rather than a quantization-degradation delta.
🧩 Backend note. This is the AWQ INT4 / compressed-tensors W4A16-asymmetric checkpoint, served through vLLM's asynchronous OpenAI-compatible server (vllm serve --quantization compressed-tensors). The asymmetric W4A16 scheme requires vLLM 0.8.5.post1, whose scheme matcher accepts asymmetric weights.
Try This Model in the Live AI Agent Demo
This model is part of the PBH Applied Systems live AI Agent Demo, where visitors can test evaluated quantized open-weight models across production-style agent workflows: reasoning and analysis, document intelligence, and code automation.
The demo uses quant_eval results to show how model selection changes by task. This AWQ build is the fast, small-footprint option in the series: a 12B model in ~8 GB, served on vLLM at sub-second latency per evaluated case, with clean tool-call output. The evaluation results below explain where it preserves useful behavior, where 4-bit quantization introduces risk, and what guardrails are recommended before production deployment.
Model Description
This repository contains the
AWQ 4-bit (compressed-tensors W4A16-asymmetric) checkpoint of
mistralai/Mistral-Nemo-Instruct-2407, a 12-billion parameter instruction-tuned model jointly developed by Mistral AI and NVIDIA. The weights were compressed with
llmcompressor (AWQ algorithm, compressed-tensors format) and are served through vLLM.
Key Characteristics
- Parameters: 12B (
MistralForCausalLM)
- Format: compressed-tensors · W4A16-asymmetric (4-bit weights, 16-bit activations) · AWQ
- Quantization tool: llmcompressor (recipe in
recipe.yaml, scheme in quantization_manifest.json)
- File size: ~8.35 GB (2 ×
safetensors shards, 8,350,915,416 bytes)
- Serving runtime: vLLM 0.8.5.post1 (async OpenAI server),
--quantization compressed-tensors
- Minimum VRAM (GPU inference): ~10–12 GB (weights + KV cache at modest context)
- Recommended GPU tier: RTX 3060 12 GB · RTX 4090 · A10G · A100
- Context window: 128K tokens (native); evaluated at
max_model_len=4096
- Inference speed (eval hardware): avg 0.936 sec/case on RTX 4090
- License: Apache 2.0
Quantization Scheme
- Weights: 4-bit, group-wise, asymmetric (zero-point), compressed-tensors packed.
- Activations: 16-bit (
float16) — W4A16.
- Algorithm: AWQ (activation-aware weight quantization) via llmcompressor.
Exact group size and per-layer scheme are recorded in recipe.yaml and quantization_manifest.json, published in this repository. The asymmetric weight scheme is the reason vLLM 0.8.5.post1 is required — earlier vLLM scheme matchers reject asymmetric W4A16.
PBH Applied Systems Evaluation — quant_eval v7.21
Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.21
Run ID: 20260629_184555 · Fixtures: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c...) · Seed: 42
Hardware: NVIDIA RTX 4090 · Runner: quantized_vllm (compressed-tensors W4A16) · Total rows: 42
Per-Family Pass Rates (AWQ W4A16)
| Family | N | Pass Rate | Avg Secs | Bucket Score | Notes |
|---|
| json_multistep | 5 | 0.400 | 2.603 | 1.800 | Inconsistent self-checks — see Finding 1 |
| stateful_followup | 2 | 1.000 | 0.290 | 2.000 | Both turns exact match |
| toolcall_only | 2 | 1.000 | 0.310 | 2.000 | Correct values; action key normalized — see Finding 4 |
| mixed_brief_json | 2 | 1.000 | 0.370 | 2.000 | Clean ANSWER + JSON |
| toolcall | 2 | 1.000 | 0.485 | 11.000 | Stage-1 and final correct — no EOS contamination |
| json | 4 | n/a | 1.203 | 10.000 | All pass |
| fuzz | 20 | n/a | 0.924 | 10.000 | All 20 pass |
| mcq | 5 | n/a | 0.020 | 0.600 | 3/5 — see Finding 2 |
Key Findings
Finding 1: Multi-Step Planning — Inconsistent Self-Checks (0.400)
json_multistep is the weakest family for this checkpoint. The output format is always valid (schema_ok 1.000), but the model's self-check reasoning is frequently internally inconsistent (checks_consistent_ok 0.400) and it tends to STOP early.
Case-level breakdown:
| Case | Difficulty | Result | Got Plan | Expected | Failure |
|---|
| ms_easy_01 | Easy | ✅ | ["A"] | ["A"] | — |
| ms_easy_02 | Easy | ❌ | ["A","STOP"] | ["A","A"] | premature STOP; cc=0 |
| ms_med_01 | Medium | ✅ | ["A","B","C"] | ["A","B","C"] | — |
| ms_med_02 | Medium | ❌ | ["A","B","C"] | ["A","B","C"] | correct plan, but cc=0 / stop=0 |
| ms_hard_01 | Hard | ❌ | ["A","A","STOP"] | ["A","B","C"] | wrong plan; cc=0 |
ms_med_02 is the telling case: the model reaches the correct final plan (oracle_equiv_ok=1) but with an inconsistent reasoning trace and incorrect stop semantics, so it does not pass. The model lands the right plan only about half the time, and its intermediate reasoning is unreliable.
What this means for production: do not depend on this model for unassisted multi-step planning. Wrap it with an external planner/oracle-validation loop.
Finding 2: MCQ — 3/5, Mild A-Bias
| Case | Result | Raw |
|---|
| mcq_01 | ✅ | B |
| mcq_02 | ❌ | A |
| mcq_03 | ✅ | C |
| mcq_04 | ✅ | B |
| mcq_05 | ❌ | A |
Extraction is clean (single letters); both misses defaulted to A. This is a genuine capability result, not a parsing artifact — a mild A-bias on harder items.
Finding 3: toolcall — Correct and Clean (No EOS Contamination)
Both toolcall cases pass stage-1 and stage-2 for a perfect bucket score of 11.000:
| Case | Raw | Final | Result |
|---|
| tool_01 | {"tool_name": "add", "args": {"a": 2, "b": 3}} ⏎ 5 | 5 | ✅ |
| tool_02 | {"tool_name": "add", "args": {"a": 10, "b": -4}} ⏎ 6 | 6 | ✅ |
When the tool schema is scaffolded, the model emits the canonical tool_name/args structure. vLLM decodes without special-token leakage, so the final answer is clean — no EOS stripping required downstream.
Finding 4: toolcall_only — Correct Values, ReAct action Key (1.000)
In the bare tool-call setting the model free-forms to the ReAct convention:
| Case | Raw |
|---|
| toolonly_01 | {"action": "add", "args": {"x": 5, "y": 10}} |
| toolonly_02 | {"action": "add", "args": {"x": 25, "y": 75}} |
The tool and argument values are correct; the model names the tool under "action" (ReAct) instead of "tool_name", and uses x/y for operands. quant_eval v7.21 normalizes tool-name aliases (tool_name / tool / name / action / function) and arg keys (x/y → a/b), so tool_name_ok=1.000 and args_ok=1.000. The outer wrapper still differs from the strict schema (schema_ok=0, hence bucket 2.000). If your downstream stack does not normalize, enforce the exact tool_name key via system prompt — note the model emits canonical tool_name when the schema is scaffolded (Finding 3).
Finding 5: Stateful and Hybrid — Clean and Fast
stateful_followup (1.000) and mixed_brief_json (1.000) pass cleanly at sub-0.4-second latency, no EOS contamination:
| Family | Case | Raw |
|---|
| stateful | state_01 | {"counter": 2} → {"counter": 5} |
| stateful | state_02 | {"items": ["a","b"]} → {"items": ["a","b","c"]} |
| mixed | mixed_01 | ANSWER: 13 ⏎ {"a": 4, "b": 9, "sum": 13} |
| mixed | mixed_02 | ANSWER: 6 ⏎ {"a": -2, "b": 8, "sum": 6} |
Signal-Level Diagnostics (AWQ W4A16)
json_multistep
| Signal | Rate | Notes |
|---|
| schema_ok | 1.000 | Perfect |
| checks_consistent_ok | 0.400 | Inconsistent self-checks |
| stop_semantics_ok | 0.800 | One stop-semantics miss |
| oracle_equiv_ok | 0.600 | Right final plan 3/5 |
toolcall_only
| Signal | Rate | Notes |
|---|
| tool_name_ok | 1.000 | add recognized (action aliased) |
| args_ok | 1.000 | Values correct (x/y → a/b) |
stateful_followup / mixed_brief_json
All gating signals (turn1/2_parse_ok, turn1/2_exact_match; answer_line_ok, json_parse_ok, schema_ok) = 1.000.
Recommended Use Cases
✅ Deploy with Confidence (AWQ W4A16)
- Tool-calling agents —
toolcall 11.000 (clean), toolcall_only 1.000. Canonical tool output when scaffolded; no EOS stripping required.
- Stateful multi-turn agents — 1.000 at both turns, ~0.29 sec/case.
- Hybrid brief + JSON responses —
mixed_brief_json 1.000.
- Structured JSON outputs (single-step) —
json and fuzz both bucket=10.000.
- Latency-sensitive / cost-sensitive deployment — 12B in ~8 GB, sub-second per evaluated case on vLLM; fits modest GPUs.
⚠️ Use with Guardrails (AWQ W4A16)
- Multi-step planning — 0.400 pass rate, inconsistent self-checks. Add an external planner/oracle-validation loop.
- MCQ / single-choice — 3/5 with a mild A-bias; verify on your option distributions.
- Bare tool-call dispatch in non-normalizing stacks — the model defaults to the ReAct
action key. Enforce tool_name via system prompt; values are already correct.
Hardware Requirements
| Configuration | VRAM Required | Notes |
|---|
| W4A16 (this repo) · modest context | ~10–12 GB | ~8 GB weights + KV cache |
| W4A16 · larger context (32K+) | ~16–24 GB | Raise --max-model-len; KV cache grows |
| FP16 (not in this repo) | ~24 GB+ | Base Mistral-Nemo-Instruct-2407 |
Usage
Requires the vLLM-compatible stack: vLLM 0.8.5.post1, transformers 4.51.3, tokenizers 0.21.1, huggingface_hub 0.30.2 in a dedicated environment. Newer transformers removes all_special_tokens_extended, which vLLM 0.8.5.post1 needs at startup.
Serving — vLLM async server (OpenAI-compatible)
1pip install "vllm==0.8.5.post1" "transformers==4.51.3" "tokenizers==0.21.1" "huggingface_hub==0.30.2"
2
3vllm serve pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async \
4 --quantization compressed-tensors \
5 --dtype float16 \
6 --max-model-len 4096
1from openai import OpenAI
2
3client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-required")
4resp = client.chat.completions.create(
5 model="pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async",
6 messages=[
7 {"role": "system", "content": "You are a precise assistant. Return structured outputs when requested."},
8 {"role": "user", "content": "Return a JSON object with keys: summary, risk_level, action_items."},
9 ],
10 temperature=0.3,
11)
12print(resp.choices[0].message.content)
Offline — vLLM LLM (synchronous, batched)
1from vllm import LLM, SamplingParams
2
3llm = LLM(
4 model="pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async",
5 quantization="compressed-tensors",
6 dtype="float16",
7 max_model_len=2560,
8 gpu_memory_utilization=0.90,
9)
10# Mistral chat format: <s>[INST] ... [/INST]
11prompts = ["<s>[INST] Briefly describe synthetic data generation. [/INST]"]
12out = llm.generate(prompts, SamplingParams(max_tokens=128, temperature=0.3, top_p=0.92))
13print(out[0].outputs[0].text)
Bare tool-call dispatch — enforce tool_name if your stack does not normalize
1SYS = (
2 'Respond ONLY with a JSON object using EXACTLY these keys:\n'
3 '{"tool_name": "add", "args": {"a": <integer>, "b": <integer>}}\n'
4 'Do not use "action" or "x"/"y". No other text, no markdown.'
5)
6# Send SYS as the system message; the model emits canonical tool_name/args when constrained.
Evaluation Artifacts
The full per-case evaluation CSV (comparison_results_v7_21_Mistral_Nemo_Instruct_2407_AWQ_20260529_152601_20260629_184555.csv) and rollup.json are published in this repository for independent verification.
Artifact Provenance
| Artifact | Format | Size (bytes) | SHA256 |
|---|
model-00001-of-00002.safetensors | compressed-tensors W4A16 | 4,987,280,248 | 85b1041123245e5876912c4ad4d2fef0b7f740b116f91631f44be8b059b7505a |
model-00002-of-00002.safetensors | compressed-tensors W4A16 | 3,363,635,168 | 075a19ac17db4a32b5adfde7bb6727e7996c6196aff22cdc65038d17dbb6e5f0 |
model.safetensors.index.json | JSON | 106,519 | b678bf572b88b130e3a3a51369ef699aaad9a8fe084bf6e044ab2aa7ed6cd215 |
quantization_manifest.json | JSON | 2,831 | d5dabea10e791f0c921c2c678e9f172cadac00bc548b0f3d674a79a76ddbcc3c |
recipe.yaml | YAML | 865 | 2ab77dc9f98273b7d01a83b3805f41c73e2f47cfa2d65d3aa34f6ffd4a1bd376 |
Total quantized weights: 8,350,915,416 bytes (~8.35 GB). Per-file SHA256 for every file in the repository is published in file_hashes.json for independent verification. Quantized from mistralai/Mistral-Nemo-Instruct-2407 with llmcompressor (AWQ, compressed-tensors W4A16-asymmetric). The FP16 base was not evaluated.
Evaluation Methodology
quant_eval v7.21 — proprietary behavioral evaluation harness, PBH Applied Systems.
Fixture set: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c079371b02a94f3c149eb78a6adc03dc16ff6833b964fbf4174f0)
| Family | Description | Pass Signals |
|---|
fuzz | Property-based regression; structured placement correctness | schema_ok, constraints_ok |
json | Single-step structured JSON with constraint rules | schema_ok, constraints_ok |
json_multistep | Multi-step planning with self-check and oracle verification | schema_ok, checks_consistent_ok, stop_semantics_ok, oracle_equiv_ok |
mcq | Multiple-choice extraction | choice_ok |
stateful_followup | Two-turn state tracking; turn-2 correct given turn-1 | turn1/2_parse_ok, turn1/2_exact_match |
mixed_brief_json | Hybrid: natural language answer + valid JSON block | answer_line_ok, json_parse_ok, schema_ok |
toolcall | Tool call embedded in response; parse + schema + final value | stage1_tool_parse_ok, stage1_tool_schema_ok |
toolcall_only | Bare schema-only tool call; tool name + args (alias-normalized) | tool_name_ok, args_ok |
Evaluation hardware: NVIDIA RTX 4090 (24 GB) · Runner: quantized_vllm (compressed-tensors W4A16) · Evaluation date: June 29, 2026 · Seed: 42
About PBH Applied Systems
PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development.
Patrick Hill, M.S. — Founder · Data Scientist · AI/ML Engineer · Author of
Applied Machine Learning: Concepts, Tools, and Case Studies (required reading, UAT CSC 373)
Core Service Areas: LLM Optimization & Deployment · AI Evaluation Frameworks · Agentic AI Infrastructure · Scalable AI Application Development · ML Pipeline Design & Analytics · Model & Agent Cataloging
🔬 About quant_eval & This Evaluation Series
quant_eval is a proprietary behavioral evaluation harness developed by
PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under
pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.
Need a deployment recommendation?
Not sure which quantization — AWQ, NF4, GGUF — is right for your hardware, latency target, or agent type?
→ pbhappliedsystems.com
📞 Work With PBH Applied Systems
This AWQ build is the series' fast, small-footprint agent backend: a 12B model in ~8 GB serving sub-second on vLLM with clean tool-call output. The evaluation tells you exactly where to lean on it (tool dispatch, stateful, structured JSON) and where to add guardrails (multi-step planning, MCQ).
Connect
License
This repository inherits the license of the base model:
Apache 2.0 —
mistralai/Mistral-Nemo-Instruct-2407The quant_eval evaluation methodology, fixture set, and scoring framework are proprietary to PBH Applied Systems, LLC and are not included in this repository.
AWQ / compressed-tensors quantization and behavioral evaluation performed by PBH Applied Systems, LLC · quant_eval v7.21 · Run ID: 20260629_184555 
