Converted and evaluated by PBH Applied Systems, LLC
— Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure
🔬 This repository is part of a production-oriented evaluation series. Every model published under pbhappliedsystems has been independently evaluated using quant_eval v7.21 — a proprietary behavioral evaluation harness developed by PBH Applied Systems. Scores measure real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning families — not perplexity or benchmark leaderboard proxies.
📌 This is the full-precision F16 baseline repository. The evaluated Q4_K_M deployment variant is published at pbhappliedsystems/mistral-nemo-instruct-2407-gguf-Q4-K-M. That card documents the full F16 vs. Q4_K_M comparison — including the json_multistep degradation (0.600 → 0.400), a complete ms_hard_01 breakdown at Q4_K_M, and a tool_02 final_mismatch finding that does not appear at F16.
This model is part of the PBH Applied Systems evaluated model series that supports the live AI Agent Demo. The demo lets visitors interact with production-style agent workflows powered by open-weight language models evaluated through PBH Applied Systems' quant_eval framework.
The F16 model serves a different role than the Q4_K_M deployment variant. F16 is the full-precision baseline used to measure what the model can do before quantization. quant_eval then compares the quantized model against this baseline to identify which capabilities are preserved, which degrade, and which tasks require guardrails or a higher-precision deployment.
This comparison is central to the demo. It helps determine which model belongs in which agent role:
Reasoning models are selected for planning, analysis, and auditable decision workflows.
Document models are selected for long-context extraction, summarization, and structured Q&A.
Code models are selected for task completion, structured output, API scaffolding, and automation workflows.
Quantized variants are selected when they preserve enough behavior to reduce cost, latency, and GPU requirements.
F16 variants remain important when maximum fidelity, cleaner tool execution, or reduced quantization risk matters more than speed or cost.
The live demo shows the deployment side of that process. The F16 card documents the reference behavior. The Q4_K_M card shows what changes after compression. Together, they explain how PBH Applied Systems uses quant_eval to choose the correct LLM for the correct agent type instead of guessing from model size or leaderboard reputation.
Model Description
This repository contains the full-precision F16 GGUF of mistralai/Mistral-Nemo-Instruct-2407, a 12-billion parameter instruction-tuned model developed by Mistral AI in collaboration with NVIDIA (July 2024 release). Mistral-Nemo features the Tekken tokenizer and a 128,000-token context window — the largest context in the PBH Applied Systems evaluated series outside of the Qwen2.5-14B-Instruct-1M, which supports a 1 million token window.
The F16 format preserves all original float16 weights without quantization. In the PBH Applied Systems evaluation pipeline, this F16 run (20260211_014604) served as the baseline cache generation pass — producing the full_weight_cache.json used as the reference anchor for the subsequent Q4_K_M comparison run (20260211_022944).
For most production deployments, the Q4_K_M variant is the appropriate choice. The F16 is the right choice when maximum output fidelity, clean tool-execution pipelines, and the full 128K context window at maximum precision are required.
Inference speed (eval hardware): avg 30.24 sec/case on RTX 4090
Multilingual: English, French, German, Spanish, Italian, Portuguese, Russian, Chinese, Japanese
Why 30.24 sec/case avg? This is a 12B non-reasoning model at full F16 precision. The json family averaged 64.02 sec/case due to multi-step token generation, with json_01 reaching 149.88 seconds — a significant single-case outlier visible in both F16 and Q4_K_M runs on this model. The Q4_K_M variant averages 1.42 sec/case (21.3× faster).
PBH Applied Systems Evaluation — quant_eval v7.21
Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.21
Run ID: 20260211_014604 · Fixtures: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c...) · Seed: 42
Hardware: NVIDIA RTX 4090 · Runner: full_weight_transformers (F16 only) · Total rows: 42
Note on aggregate scores: The normalized aggregate dimensions (task completion, reasoning, coherence, instruction following) are computed from the combined F16 + Q4_K_M comparison run and are reported on the Q4_K_M card. This F16 card reports per-family pass rates from the full_weight_transformers runner.
F16 passes both medium cases. At Q4_K_M, ms_med_02 additionally fails on checks_consistent_ok, and ms_hard_01 degrades from a partial failure (cc+oracle) to a complete failure across all four gating signals. This is the direct capability argument for F16 in planning-adjacent workloads.
The json_multistep failures at F16 are scoped: ms_easy_02 fails only on oracle equivalence (the model plans incorrectly but is internally consistent), and ms_hard_01 fails on both consistency and oracle (a harder failure that tracks with the difficulty level). No schema or STOP semantics failures occur at F16.
toolcall — Fully Clean at F16
Both tool_01 and tool_02 pass at bucket=11 — the maximum score. Tool parse is valid, schema is valid, and the final answer matches expected output. This contrasts with the Q4_K_M variant where tool_02 produces a final_mismatch (bucket=0) despite a valid tool dispatch.
Case
F16 bucket
F16 detail
Q4_K_M bucket
Q4_K_M detail
tool_01
11
ok
11
ok
tool_02
11
ok
0
final_mismatch
At F16, tool dispatch and post-tool answer accuracy are both reliable. If your application depends on the model correctly processing tool outputs and reporting accurate results — not just calling the right tool — F16 is the safer choice.
toolcall_only passes gating at 1.000 (tool_name_ok=1, args_ok=1) but carries schema_ok=0 on both cases with detail=schema_error. The model correctly identifies the tool and extracts valid arguments but wraps the output using "tool" as the outer key instead of the expected "tool_name".
Signal
Rate
Tier
tool_name_ok
1.000
Tier-1 (gating) ✅
args_ok
1.000
Tier-1 (gating) ✅
schema_ok
0.000
Non-gating (tracked)
This is a schema discipline issue, not a capability failure. A one-line normalization step ("tool" → "tool_name") in the response parser resolves it for strict schema enforcement environments. See the Q4_K_M companion card for the normalize_tool_wrapper() implementation pattern.
MCQ — A-Bias at F16
mcq_02 and mcq_05 both fail with wrong_choice got=A. Both failures occur at 0.46 seconds — fast, confident, and wrong. This model defaults to option A when uncertain. The Q4_K_M variant adds mcq_04 to the failure set (also got=A), but the A-bias is a model-level characteristic present at full precision, not a quantization artifact.
Case
Result
Detail
mcq_01
✅ PASS
ok
mcq_02
❌ FAIL
wrong_choice got=A
mcq_03
✅ PASS
ok
mcq_04
✅ PASS
ok
mcq_05
❌ FAIL
wrong_choice got=A
Signal-Level Diagnostics (F16)
json_multistep
Signal
Rate
Tier
schema_ok
1.000
Tier-1 (gating)
checks_consistent_ok
0.800
Tier-1 (gating)
stop_semantics_ok
1.000
Tier-1 (gating)
oracle_equiv_ok
0.600
Tier-1 (gating)
final_consistent_ok
0.000
Tier-2 (tracked, non-gating)
final_match_reported
0.000
Tier-2 (tracked, non-gating)
schema_ok=1.000 and stop_semantics_ok=1.000 at F16 — both drop to 0.800 at Q4_K_M. These signal-level regressions are what drive the pass rate from 0.600 to 0.400 under quantization.
stateful_followup
Signal
Rate
turn1_parse_ok
1.000
turn2_parse_ok
1.000
turn1_exact_match
1.000
turn2_exact_match
1.000
toolcall_only
Signal
Rate
Tier
tool_name_ok
1.000
Tier-1 (gating)
args_ok
1.000
Tier-1 (gating)
schema_ok
0.000
Non-gating (tracked)
mixed_brief_json
Signal
Rate
answer_line_ok
1.000
json_parse_ok
1.000
schema_ok
1.000
When to Deploy F16 vs. Q4_K_M
Criterion
F16 (this repo)
Q4_K_M
VRAM available
~26 GB
~10 GB
Latency acceptable
~30 sec/case avg
~1.4 sec/case avg
json_multistep pass rate
0.600
0.400
ms_med_02
✅ PASS
❌ FAIL
ms_hard_01
❌ partial fail (cc+oracle)
❌ complete fail (all 4)
toolcall accuracy
✅ Clean (both bucket=11)
⚠️ tool_02 final_mismatch
toolcall_only gating
✅ 1.000
❌ 0.000 (args fail)
Schema wrapper norm
Needed
Not applicable
MCQ failures
2/5 (mcq_02, mcq_05)
3/5 (+mcq_04)
128K context at full precision
✅
Limited by VRAM
Recommended Use Cases — F16
✅ Deploy with Confidence
Tool-calling pipelines requiring answer accuracy — Both toolcall cases pass at bucket=11 with no final_mismatch. F16 is the correct choice when tool execution results feed downstream computation that must be correct.
Stateful multi-turn agents — Perfect two-turn state retention (1.000) at 7.60 sec/case avg.
Structured JSON outputs (single-step) — json and fuzz both achieve bucket_score 10.000.
Hybrid brief + JSON responses — mixed_brief_json passes at 1.000.
Medium-difficulty multi-step planning — ms_med_01 and ms_med_02 both pass. F16 retains ms_med_02 that Q4_K_M loses.
Long-document processing at full precision — 128K context with F16 weights provides maximum fidelity for large-scale document Q&A, multi-document comparison, and long-form extraction.
Multilingual structured tasks — All 9 supported languages at maximum precision.
Tool-only dispatch with schema normalization — toolcall_only passes gating at 1.000 with a one-line wrapper key fix.
⚠️ Use with Guardrails
Hard multi-step planning — ms_hard_01 fails at F16 too (cc+oracle), though less catastrophically than at Q4_K_M. Use with an external validator for hard planning tasks.
MCQ with A-bias mitigation — Two of five cases fail at F16 with wrong_choice got=A. Add chain-of-thought prompting or response validation for MCQ-style pipelines.
❌ Not Recommended
High-throughput pipelines — At 30.24 sec/case average and up to 149.88 seconds on a single json case, F16 is not suitable for latency-sensitive or batch workloads.
1from huggingface_hub import hf_hub_download
2from llama_cpp import Llama
34# Note: 24.5 GB download — ensure sufficient disk space and ~26 GB VRAM5model_path = hf_hub_download(6 repo_id="pbhappliedsystems/mistral-nemo-instruct-2407-gguf-F16",7 filename="mistral-nemo-instruct-2407-gguf-F16.gguf"8)910llm = Llama(11 model_path=model_path,12 n_ctx=32768,# Adjust to use case; supports up to 128K13 n_gpu_layers=-1,# -1 offloads all layers to GPU; reduce if VRAM < 26 GB14 verbose=False,15)1617response = llm.create_chat_completion(18 messages=[19{20"role":"system",21"content":"You are a precise assistant. Follow instructions exactly and return structured outputs when requested."22},23{24"role":"user",25"content":"Analyze the following document and return a JSON object with keys: summary, key_entities, sentiment, action_items."26}27],28 temperature=0.3,29 max_tokens=1024,30)3132print(response["choices"][0]["message"]["content"])
For partial GPU offload when VRAM is between 16–24 GB:
python
1llm = Llama(2 model_path=model_path,3 n_ctx=16384,4 n_gpu_layers=20,# Tune based on available VRAM5 verbose=True,# Enable to monitor layer offload and memory usage6)
For tool-calling with schema normalization (addresses the toolcall_only wrapper issue):
The F16 GGUF was converted from the mistralai/Mistral-Nemo-Instruct-2407 HuggingFace snapshot using a custom-built llama.cpp conversion pipeline developed by PBH Applied Systems, without modification to model weights.
Two-pass evaluation architecture: The F16 evaluation run (20260211_014604) operated in cache-generation mode (skip_quant=true), producing the full_weight_cache.json used as the reference baseline for the Q4_K_M comparison run (20260211_022944). This ensures that F16 and Q4_K_M results are measured against the identical fixture set under controlled, reproducible conditions.
Evaluation Methodology
quant_eval v7.21 is a proprietary behavioral evaluation harness developed by PBH Applied Systems. The two-run architecture evaluates the full-precision (F16) model first, caches its results, then evaluates the quantized variant against the same fixture set.
quant_eval is a proprietary behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.
See it in action:Live AI Agent Demo →
The demo runs production-style agent workflows powered by open-weight models selected through the quant_eval evaluation pipeline.
Need a deployment recommendation?
Not sure which quantization level is right for your hardware, latency target, or agent type?
→ pbhappliedsystems.com
PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development. The organization operates with a strong emphasis on engineering rigor, reproducibility, and real-world deployment constraints — particularly in environments where performance, cost efficiency, and reliability must be balanced against available hardware and budget.
Founder — Patrick Hill, M.S.
PBH Applied Systems was founded by Patrick Hill, a Data Scientist and AI/ML Engineer with 10+ years of experience delivering advanced analytics, predictive modeling, and decision-support solutions across high-stakes operational environments. Patrick holds a Master of Science in Software Engineering with concentrations in Artificial Intelligence and Machine Learning and a B.S. in Business Finance.
Technical expertise spans:
Languages & Data: Python, SQL, Linux, Pandas, NumPy, scikit-learn
ML & Modeling: Supervised and unsupervised learning, neural networks, NLP, transformers, regression, classification, forecasting, and feature engineering
1. LLM Optimization & Deployment — End-to-end GGUF conversion and quantization with custom llama.cpp pipelines and adapter-per-model architecture.
2. AI Evaluation Frameworks — Proprietary behavioral evaluation via quant_eval: per-family pass rates, F16 vs. quantized delta analysis, failure cluster diagnostics, deployment recommendations.
3. Agentic AI Infrastructure — LlamaIndex ReAct agents, Flask orchestration, serverless GPU inference, full pipeline from model selection to production serving.
4. Scalable AI Application Development — Multimodal applications (quantized LLMs + Whisper + BLIP), Dockerized Flask APIs, advanced time-series forecasting with custom attention mechanisms, Bayesian hyperparameter optimization, and FinBERT sentiment fusion.
5. ML Pipeline Design & Analytics — Feature engineering, forward-chaining cross-validation, KPI dashboards, analytical governance at scale.
6. Model & Agent Cataloging — Structured catalog publishing with reproducible artifacts and clear performance tradeoff documentation.
📞 Work With PBH Applied Systems
This F16 card documents clean tool execution that degrades at Q4_K_M, medium planning cases that survive F16 but fail under quantization, and an MCQ A-bias that is a model characteristic — not a precision artifact. The Q4_K_M companion card maps exactly where each of these findings changes under quantization. The decision between F16 and Q4_K_M should be made with this data, not guessed at.
👉 Book a Scoping Call — Discuss your model selection, quantization strategy, or deployment architecture directly with Patrick.
👉 Request an Evaluation Report — A full quant_eval behavioral audit: per-family pass rates, F16 vs. quantized delta analysis, failure cluster diagnostics, and a deployment recommendation. Engagements from $2,500.
The quant_eval evaluation methodology, fixture set, and scoring framework are proprietary to PBH Applied Systems, LLC and are not included in this repository.
GGUF conversion and behavioral evaluation performed by PBH Applied Systems, LLC · quant_eval v7.21 · F16 Run ID: 20260211_014604