Quantized, converted, and evaluated by PBH Applied Systems, LLC — Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure

🔬 This repository is part of a production-oriented evaluation series. Every model published under pbhappliedsystems has been independently evaluated using quant_eval v7.21 — a proprietary behavioral evaluation harness developed by PBH Applied Systems. Scores measure real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning families — not perplexity or benchmark leaderboard proxies.

⚖️ License Notice: This model is governed by the Qwen Research License, which permits non-commercial use only. Commercial use requires a separate license from Alibaba Cloud. See the License section for full details.

Try This Model in the Live AI Agent Demo

Launch the PBH Applied Systems AI Agent Demo →

This model is part of the PBH Applied Systems live AI Agent Demo, where visitors can test evaluated quantized open-weight models across production-style agent workflows: reasoning and analysis, document intelligence, and code automation.

The demo uses quant_eval results to show how model selection changes by task. A model that performs well for long-context document analysis may not be the best choice for hard multi-step planning, strict tool-use workflows, or production code generation. Each deployed model is evaluated for practical agent behavior, including coherence, instruction following, reasoning, task completion, structured output reliability, tool-use behavior, and quantization impact.

For this repository, the Q4_K_M variant represents the deployment-focused model: smaller, faster, and more cost-efficient than the F16 baseline. The evaluation results below explain where this quantized model preserves useful behavior, where quantization introduces risk, and what guardrails are recommended before production deployment.

The purpose of the demo is simple: let prospects test the same kind of evaluated quantized models that PBH Applied Systems deploys for real agentic AI systems.

Model Description

This repository contains the 4-bit quantized (Q4_K_M) GGUF of Qwen/Qwen2.5-3B-Instruct, a 3-billion parameter instruction-tuned model from Alibaba Cloud (September 2024 release). Qwen2.5-3B-Instruct is the smallest model in the PBH Applied Systems evaluated series, and at Q4_K_M precision it represents the most hardware-accessible deployment option — requiring as little as 4 GB VRAM and fitting on consumer edge hardware.

The full-precision F16 baseline is published separately at pbhappliedsystems/qwen-2.5-3B-instruct-gguf-F16.

Key Characteristics

Parameters: 3B
Format: GGUF Q4_K_M
File size: 1.93 GB
SHA256: 9ab3bc9beaddaec3700d5cc754b52e1501a3fd172bc7fc3ee3eb8e1d388ee043
Minimum VRAM (GPU inference): ~4 GB
Recommended GPU tier: Any CUDA-capable GPU · CPU inference viable
Context window: 32,768 tokens
Inference speed (eval hardware): avg 0.390 sec/case on RTX 4090 — fastest Q4_K_M in the evaluated series
License: Qwen Research License (non-commercial)

PBH Applied Systems Evaluation — quant_eval v7.21

Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.21 Run ID: 20260221_041137 · Fixtures: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c...) · Seed: 42 Hardware: NVIDIA RTX 4090 · Total rows evaluated: 84 (42 F16 · 42 Q4_K_M)

Aggregate Scores (Q4_K_M)

Scores are normalized to [0.0 – 1.0]. Higher is better.

Dimension	Score	Series Context
Task Completion	0.4905	Expected at 3B scale
Reasoning	0.3704	Limited by parameter count
Coherence	0.9074	Strong at 3B
Instruction Following	0.6599	Moderate
Avg inference time	0.390 sec/case	Fastest in series

Per-Family Pass Rates

F16 Baseline (`full_weight_transformers`)

Family	N	Pass Rate	Avg Secs	Notes
json_multistep	5	0.000	3.704	checks_consistent_ok bottleneck
stateful_followup	2	1.000	0.525	Both turns exact match
toolcall_only	2	0.000	0.405	tool_name_ok=1, args_ok=0
mixed_brief_json	2	0.000	0.695	JSON valid; ANSWER line absent — see note
toolcall	2	1.000	0.745	Stage-1 passes; final_mismatch — see note
json	4	n/a	2.072	bucket_score avg = 10.000
fuzz	20	n/a	1.437	bucket_score avg = 7.500 — 15/20 pass
mcq	5	n/a	0.022	Empty raw output on all 5

Q4_K_M (`quantized_llama_cpp`)

Family	N	Pass Rate	Δ vs F16	Avg Secs	Notes
json_multistep	5	0.200	+0.200	0.985	ms_easy_01 recovers
stateful_followup	2	1.000	0.000	0.125	Perfect retention
toolcall_only	2	0.000	0.000	0.100	Wrong arg key names
mixed_brief_json	2	1.000	+1.000	0.155	Full recovery at Q4_K_M
toolcall	2	1.000	0.000	0.215	Stage-1 passes; final_mismatch — see note
json	4	n/a	—	0.510	bucket_score avg = 10.000
fuzz	20	n/a	—	0.408	bucket_score avg = 7.500 — same 15/20
mcq	5	n/a	—	0.010	3/5 pass at Q4_K_M

Key Findings

Finding 1: The Speed Story — Fastest Model in the Evaluated Series

At 0.390 sec/case average, this Q4_K_M variant is the fastest model evaluated in the PBH Applied Systems series. Individual family timings:

Family	Q4_K_M Avg Secs
mcq	0.010
stateful_followup	0.125
toolcall_only	0.100
mixed_brief_json	0.155
toolcall	0.215
fuzz	0.408
json	0.510
json_multistep	0.985

MCQ responses at 10 milliseconds. Stateful follow-up at 125 milliseconds. Even the most complex planning cases complete in under one second. At 1.93 GB, this model runs on hardware where no other model in this series fits — including systems without dedicated GPUs.

This speed profile is the primary deployment argument for Qwen2.5-3B Q4_K_M. It is not the most capable model evaluated. It is the most accessible model evaluated, with a capability ceiling that is clearly defined by the evaluation data below.

Finding 2: checks_consistent_ok = 0.200 — The 3B Ceiling

The single most consistent failure signal across both runners and all json_multistep cases is checks_consistent_ok. Both runners achieve the same rate (0.200) — 1/5 cases pass. This is the signal that measures whether the model's intermediate reasoning steps are internally self-consistent.

This is a parameter-count limitation, not a quantization artifact. The same 5 fuzz cases fail on both runners (fuzz_0001, _0002, _0003, _0010, _0012) at the same bucket scores, confirming the failure boundary is set by the model's capacity, not by the precision level. A 3B model that can produce valid JSON schemas reliably but cannot maintain consistent multi-step reasoning chains is behaving exactly as expected at this scale.

Signal	F16 Rate	Q4_K_M Rate
schema_ok	0.800	1.000
checks_consistent_ok	0.200	0.200
stop_semantics_ok	0.200	0.400
oracle_equiv_ok	0.200	0.400

Q4_K_M actually improves over F16 on schema_ok (0.800 → 1.000), stop_semantics_ok (0.200 → 0.400), and oracle_equiv_ok (0.200 → 0.400). The checks_consistent_ok signal, which is the deepest reasoning consistency test, holds at 0.200 regardless of precision.

Finding 3: F16 Role-Token Contamination

The F16 evaluation reveals a specific issue with how the HuggingFace Transformers runner handles this model's chat template. Raw outputs for several families show truncated role token prefixes:

toolcall cases: ician {"tool_name": "add", ...} — the role token "technician" is truncated to "ician" and appears in the response body
stateful cases: ician {"counter": 2} — same prefix contamination
mixed_brief_json: user ANSWER: 13 {...} — the "user" role token appears as literal text

The JSON content in these cases is correct, but the role prefix causes extraction failures. This explains why toolcall shows final_mismatch at F16 (the stage-1 JSON is valid but the post-response text includes role tokens), and why mixed_brief_json fails answer_line_ok at F16 despite having json_parse_ok=1 and schema_ok=1.

This is a runner configuration issue, not a model quality issue. The Q4_K_M llama.cpp runner does not exhibit this behavior.

Finding 4: F16 ms_easy_01 — Wrong Output Type

ms_easy_01 at F16 is the only case in the evaluated series where a model returns an entirely wrong output type for json_multistep. The F16 model produces:

[{"shelf": "A", "item": "P", "can_place": 1}]

An array, where the task requires a specific schema object with plan, checks, and final keys. This causes schema_ok=0 — the only F16 json_multistep case with a schema failure. At Q4_K_M, ms_easy_01 recovers: schema_ok=1, oracle_equiv_ok=1, checks_consistent_ok=1 — a clean pass.

Finding 5: toolcall_only — Schema Vocabulary Errors

Both toolcall_only cases at Q4_K_M show a consistent vocabulary mismatch:

Case	Q4_K_M Raw Output	Expected
toolonly_01	`{"tool": "add", "operands": [5, 10]}`	`{"tool_name": "add", "args": {"a": 5, "b": 10}}`
toolonly_02	`{"tool": "add", "operands": [25, 75]}`	`{"tool_name": "add", "args": {"a": 25, "b": 75}}`

The model uses "tool" instead of "tool_name" and "operands" (array) instead of "args" (object with named keys). Tool name recognition passes (tool_name_ok=1) because the extractor finds "add" in the payload. Argument extraction fails because "operands": [5, 10] is not a valid schema for {"a": integer, "b": integer}.

This is a schema vocabulary issue at 3B scale — the model knows the tool concept but not the exact field name schema. A system prompt that explicitly states the required key names (tool_name, args.a, args.b) would likely resolve this for basic tools.

Finding 6: toolcall — Correct Arithmetic, EOS Contamination

Both toolcall cases at Q4_K_M pass stage-1 (tool dispatch valid) but produce final_mismatch:

Case	Raw Output	Expected
tool_01	`{...add(2,3)...}<\|im_end\|> 5<\|im_end\|>`	`5`
tool_02	`{...add(10,-4)...}<\|im_end\|> 6<\|im_end\|>`	`6`

The arithmetic is correct. The EOS token (<|im_end|>) appended to the answer string causes the string comparison to fail. A re.sub(r'<\|im_end\|>', '', raw) strip resolves it. See the Usage section for the validated extraction pattern.

Finding 7: MCQ — A-Bias Pattern

Case	Q4_K_M Result	Raw
mcq_01	✅ PASS	`B`
mcq_02	❌ FAIL	`A` (wrong)
mcq_03	✅ PASS	`C`
mcq_04	✅ PASS	`B`
mcq_05	❌ FAIL	`A` (wrong)

The two failures both produce A. The same A-bias pattern observed in Mistral-Nemo appears here at 3B scale. At F16, all 5 MCQ cases produce empty raw output (raw=''), suggesting the F16 runner cannot extract the choice from the model's response format at this parameter count.

Signal-Level Diagnostics

Q4_K_M — json_multistep

Signal	F16 Rate	Q4_K_M Rate	Delta
schema_ok	0.800	1.000	+0.200
checks_consistent_ok	0.200	0.200	0.000
stop_semantics_ok	0.200	0.400	+0.200
oracle_equiv_ok	0.200	0.400	+0.200

Q4_K_M — stateful_followup

Signal	Rate
turn1_parse_ok	1.000
turn2_parse_ok	1.000
turn1_exact_match	1.000
turn2_exact_match	1.000

Q4_K_M — toolcall_only

Signal	Rate
tool_name_ok	1.000
args_ok	0.000

Q4_K_M — mixed_brief_json

Signal	Rate
answer_line_ok	1.000
json_parse_ok	1.000
schema_ok	1.000

Recommended Use Cases

✅ Deploy with Confidence (Q4_K_M, Non-Commercial)

Stateful multi-turn agents — Perfect two-turn retention (1.000) at 0.125 sec/case. The fastest stateful performance in the evaluated series.
Structured JSON outputs (single-step) — json bucket_score 10.000, fuzz bucket_score 7.500. Fast and reliable for constraint-adherent single-step placements.
Hybrid brief + JSON responses — mixed_brief_json passes at 1.000 in 0.155 sec. Clean ANSWER line + valid JSON.
Edge and resource-constrained deployment — 1.93 GB at Q4_K_M. Runs on 4 GB VRAM, embedded GPUs, and CPU-only environments where no other evaluated model fits.
High-throughput batch processing — At 0.390 sec/case average, this model can process thousands of structured inference tasks per hour on modest hardware.

⚠️ Use with Guardrails (Q4_K_M)

Scaffolded tool-calling — toolcall stage-1 passes at 1.000 but add EOS stripping before final answer extraction. Arithmetic results are correct.
Bare tool-call dispatch — toolcall_only fails on args schema ("operands" vs "args"). Provide explicit schema examples in the system prompt or add a normalization layer.
Multi-step planning (easy difficulty only) — ms_easy_01 passes. Higher difficulties fail consistently on checks_consistent_ok. Use with external validation.
MCQ applications — 3/5 pass but A-bias exists. Add chain-of-thought prompting or validation for MCQ pipelines.

❌ Not Recommended (Q4_K_M)

Medium-to-hard multi-step planning — ms_med and ms_hard cases fail on internal consistency. The 3B parameter count is the limiting factor, not the quantization.
Any commercial use without a Qwen commercial license — The Qwen Research License restricts use to non-commercial research and evaluation purposes. Contact Alibaba Cloud for commercial licensing.

Hardware Requirements

Configuration	VRAM Required	Notes
Q4_K_M (this repo) · GPU	~4 GB	Fits on embedded/mobile GPUs
Q4_K_M · CPU only	4 GB RAM	Viable; slower inference
Q4_K_M · 32K context	~8 GB	Full context window
F16 (companion repo) · GPU	~8 GB	6.18 GB model + overhead

Usage

Installation

pip install llama-cpp-python huggingface_hub

For GPU acceleration (CUDA):

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir

Python — llama-cpp-python

python

from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="pbhappliedsystems/qwen-2.5-3B-instruct-gguf-Q4-K-M",
    filename="qwen-2.5-3B-instruct-gguf-Q4-K-M.gguf"
)

llm = Llama(
    model_path=model_path,
    n_ctx=4096,
    n_gpu_layers=-1,  # -1 for full GPU; set 0 for CPU-only
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": "You are a precise assistant. Return structured JSON when asked. Use exactly the key names specified."
        },
        {
            "role": "user",
            "content": "Summarize the following and return JSON with keys: summary, sentiment, action_items."
        }
    ],
    temperature=0.15,
    max_tokens=512,
)

print(response["choices"][0]["message"]["content"])

For CPU-only deployment (no GPU required):

python

llm = Llama(
    model_path=model_path,
    n_ctx=2048,
    n_gpu_layers=0,    # CPU-only
    n_threads=8,       # Tune to your CPU core count
    verbose=False,
)

For tool-calling with EOS stripping (addresses toolcall final_mismatch finding):

python

import json, re
from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="pbhappliedsystems/qwen-2.5-3B-instruct-gguf-Q4-K-M",
    filename="qwen-2.5-3B-instruct-gguf-Q4-K-M.gguf"
)

llm = Llama(model_path=model_path, n_ctx=2048, n_gpu_layers=-1, verbose=False)

def call_tool_with_cleanup(prompt: str) -> dict:
    """
    Tool dispatch with EOS stripping.
    quant_eval v7.21: toolcall stage-1 pass=1.000; final_mismatch due to <|im_end|> suffix.
    Arithmetic is correct — strip EOS before downstream processing.
    """
    response = llm.create_chat_completion(
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a tool-calling assistant. "
                    "When calling a tool, output JSON as: "
                    '{"tool_name": "<name>", "args": {"a": <n>, "b": <n>}}\n'
                    "Then on the next line, output the result as a plain number."
                )
            },
            {"role": "user", "content": prompt}
        ],
        temperature=0.0,
        max_tokens=128,
    )
    raw = response["choices"][0]["message"]["content"]
    # Strip EOS tokens before processing
    clean = re.sub(r'<\|im_end\|>', '', raw).strip()
    return {"raw": raw, "clean": clean}

result = call_tool_with_cleanup("Use the add tool to compute 5 plus 10.")
print(result["clean"])

CLI — llama-cli

bash

llama-cli \
  --model qwen-2.5-3B-instruct-gguf-Q4-K-M.gguf \
  --chat-template qwen2 \
  --system-prompt "You are a precise assistant. Use exactly the key names specified in any JSON schema." \
  --prompt "Return a JSON object with keys: summary, risk_level, action_items." \
  --n-predict 512 \
  --ctx-size 4096 \
  --n-gpu-layers -1 \
  --temp 0.15

For server deployment:

bash

llama-server \
  --model qwen-2.5-3B-instruct-gguf-Q4-K-M.gguf \
  --chat-template qwen2 \
  --ctx-size 4096 \
  --n-gpu-layers -1 \
  --port 8080 \
  --host 0.0.0.0

Query via the OpenAI-compatible API:

python

from openai import OpenAI
import re

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-required")

response = client.chat.completions.create(
    model="qwen-2.5-3B-instruct-gguf-Q4-K-M",
    messages=[{"role": "user", "content": "Your prompt here"}],
    temperature=0.15,
)
# Strip EOS tokens from output
clean = re.sub(r'<\|im_end\|>', '', response.choices[0].message.content).strip()
print(clean)

Evaluation Artifacts

The full per-case evaluation CSV (comparison_results_v7_21_Qwen2.5_3B_Instruct_20260221_041137.csv) and rollup.json are published in this repository for independent verification.

Artifact Provenance

Artifact	Format	Size	SHA256
`qwen-2.5-3B-instruct-gguf-Q4-K-M.gguf`	GGUF Q4_K_M	1.93 GB	`9ab3bc9beaddaec3700d5cc754b52e1501a3fd172bc7fc3ee3eb8e1d388ee043`
F16 (companion repo)	GGUF F16	6.18 GB	`65a0239fc9f9a40e2d4f79ae5e158cad423c2476fe089c744d5e6a6ff6fc9330`

Both artifacts were produced from Qwen/Qwen2.5-3B-Instruct using a custom-built llama.cpp conversion and quantization pipeline developed by PBH Applied Systems.

Evaluation Methodology

quant_eval v7.21 is a proprietary behavioral evaluation harness developed by PBH Applied Systems.

Fixture set: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c079371b02a94f3c149eb78a6adc03dc16ff6833b964fbf4174f0)

Family	Description	Pass Signals
`fuzz`	Property-based regression; structured placement correctness	schema_ok, constraints_ok
`json`	Single-step structured JSON with constraint rules	schema_ok, constraints_ok
`json_multistep`	Multi-step planning with self-check and oracle verification	schema_ok, checks_consistent_ok, stop_semantics_ok, oracle_equiv_ok
`mcq`	Multiple-choice extraction	choice_ok
`stateful_followup`	Two-turn state tracking; turn-2 correct given turn-1	turn1/2_parse_ok, turn1/2_exact_match
`mixed_brief_json`	Hybrid: natural language answer + valid JSON block	answer_line_ok, json_parse_ok, schema_ok
`toolcall`	Tool call embedded in response; parse + schema validation	stage1_tool_parse_ok, stage1_tool_schema_ok
`toolcall_only`	Bare schema-only tool call; strict tool name + args check	tool_name_ok, args_ok

Evaluation hardware: NVIDIA RTX 4090 (24 GB VRAM) Evaluation date: February 21, 2026 quant_eval seed: 42

🔬 About quant_eval & This Evaluation Series

quant_eval is a proprietary behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.

See it in action: Live AI Agent Demo → The demo runs production-style agent workflows powered by open-weight models selected through the quant_eval evaluation pipeline.

Need a deployment recommendation? Not sure which quantization level is right for your hardware, latency target, or agent type? → pbhappliedsystems.com

Evaluated and published by PBH Applied Systems, LLC · patrick@pbhappliedsystems.com

About PBH Applied Systems

PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development. The organization operates with a strong emphasis on engineering rigor, reproducibility, and real-world deployment constraints — particularly in environments where performance, cost efficiency, and reliability must be balanced against available hardware and budget.

Founder — Patrick Hill, M.S.

PBH Applied Systems was founded by Patrick Hill, a Data Scientist and AI/ML Engineer with 10+ years of experience delivering advanced analytics, predictive modeling, and decision-support solutions across high-stakes operational environments. Patrick holds a Master of Science in Software Engineering with concentrations in Artificial Intelligence and Machine Learning and a B.S. in Business Finance.

Technical expertise spans:

Languages & Data: Python, SQL, Linux, Pandas, NumPy, scikit-learn
ML & Modeling: Supervised and unsupervised learning, neural networks, NLP, transformers, regression, classification, forecasting, and feature engineering
AI/ML Frameworks: PyTorch, TensorFlow/Keras, HuggingFace Transformers, GGUF, llama.cpp, BitsAndBytes, PEFT, QLoRA
Deployment & MLOps: Flask APIs, Docker, CI/CD pipelines, REST endpoints, streaming inference, version control
Data Platforms: Jupyter, Databricks, Power BI, Matplotlib
Quantization: GGUF conversion, Q4_K_M / Q5_K_M / Q8_0 strategies, adapter-per-model evaluation architecture

Published Author

Patrick is the author of Applied Machine Learning: Concepts, Tools, and Case Studies — a 1,200+ page practitioner-oriented textbook adopted as required reading for CSC 373 – Machine Learning at the University of Advancing Technology.

Core Service Areas

1. LLM Optimization & Deployment · 2. AI Evaluation Frameworks · 3. Agentic AI Infrastructure · 4. Scalable AI Application Development · 5. ML Pipeline Design & Analytics · 6. Model & Agent Cataloging

📞 Work With PBH Applied Systems

The Qwen2.5-3B Q4_K_M evaluation tells a clear story about what to expect at the 3B scale: sub-second inference, reliable structured outputs, solid stateful retention, and a well-defined ceiling on multi-step reasoning consistency. Every finding is verifiable — download the CSV and check the checks_consistent_ok signal yourself.

👉 Book a Scoping Call — Discuss model selection, edge deployment strategy, or evaluation needs directly with Patrick.

👉 Request an Evaluation Report — Full quant_eval behavioral audit for your target model(s). Engagements from $2,500.

Connect


🌐 Website	pbhappliedsystems.com
📧 Email	patrick@pbhappliedsystems.com
💼 LinkedIn	PBH Applied Systems, LLC
▶️ YouTube	@pbhappliedsystems
📸 Instagram	@pbhappliedsystems
👍 Facebook	pbhappliedsystems

License

This GGUF repository is governed by the Qwen Research License Agreement, inherited from the base model Qwen/Qwen2.5-3B-Instruct.

Key terms:

Non-commercial use only — The license grants a non-exclusive, royalty-free license for research or evaluation purposes only
Commercial use requires a separate license — Contact Alibaba Cloud to request a commercial license
Redistribution — Permitted with attribution and license copy included
Attribution requirement — If used to train or improve another AI model, display "Built with Qwen" or "Improved using Qwen" in documentation
Governing law — People's Republic of China; exclusive jurisdiction in Hangzhou City People's Courts

Full license text: Qwen Research License Agreement

The quant_eval evaluation methodology, fixture set, and scoring framework are proprietary to PBH Applied Systems, LLC and are not included in this repository.

GGUF conversion, quantization, and behavioral evaluation performed by PBH Applied Systems, LLC · quant_eval v7.21 · Run ID: 20260221_041137

pbhappliedsystems/qwen-2.5-3B-instruct-gguf-Q4-K-M

Qwen2.5-3B-Instruct · GGUF Q4_K_M

Try This Model in the Live AI Agent Demo

Model Description

Key Characteristics

PBH Applied Systems Evaluation — quant_eval v7.21

Aggregate Scores (Q4_K_M)

Per-Family Pass Rates

F16 Baseline (full_weight_transformers)

Q4_K_M (quantized_llama_cpp)

Key Findings

Finding 1: The Speed Story — Fastest Model in the Evaluated Series

Finding 2: checks_consistent_ok = 0.200 — The 3B Ceiling

Finding 3: F16 Role-Token Contamination

Finding 4: F16 ms_easy_01 — Wrong Output Type

Finding 5: toolcall_only — Schema Vocabulary Errors

Finding 6: toolcall — Correct Arithmetic, EOS Contamination

Finding 7: MCQ — A-Bias Pattern

Signal-Level Diagnostics

Q4_K_M — json_multistep

Q4_K_M — stateful_followup

Q4_K_M — toolcall_only

Q4_K_M — mixed_brief_json

Recommended Use Cases

✅ Deploy with Confidence (Q4_K_M, Non-Commercial)

⚠️ Use with Guardrails (Q4_K_M)

❌ Not Recommended (Q4_K_M)

Hardware Requirements

Usage

Installation

Python — llama-cpp-python

CLI — llama-cli

Evaluation Artifacts

Artifact Provenance

Evaluation Methodology

🔬 About quant_eval & This Evaluation Series

About PBH Applied Systems

Founder — Patrick Hill, M.S.

Published Author

Core Service Areas

📞 Work With PBH Applied Systems

Connect

License

pbhappliedsystems/
qwen-2.5-3B-instruct-gguf-Q4-K-M

F16 Baseline (`full_weight_transformers`)

Q4_K_M (`quantized_llama_cpp`)