Converted by PBH Applied Systems, LLC
— Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure
📌 Provenance repository — no behavioral evaluation performed. This repository contains the full-precision F16 GGUF of Qwen2.5-32B-Instruct. At 65.5 GB, the F16 artifact exceeds the VRAM capacity of the evaluation hardware (NVIDIA RTX 4090, 24 GB). All behavioral evaluation data for this model is in the Q4_K_M companion repository:pbhappliedsystems/qwen-2.5-32B-instruct-gguf-Q4-K-M.
🔬 About the evaluation series. Every other model published under pbhappliedsystems has been independently evaluated using quant_eval v7.21 — a proprietary behavioral evaluation harness developed by PBH Applied Systems — at both F16 and Q4_K_M precision. For Qwen2.5-32B, the F16 GGUF was produced and its artifact provenance is recorded here, but the evaluation constraint is documented honestly rather than omitted.
Why No Evaluation
In the PBH Applied Systems evaluation pipeline, F16 GGUFs serve as cache-generation baselines for Q4_K_M comparison runs. For every other model in the series, the F16 run produces a full_weight_cache.json that the Q4_K_M run reuses, enabling a direct two-run comparison against identical fixtures.
For Qwen2.5-32B, the F16 GGUF is 65.5 GB. Loading this into the evaluation hardware (NVIDIA RTX 4090, 24 GB VRAM) is not possible — not even with partial CPU offload at the precision required for a valid cache-generation baseline. The Q4_K_M comparison run (20260221_144732) was therefore run as a standalone evaluation against freshly generated responses rather than against a cached F16 baseline.
The consequence for this repository: There is no full_weight_cache.json, no F16 evaluation CSV, and no cross-precision comparison data. This card exists to document the F16 artifact for provenance and to make the 65.5 GB GGUF accessible to users with appropriate hardware.
For full behavioral analysis, cross-series comparisons, and deployment recommendations, see the Q4_K_M card.
This model is part of the PBH Applied Systems evaluated model series that supports the live AI Agent Demo. The demo lets visitors interact with production-style agent workflows powered by open-weight language models evaluated through PBH Applied Systems' quant_eval framework.
The F16 model serves a different role than the Q4_K_M deployment variant. F16 is the full-precision baseline used to measure what the model can do before quantization. quant_eval then compares the quantized model against this baseline to identify which capabilities are preserved, which degrade, and which tasks require guardrails or a higher-precision deployment.
This comparison is central to the demo. It helps determine which model belongs in which agent role:
Reasoning models are selected for planning, analysis, and auditable decision workflows.
Document models are selected for long-context extraction, summarization, and structured Q&A.
Code models are selected for task completion, structured output, API scaffolding, and automation workflows.
Quantized variants are selected when they preserve enough behavior to reduce cost, latency, and GPU requirements.
F16 variants remain important when maximum fidelity, cleaner tool execution, or reduced quantization risk matters more than speed or cost.
The live demo shows the deployment side of that process. The F16 card documents the reference behavior. The Q4_K_M card shows what changes after compression. Together, they explain how PBH Applied Systems uses quant_eval to choose the correct LLM for the correct agent type instead of guessing from model size or leaderboard reputation.
Model Description
This repository contains the full-precision F16 GGUF of Qwen/Qwen2.5-32B-Instruct, a 32.5-billion parameter instruction-tuned model from Alibaba Cloud (September 2024 release).
The F16 GGUF was converted from Qwen/Qwen2.5-32B-Instruct using a custom-built llama.cpp conversion pipeline developed by PBH Applied Systems, without modification to model weights.
quant_eval context: F16 run 20260221_144732 was requested with quant_types: ['Q4_K_M']. Due to VRAM constraints, no F16 full_weight_cache.json was written. The Q4_K_M evaluation ran as a standalone quantized_llama_cpp runner evaluation.
Hardware Requirements
Configuration
VRAM Required
Notes
F16 (this repo) · full GPU
~80 GB
2× A100 80 GB minimum
F16 · 4-GPU split
~20 GB per GPU
4× A100 40 GB or 4× A10G 24 GB
F16 · partial CPU offload
~40–50 GB VRAM + 64 GB RAM
Reduced context; slower inference
Q4_K_M (companion repo)
~24 GB
Single A10G or RTX 4090
For most production use cases, the Q4_K_M variant is the correct choice. It runs on single-GPU hardware, is fully evaluated, and delivers the same structured behavioral outputs that F16 would produce on this class of tasks. F16 is appropriate for compliance environments requiring full-weight artifacts, research settings requiring exact weight fidelity, or future evaluation runs on multi-GPU infrastructure.
Long-Context Deployment
The default config.json targets 32,768 tokens. To enable the full 131,072-token context window, apply YaRN scaling by adding the following to config.json before conversion, or configure it at the llama.cpp level:
Note that YaRN in llama.cpp uses static scaling — the factor is constant regardless of actual input length. Apply only when processing long contexts is required; shorter-context inference may be marginally affected.
1from huggingface_hub import hf_hub_download
2from llama_cpp import Llama
34# Note: 65.5 GB download — requires ~80 GB total VRAM for full GPU offload5model_path = hf_hub_download(6 repo_id="pbhappliedsystems/qwen-2.5-32B-instruct-gguf-F16",7 filename="qwen-2.5-32B-instruct-gguf-F16.gguf"8)910# Multi-GPU: tensor_split distributes model layers across available GPUs11# Example: 2× A100 80 GB12llm = Llama(13 model_path=model_path,14 n_ctx=8192,15 n_gpu_layers=-1,# Offload all layers16 tensor_split=[1,1],# Equal split across 2 GPUs17 verbose=True,# Monitor GPU memory allocation18)1920response = llm.create_chat_completion(21 messages=[22{23"role":"system",24"content":"You are a precise assistant. Follow instructions exactly."25},26{27"role":"user",28"content":"Analyze the following and return a JSON object with keys: summary, risk_level, action_items."29}30],31 temperature=0.7,32 max_tokens=1024,33)34print(response["choices"][0]["message"]["content"])
For partial CPU offload when full VRAM is unavailable (slower, but functional):
python
1# Example: 2× A10G 24 GB (48 GB total VRAM) + 64 GB system RAM2llm = Llama(3 model_path=model_path,4 n_ctx=4096,5 n_gpu_layers=48,# Offload first N layers; tune based on available VRAM6 tensor_split=[1,1],7 verbose=True,8)
CLI — llama-cli (multi-GPU)
bash
1llama-cli \2 --model qwen-2.5-32B-instruct-gguf-F16.gguf \3 --chat-template qwen2 \4 --system-prompt "You are a precise assistant. Follow instructions exactly."\5 --prompt "Return a JSON object with keys: summary, risk_level, action_items."\6 --n-predict 1024\7 --ctx-size 8192\8 --n-gpu-layers -1 \9 --tensor-split 1,1\10 --temp 0.7
quant_eval is a proprietary behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.
See it in action:Live AI Agent Demo →
The demo runs production-style agent workflows powered by open-weight models selected through the quant_eval evaluation pipeline.
Need a deployment recommendation?
Not sure which quantization level is right for your hardware, latency target, or agent type?
→ pbhappliedsystems.com
PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development.
Core Service Areas: LLM Optimization & Deployment · AI Evaluation Frameworks · Agentic AI Infrastructure · Scalable AI Application Development · ML Pipeline Design & Analytics · Model & Agent Cataloging
