Phi-4 Mini Instruct Q8_0 GGUF
Overview
This repository contains a Post-Training Quantized (PTQ) GGUF version of Microsoft's Phi-4 Mini Instruct model.
The original model was converted from Hugging Face Safetensors format to GGUF (F16) and subsequently quantized to Q8_0 using llama.cpp to enable efficient CPU-based inference while significantly reducing storage requirements.
Base Model Information
Item Value Base Model microsoft/Phi-4-mini-instruct Original Author Microsoft Original License MIT Original Format Safetensors Quantized Format GGUF Quantization Method Post-Training Quantization (PTQ) Quantization Type Q8_0
Original model:
Quantization Pipeline
The following workflow was used to create this model:
1 Phi-4 Mini Instruct (Safetensors)
2 ↓
3 GGUF Conversion (F16)
4 ↓
5 Post-Training Quantization (Q8_0)
6 ↓
7 Optimized GGUF Model
Conversion Process
Downloaded Phi-4 Mini Instruct from Hugging Face.
Converted the original Safetensors weights to GGUF (F16) using llama.cpp.
Generated an intermediate F16 GGUF model.
Applied Q8_0 Post-Training Quantization.
Verified model functionality using llama.cpp inference.
Validated compatibility with local deployment frameworks.
Quantization Results
Metric Value Original GGUF (F16) Size 7.15 GB Quantized GGUF (Q8_0) Size 4.08 GB Storage Reduction ~43% GPU Required No CPU Inference Supported Yes Quantization Backend llama.cpp
Hardware Used
Intel Core i7-1165G7
Windows 11
CPU-only quantization workflow
No NVIDIA GPU required
Repository Contents
File Description phi4-q8_0.gguf Quantized GGUF model README.md Documentation and usage instructions LICENSE Original MIT License from Microsoft
Using with llama.cpp
Run directly with llama.cpp:
llama-cli -m phi4-q8_0.ggufExample:
llama-cli -m phi4-q8_0.gguf -p "Explain post-training quantization."Using with Ollama
Create a file named:
Contents:
Create the model:
ollama create phi4-mini-q8 -f ModelfileRun:
Using with Python (llama-cpp-python)
Install:
pip install llama-cpp-pythonExample:
1 from llama_cpp import Llama
2
3 llm = Llama (
4 model_path = "phi4-q8_0.gguf" ,
5 n_ctx = 4096
6 )
7
8 response = llm (
9 "Explain quantization." ,
10 max_tokens = 200
11 )
12
13 print ( response [ "choices" ] [ 0 ] [ "text" ] )
Intended Use
This model is suitable for:
Local LLM deployment
CPU-only inference
Educational and research purposes
Edge AI applications
Resource-constrained environments
GGUF-compatible inference engines
License
This repository contains a quantized conversion of Microsoft's Phi-4 Mini Instruct model.
The original model is distributed under the MIT License by Microsoft. The included LICENSE file is retained from the original model repository.
All rights, ownership, model architecture, training methodology, and intellectual property remain with Microsoft.
This repository only provides a GGUF conversion and Q8_0 post-training quantized version of the original model.
Acknowledgements
Microsoft for the Phi-4 Mini Instruct model.
llama.cpp for GGUF conversion and quantization tooling.
Hugging Face for model hosting and distribution.
Quantization Author
K VIGNESH
Performed:
GGUF conversion
Q8_0 Post-Training Quantization
Validation and testing
Local deployment verification
CPU inference benchmarking
using llama.cpp and open-source tooling.
