MediPhi-Instruct GGUF
GGUF versions of MediPhi-Instruct prepared for local inference.
This repository provides both an F16 version and a Q4_K_M
quantized version for use with llama.cpp and other GGUF-compatible
inference engines.
Available Files
| File | Format | Approx. Size |
|---|
MediPhi-Instruct-F16.gguf | F16 | 7.2 GB |
MediPhi-Instruct-Q4_K_M.gguf | Q4_K_M | 2.3 GB |
Recommended Version
MediPhi-Instruct-Q4_K_M.gguf is recommended for most local
inference setups because it requires substantially less memory
than the F16 version.
The F16 version is provided when higher precision is preferred
and sufficient memory is available.
Quantization
The Q4_K_M file was generated from the F16 GGUF model using
the quantization tools provided by llama.cpp.
Quantization format:
Q4_K_M
llama.cpp
Basic usage:
llama-cli -m MediPhi-Instruct-Q4_K_M.ggufGPU offloading:
llama-cli -m MediPhi-Instruct-Q4_K_M.gguf -ngl 99Local server:
llama-server -m MediPhi-Instruct-Q4_K_M.gguf -ngl 99Compatible Software
- llama.cpp
- llama-cpp-python
- LM Studio
- Jan
- Other GGUF-compatible runtimes
Original Model
This repository contains a community GGUF conversion and
quantization of MediPhi-Instruct.
Original model:
PLACE_ORIGINAL_MODEL_URL_HERE
Please refer to the original model repository for the model
license, intended use, limitations, and attribution requirements.
Disclaimer
This repository is not an official repository of the original
model authors unless explicitly stated.
SHA256
Checksums can be generated with:
