# Download the model
huggingface-cli download Fu01978/Llama-3.2-3B-MoE-4Expert-Q4_K_M-GGUF Llama-3.2-3B-MoE-4Expert.Q4_K_M.gguf --local-dir .

# Run with llama.cpp
./llama-cli -m Llama-3.2-3B-MoE-4Expert.Q4_K_M.gguf -p "Write a Python function to reverse a string" -n 512

Python (llama-cpp-python)

python

from llama_cpp import Llama

llm = Llama(
    model_path="Llama-3.2-3B-MoE-4Expert.Q4_K_M.gguf",
    n_ctx=2048,
    n_threads=8,
)

output = llm(
    "Explain quantum entanglement in simple terms",
    max_tokens=512,
    temperature=0.7,
)
print(output['choices'][0]['text'])

Performance Notes

The Q4_K_M quantization provides excellent quality with minimal degradation compared to the original model while using ~65% less disk space and memory. Recommended for most use cases. The BF16 version maintains full original quality and is recommended if you have sufficient VRAM/RAM.

Conversion Details

Original Model: Fu01978/Llama-3.2-3B-MoE-4Expert
Conversion Tool: llama.cpp
Quantization Method: Q4_K_M via llama.cpp quantization — BF16 via llama.cpp convert_hf_to_gguf.py script

Acknowledgments

Original MoE model by Fu01978
GGUF conversion using llama.cpp
Base models: Meta AI (Llama 3.2), unsloth, prithivMLmods, DavidAU