Mixtral-8x7B-Instruct-v0.1 - GGUF Quantized by worthdoing

Quantized for local Mac inference (Apple Silicon / Metal) by worthdoing

About

• Original model: mistralai/Mixtral-8x7B-Instruct-v0.1
• Parameters: 46.7B
• Quantized by: worthdoing
• Pipeline: corelm-model v1.0

Description

Available Quantizations

How to Use

With Ollama

With llama.cpp

With LM Studio

1. Download the GGUF file
2. Open LM Studio -> My Models -> Import
3. Select the GGUF file and start chatting

Quantization Method

Step 1 — Download & Validation

• Model weights are downloaded from HuggingFace Hub in SafeTensors format (.safetensors)
• Legacy formats (.bin, .pt) are excluded to ensure clean, verified weights
• Tokenizer, configuration, and all metadata are preserved

Step 2 — Conversion to GGUF F16 Baseline

• The original model is converted to GGUF format at FP16 precision using convert_hf_to_gguf.py from llama.cpp
• This lossless baseline preserves the full original model quality
• Architecture-specific tensors (attention, FFN, embeddings, MoE routing) are mapped to their GGUF equivalents

Step 3 — K-Quant Quantization

• The F16 baseline is quantized using llama-quantize with k-quant methods
• K-quants use a mixed-precision approach: more important layers (attention, output) retain higher precision, while less sensitive layers (FFN) are compressed more aggressively
• Each quantization level offers a different quality/size tradeoff:

Step 4 — Metadata Injection

• Custom metadata is embedded directly in each GGUF file:
  • general.quantized_by: worthdoing
  • general.quantization_version: corelm-1.0
• This ensures full traceability and provenance of every quantized file

Tools & Environment

• llama.cpp: Used for both conversion and quantization — the industry-standard open-source LLM inference engine
• Target platform: Apple Silicon Macs (M1/M2/M3/M4) with Metal GPU acceleration
• Inference runtimes: Compatible with llama.cpp, Ollama, LM Studio, koboldcpp, and any GGUF-compatible runtime

Recommended Hardware

Tags

---