# Download model (requires HF authentication for private repo)
huggingface-cli login
huggingface-cli download rihuwa/Reina_Magitral_gguf magistral-small-2509-text-only-Q5_K_M.gguf --local-dir ./models

# Run inference
./llama-cli -m ./models/magistral-small-2509-text-only-Q5_K_M.gguf -p "Your prompt here" -n 256 \
  --temp 0.7 --top-p 0.95

With Ollama

bash

# Download the GGUF file first
huggingface-cli login
huggingface-cli download rihuwa/Reina_Magitral_gguf magistral-small-2509-text-only-Q5_K_M.gguf --local-dir ./models

# Create Modelfile (the template uses Mistral v7 format)
cat > Modelfile << 'EOF'
FROM ./models/magistral-small-2509-text-only-Q5_K_M.gguf

PARAMETER temperature 0.7
PARAMETER top_p 0.95
PARAMETER num_ctx 8192
EOF

# Create and run model
ollama create reina-magistral -f Modelfile
ollama run reina-magistral

Important Notes for Magistral Models

Special Tokens: Magistral uses [THINK] and [/THINK] tags for reasoning
Context Window: 128k tokens (optimal performance under 40k)
Chat Format: Mistral v7 Tekken template
Recommended Settings:
- Temperature: 0.7
- Top-p: 0.95
- Top-k: 40

Model Architecture

Type: Mistral-based (24B parameters)
Tokenizer: Tekken tokenizer with special reasoning tokens
Vocabulary: Extended for reasoning capabilities
Attention: Sliding window + global attention

Quantization Details

Both quantizations preserve the quality of the LoRA fine-tuning:

Q8_0: 8-bit quantization, virtually no quality loss
Q5_K_M: 5-bit with K-quant method, excellent quality/size balance

Performance

Tested on NVIDIA GPUs:

Q8_0: Requires ~28GB VRAM
Q5_K_M: Requires ~18GB VRAM (fits on single 24GB GPU)

Related Repositories

LoRA Adapter: rihuwa/Reina_Magistral_LoRA
Base Model: Darkhn/Magistral-Small-2509-Text-Only

License

This model inherits the Apache 2.0 license from the base Magistral model.

Last Updated: 2025-11-23

rihuwa/Reina-Magistral-GGUF

Available Quantizations

Training Details

Usage

With llama.cpp

With Ollama

Important Notes for Magistral Models

Model Architecture

Quantization Details

Performance

Related Repositories

License

rihuwa/
Reina-Magistral-GGUF