North-Mini-Code-1.0-GGUF
GGUF conversions and quantizations of CohereLabs/North-Mini-Code-1.0 for use with:
- llama.cpp
- LM Studio
- Ollama
- Jan
- KoboldCpp
- Text Generation WebUI
- Open WebUI
- Other GGUF-compatible runtimes
About the Model
North-Mini-Code-1.0 is a code-focused Mixture-of-Experts (MoE) model released by CohereLabs.
This repository provides ready-to-use GGUF conversions for local inference across a range of hardware configurations.
Available Files
Full Precision
North-Mini-Code-1.0-F16.gguf
Quantized Versions
North-Mini-Code-1.0-Q4_K_M.ggufNorth-Mini-Code-1.0-Q5_K_M.ggufNorth-Mini-Code-1.0-Q6_K.ggufNorth-Mini-Code-1.0-Q8_0.gguf
Recommended Quantization
For most users:
North-Mini-Code-1.0-Q4_K_M.ggufIt offers the best balance of:
- Quality
- Memory usage
- Inference speed
If you have more available RAM/VRAM, consider:
North-Mini-Code-1.0-Q5_K_M.ggufor
North-Mini-Code-1.0-Q6_K.gguffor slightly higher output quality.
Approximate File Sizes
1F16 ~60+ GB
2Q4_K_M ~20 GB
3Q5_K_M ~23 GB
4Q6_K ~27 GB
5Q8_0 ~34 GB
Actual sizes may vary slightly depending on conversion tooling versions.
Usage
llama.cpp
Prompt mode:
1./llama-cli \
2 -m North-Mini-Code-1.0-Q4_K_M.gguf \
3 -p "Write a Python function that reverses a linked list."
Chat mode:
1./llama-cli \
2 -m North-Mini-Code-1.0-Q4_K_M.gguf \
3 -cnv
LM Studio
- Download your preferred GGUF file.
- Open LM Studio.
- Import the model.
- Start chatting.
Ollama
Create a Modelfile:
FROM North-Mini-Code-1.0-Q4_K_M.ggufCreate the model:
ollama create north-mini-code -f ModelfileRun it:
ollama run north-mini-code
Hardware Recommendations
Q4_K_M
Recommended minimum:
Q5_K_M
Recommended minimum:
Q6_K
Recommended minimum:
Q8_0
Recommended minimum:
F16
Recommended minimum:
Prompting Tips
This model is optimized for programming-related tasks.
Example prompts:
Implement a fast Rust HTTP server.Explain this C++ compiler error.Write comprehensive unit tests for the following Python code.Convert this JavaScript function to TypeScript.
Base Model
Base model:
CohereLabs/North-Mini-Code-1.0All training, architecture, benchmarks, licensing terms, and usage restrictions belong to the original model authors.
Please refer to the original repository for official documentation and licensing information.
Conversion Details
Converted using:
Generated quantizations:
1F16
2Q4_K_M
3Q5_K_M
4Q6_K
5Q8_0
A tokenizer compatibility workaround was applied during conversion to support current GGUF conversion tooling.
Credits
- Base Model: CohereLabs
- GGUF Conversion & Quantization: NANI-Nithin
- Tooling: llama.cpp
Repository
