Granite 4 GGUF (4-bit Quantized)
This repository hosts GGUF-format quantized versions of IBM Granite 4 models at multiple parameter sizes.
The models provided here are intended for local inference and are suitable for use with SciTools’ Understand and Onboard, as well as other tools and runtimes that support the GGUF format (for example, llama.cpp-based applications).
Model Details
- Base models: IBM Granite 4
- Variants provided: Micro 3B
- Format: GGUF
- Quantization: 4-bit (model-specific, see table below)
- Intended use: Local inference, code understanding, general-purpose chat
- Languages: English (as supported by Granite 4)
Quantization Process
- The 3B Granite 4 Micro model is quantized using Unsloth tooling.
- No additional fine-tuning, rebalancing, or prompt modification was applied.
- Quantization parameters were not altered from their original sources.
These models are redistributed as-is to provide reproducible, efficient GGUF variants suitable for local workflows.
What We Did Not Do
To be explicit:
- No additional fine-tuning
- No instruction rebalancing
- No safety, alignment, or prompt modifications
- No merging or model surgery
Any observed behavior is attributable to Granite 4 and the applied quantization, not downstream changes.
Intended Use
These models are suitable for:
- SciTools Understand and SciTools Onboard
- Local AI workflows
- Code comprehension and exploration
- Interactive chat and analysis
- Integration into developer tools that support GGUF
They are not intended for:
- Safety-critical or regulated decision-making
- Use cases requiring guaranteed factual accuracy
- Production deployment without independent evaluation
Limitations
- As 4-bit quantized models, some reduction in reasoning depth and precision is expected compared to full-precision checkpoints.
- Output quality varies between the 1B and 3B variants.
- Like all large language models, Granite 4 may produce incorrect or misleading outputs.
Evaluate carefully for your specific workload.
License & Attribution
- Original models: IBM (Granite 4)
- Quantization: IBM and Unsloth
- Format: GGUF (llama.cpp ecosystem)
Please refer to the original Granite 4 license and usage terms. This repository redistributes quantized artifacts only and does not modify the underlying licensing conditions.
Acknowledgements
Thanks to IBM for releasing the Granite 4 models and to Unsloth for providing efficient, reproducible quantization that enables practical local inference.
