This is a GGUF quantized version of ruslanmv/granite-3.1-2b-Reasoning, fine-tuned from ibm-granite/granite-3.1-2b-instruct. The GGUF format allows for efficient inference on CPU and GPU, optimized for use with Kbit quantization levels (4-bit, 5-bit, and 8-bit).

Developed by: ruslanmv
License: Apache 2.0
Base Model: ibm-granite/granite-3.1-2b-instruct
Fine-tuned for: Logical reasoning, structured problem-solving, long-context tasks
Quantized GGUF versions available:
- 4-bit: Q4_K_M
- 5-bit: Q5_K_M
- 8-bit: Q8_0
Supported Languages: English
Architecture: Granite
Model Size: 2.53B params

Why Use the GGUF Quantized Version?

The GGUF format is designed for optimized CPU and GPU inference, enabling:

✅ Lower memory usage for running on consumer hardware
✅ Faster inference speeds without compromising reasoning ability
✅ Compatibility with popular inference engines like llama.cpp, ctransformers, and KoboldCpp

Installation & Usage

To use this model with llama.cpp, install the required dependencies:

pip install llama-cpp-python

Running the Model

To run the model using llama.cpp:

bash

from llama_cpp import Llama

model_path = "path/to/ruslanmv/granite-3.1-2b-Reasoning-GGUF.Q4_K_M.gguf"

llm = Llama(model_path=model_path)

input_text = "Can you explain the difference between inductive and deductive reasoning?"
output = llm(input_text, max_tokens=400)

print(output["choices"][0]["text"])

Alternatively, using ctransformers:

pip install ctransformers

python

from ctransformers import AutoModelForCausalLM

model_path = "path/to/ruslanmv/granite-3.1-2b-Reasoning-GGUF.Q4_K_M.gguf"

model = AutoModelForCausalLM.from_pretrained(model_path, model_type="llama", gpu_layers=50)

input_text = "What are the key principles of logical reasoning?"
output = model(input_text, max_new_tokens=400)

print(output)

Intended Use

Granite-3.1-2B-Reasoning-GGUF is optimized for efficient inference while maintaining strong reasoning capabilities, making it ideal for:

Logical and analytical problem-solving
Text-based reasoning tasks
Mathematical and symbolic reasoning
Advanced instruction-following

This model is particularly useful for CPU-based deployments and users who need low-memory, high-performance text generation.

License & Acknowledgments

This model is released under the Apache 2.0 license. It is fine-tuned from IBM’s Granite 3.1-2B-Instruct model and quantized using GGUF for optimal efficiency. Special thanks to the IBM Granite Team for developing the base model.

For more details, visit the IBM Granite Documentation.

Citation

If you use this model in your research or applications, please cite:

@misc{ruslanmv2025granite,
  title={Fine-Tuning and GGUF Quantization of Granite-3.1 for Advanced Reasoning},
  author={Ruslan M.V.},
  year={2025},
  url={https://huggingface.co/ruslanmv/granite-3.1-2b-Reasoning-GGUF}
}