Granite-4.1-3B is a compact instruction-tuned language model developed by IBM, designed for efficient conversational AI, reasoning, coding, retrieval-augmented generation (RAG), and enterprise NLP workloads. This repository contains GGUF quantized variants of the model optimized for efficient local inference using llama.cpp.

The model emphasizes strong instruction following, structured response generation, multilingual language understanding, and practical deployment efficiency while maintaining a significantly smaller footprint than larger Granite family models. Its compact architecture enables high-quality local inference across a wide variety of production AI workflows.

The quantized formats significantly reduce memory requirements while preserving language understanding and reasoning capability, making the model suitable for consumer hardware, edge deployments, and enterprise AI systems.

Model Overview

Model Name: Granite-4.1-3B
Base Model: ibm-granite/granite-4.1-3b
Architecture: Decoder-Only Transformer
Parameter Count: Approximately 3 Billion Parameters
Modalities: Text
Languages: Multilingual
Developer: IBM
License: Apache 2.0

Quantization Formats

This repository provides various GGUF quantized versions of the Granite-4.1-3B model optimized for efficient local inference using llama.cpp.

IQ3_M

Size reduction of approx 76.18% (1.51 GB) compared to 16-bit (6.34 GB)
Aggressive 3-bit quantization optimized for lightweight enterprise language model deployments
Suitable for conversational AI, lightweight coding assistance, and resource-efficient inference environments
Enables practical execution on consumer hardware while minimizing memory usage
Complex reasoning, long-context generation, and advanced analytical tasks may exhibit reduced fidelity compared to higher-precision variants

IQ4_NL

Size reduction of approx 70.50% (1.87 GB) compared to 16-bit (6.34 GB)
Advanced 4-bit non-linear quantization designed to better preserve reasoning capability and instruction-following quality
Better suited for production assistants, RAG pipelines, structured generation, and coding workflows
Designed to reduce quantization loss while maintaining response consistency across diverse NLP tasks
May require slightly increased computational overhead during inference

IQ4_XS

Size reduction of approx 71.92% (1.78 GB) compared to 16-bit (6.34 GB)
Balanced 4-bit quantization focused on efficient inference and dependable language generation
Provides a practical balance between memory efficiency, response quality, and runtime performance
Suitable for enterprise assistants, workflow automation, summarization, and production AI services
Maintains reliable performance across most real-world language understanding and text-generation workloads

Training Background (Original Model)

Granite-4.1-3B is trained with an emphasis on efficient language understanding, instruction following, reasoning, coding, and enterprise-oriented NLP tasks across diverse multilingual datasets.

Pretraining

Large-scale language pretraining across multilingual and domain-diverse text corpora
Focus on contextual language understanding, reasoning, and knowledge representation
Optimized for downstream conversational AI, coding, retrieval, and text-generation workloads

Instruction Tuning

Refined using high-quality instruction-following and dialogue datasets
Enhanced for conversational consistency, structured response generation, and tool-oriented interactions
Improved performance across coding, summarization, question answering, and enterprise assistant workflows

Key Capabilities

Instruction Following Produces reliable responses aligned with user instructions across a broad range of tasks.
Reasoning Supports logical reasoning and multi-step analytical problem solving.
Code Generation Assists with code completion, explanation, debugging, and programming tasks.
Retrieval-Augmented Generation (RAG) Integrates effectively into retrieval-based AI systems and knowledge assistants.
Structured Output Generation Generates organized responses suitable for automation, APIs, and downstream applications.
Efficient Local Deployment Quantized variants enable practical deployment on consumer hardware and edge systems.

Usage Example

Using llama.cpp

bash

./llama-cli \
  -m SandLogicTechnologies/Granite-4.1-3B_IQ4_NL.gguf \
  -p "Summarize this document and extract the key action items."

Recommended Usecases

Conversational AI Deploy efficient local AI assistants.
Coding Assistance Generate, explain, and debug source code.
Enterprise AI Build knowledge assistants and internal productivity tools.
Workflow Automation Generate structured responses for business processes and automation pipelines.
RAG Applications Power retrieval-augmented enterprise search and question-answering systems.
Research & Experimentation Evaluate compact language models for efficient local inference.

Acknowledgments

These quantized models are based on the original work by the IBM Granite development team.

Special thanks to:

The IBM team for developing and releasing the Granite-4.1-3B model.
Georgi Gerganov and the llama.cpp open-source community for enabling efficient quantization and inference via the GGUF format.

Contact

For questions, feedback, or support, please reach out at support@sandlogic.com or visit https://www.sandlogic.com/

SandLogicTechnologies/granite-4.1-3b-GGUF

Granite-4.1-3B