Quantized Falcon3-7B-Base Models

Model Overview

• Original Model: Falcon3-7B-Base
• Quantized Versions:
  • Q4_K_M (4-bit quantization)
  • Q5_K_M (5-bit quantization)
• Architecture: Decoder-only transformer
• Base Model: Falcon3-7B-Base
• Modalities: Text only
• Developer: Technology Innovation Institute
• License: falcon-llm-license
• Language: English, French, Spanish, Portuguese

Quantization Details

Q4_K_M Version

• Approx. ~71% size reduction
• Lower memory footprint (~4.26 GB)
• Best suited for deployment on edge devices or low-resource GPUs
• Slight performance degradation in complex reasoning scenarios

Q5_K_M Version

• Approx. ~67% size reduction
• Higher fidelity (~4.96 GB)
• Better performance retention, recommended when quality is a priority

Key Features

• Pretrained on 14 Teratokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 1024 H100 GPU chips.
• Text-only instruction-following model optimized for multi-turn scientific question answering.
• Includes multilingual data, but with a primary focus on English (plus French, Spanish, Portuguese).
• 32K token context length with Grouped Query Attention (12 query heads, 4 KV heads) for scalable inference.

Usage

Recommended Use Cases

• Code generation & programming
  Assisting with code completion, debugging, or generating small snippets, especially for use in developer tools or coding assistants.
• Scientific & technical research
  Useful for answering complex scientific questions, solving mathematical problems, and working with STEM content.
• Long-context workflows
  Good for documents, research papers, logs, transcripts etc., where you need to process or reference up to ~32K tokens in a single input.
• Low-resource deployment
  Low-resource deployment runs AI models efficiently on limited hardware like CPUs, edge devices, or small GPUs.

Acknowledgments

• The Technology Innovation Institute team for developing and releasing the Falcon3-7B-Base model.
• Georgi Gerganov and the entire llama.cpp open-source community for enabling efficient model quantization and inference via the GGUF format.

Contact