Model is Released by Microsoft Research under MIT License :

Further Instructions are provided to convert this modelfile into its GGUF format. For Demo purpose, Ollama inference is also demonstrated at the end.

Quantization Process

Getting Ready Step-1 - Install Essential Tools and Dependencies: The llama.cpp project requires several tools for compilation and networking.

Getting Ready Step-2 - Clone and Configure llama.cpp: Get the latest source code for the project.

Getting Ready Step-3 - Build the llama.cpp Executables

Compile all targets (including llama-quantize, llama-cli, and llama-server)

-j$(nproc) uses all available CPU cores for faster compilation

Build Step-1 - Download and Convert the Model (HF to GGUF FP16)

This step downloads the original Hugging Face model and converts it into the initial, unquantized GGUF FP16 format. This process requires ~18 GB of System RAM and produces a file around 15.2 GB.

Create a Directory for the Model Weights:

Build Step-2 - Download the Model Weights: Use git-lfs to clone the specific model weights from Hugging Face into the new directory. (Note: Replace the URL below with the actual Hugging Face repo for Fara-7B if it differs)

Run the Conversion Script: This script converts the Hugging Face Safetensors into the FP16 GGUF file

Result: A file named Fara-7B-F16.gguf (approx. 15.2 GB) is created in the llama.cpp root directory.

Quantize Model ( FP16 to Q8_0 ) - Single Step Process

Run the Quantization Tool: We use the built executable ./build/bin/llama-quantize to compress the model to the high-quality Q8_0 format.

Result: A file named Fara-7B-Q8_0.gguf (approx. 8.1 GB) is created.

Running Inference with Ollama

update Ollama on your system before running this model.

If Ollama is running inside docker :

Running Inference with llama.cpp

The llama.cpp tools (llama-cli and llama-server) have two primary ways to load a GGUF file: Local File (easiest if already downloaded) and Hugging Face Download (easiest for first-time users).

1. Download the GGUF File

The optimized Fara-7B-Q8_0-GGUF file is approximately 8.1 GB. Download it directly from the Files tab of this repository, or use huggingface-cli:

2. Run Inference (Local File)

Once the file is in your llama.cpp/ directory and you have compiled the binaries (Step 2 of the Conversion Guide), you can run the model directly. Use the -ngl (n-gpu-layers) flag to offload most of the model onto your GPU's VRAM

Command Line Interface (CLI): Multimodal Use

To use the model's Vision-Language (V-L) capabilities, include the --image flag pointing to a local image file.

3. Web Server:

The Web Server is the easiest way to interact with multimodal models, allowing for image drag-and-drop.

Starts a web server for interactive chat (access at http://127.0.0.1:8080)

4. Alternatively - Direct Hugging Face Download & Run

If you want llama.cpp to handle the download and caching automatically without manually using huggingface-cli, use the --hf-repo flags

This command automatically finds, downloads, and runs the model.