File Name	Size	Quant Method	Description
`Nanbeige4.2-3B-Q4_K_M.gguf`	~2.57 GB	`Q4_K_M`	4-bit medium. Recommended balance of speed, memory usage, and quality.
`Nanbeige4.2-3B-Q4_K_S.gguf`	~2.50 GB	`Q4_K_S`	4-bit small. Slightly lower memory footprint.
`Nanbeige4.2-3B-Q5_K_M.gguf`	~2.99 GB	`Q5_K_M`	5-bit medium. Higher precision with slight increase in size.
`Nanbeige4.2-3B-Q6_K.gguf`	~3.42 GB	`Q6_K`	6-bit quantization. Very close to FP16 performance.
`Nanbeige4.2-3B-Q8_0.gguf`	~4.43 GB	`Q8_0`	8-bit quantization. Maximum quality for GGUF.

Usage Guide

1. Running with `llama.cpp`

For full support, clone the official or nanbeige42 fork of llama.cpp:

bash

# Clone the repository with Nanbeige support
git clone -b nanbeige42 [https://github.com/Nanbeige/llama.cpp.git](https://github.com/Nanbeige/llama.cpp.git)
cd llama.cpp

# Build with CUDA support
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

# Download a model from this repository
huggingface-cli download Abiray/Nanbeige4.2-3B-GGUF Nanbeige4.2-3B-Q4_K_M.gguf --local-dir .

# Run CLI inference
./build/bin/llama-cli -m Nanbeige4.2-3B-Q4_K_M.gguf -ngl 99 -p "Which number is bigger, 9.11 or 9.8?"

Abiray/Nanbeige4.2-3B-GGUF

Nanbeige4.2-3B - GGUF Quants

Available Files & Quantization Details

Usage Guide

1. Running with llama.cpp

Abiray/
Nanbeige4.2-3B-GGUF

1. Running with `llama.cpp`