G9v3-3B GGUF
GGUF quantized versions of
ai9stars/G9v3-3B, a compact 3B-parameter Llama-architecture language model supporting English and Chinese, with a 128K context window, tool-calling, and a built-in thinking/reasoning mode.
Model Overview
G9v3-3B is a lightweight text-generation model built on the Llama architecture. Despite its small size (~3B parameters), it supports a 131,072-token context window (128K) thanks to a high RoPE θ of 5,000,000. The model uses Grouped-Query Attention (GQA) with 2 key-value heads across 16 attention heads for efficient inference, and includes special tokens for thinking/reasoning (<|thought_begin|> / <|thought_end|>, /think, /no_think) and tool calling (<function>, <|tool_call|>, etc.).
The model uses a ChatML-style conversation format with <|im_start|> / <|im_end|> delimiters.
Model Architecture
| Property | Value |
|---|
| Architecture | Llama |
| Parameters | ~3B |
| Hidden Size | 2048 |
| Intermediate Size | 6144 |
| Layers | 52 |
| Attention Heads | 16 |
| KV Heads | 2 (GQA) |
| Head Dimension | 128 |
| Context Length | 131,072 |
| Vocabulary Size | 130,560 |
| RoPE Theta | 5,000,000 |
| Original Precision | bfloat16 |
| Supported Languages | en, zh |
Available GGUF Files
| File | Quantization | Size | Use Case |
|---|
g9v3-3b-bf16.gguf | BF16 | 5.6 GB | Max precision, reference model |
g9v3-3b-Q8_0.gguf | Q8_0 | 3.0 GB | Near-lossless, good speed/quality balance |
g9v3-3b-Q6_K.gguf | Q6_K | 2.3 GB | Very high quality, recommended for quality |
g9v3-3b-Q5_K_M.gguf | Q5_K_M | 2.0 GB | High quality, balanced |
g9v3-3b-Q5_K_S.gguf | Q5_K_S | 2.0 GB | High quality, slightly smaller than M |
g9v3-3b-Q4_K_M.gguf | Q4_K_M | 1.8 GB | Good quality, recommended default |
g9v3-3b-Q4_K_S.gguf | Q4_K_S | 1.7 GB | Smaller, acceptable quality |
g9v3-3b-Q4_0.gguf | Q4_0 | 1.7 GB | Legacy quant, fastest inference |
Recommended: Q4_K_M or Q5_K_M offer the best quality-to-size trade-off for most use cases.
Usage
llama.cpp CLI
1./llama-cli \
2 -m g9v3-3b-Q4_K_M.gguf \
3 -p "Explain quantum computing in simple terms." \
4 --temp 0.9 --top-p 0.95
llama-server (OpenAI-compatible API)
1./llama-server \
2 -m g9v3-3b-Q4_K_M.gguf \
3 --host 0.0.0.0 --port 8080
Thinking Mode
The model supports a thinking/reasoning mode controlled via special tokens. Use /think to enable extended reasoning or /no_think to disable it. When thinking is enabled, the model outputs its reasoning between <|thought_begin|> and <|thought_end|> tokens before providing the final answer.
Tool Calling
The model supports function/tool calling via XML-style <function> tags. Tool definitions are injected into the system prompt, and the model responds with <function name="..."> blocks containing <param> elements.
Generation Parameters
Recommended parameters from the original model's generation_config.json:
| Parameter | Value |
|---|
| Temperature | 0.9 |
| Top-P | 0.95 |
| BOS Token ID | 0 |
| EOS Token IDs | 1, 130073 |
| Pad Token ID | 1 |
Quantization
These GGUF files were created from the BF16 source model using
llama-quantize from the llama.cpp project.
Acknowledgements
- Original model: ai9stars/G9v3-3B
- Quantization tool: llama.cpp
License
