This repository contains unofficial BF16, Q8_0, and Q4_K_M GGUF conversions of ai9stars/G9v3-39A5B, a preview Mixture-of-Experts language model from AI9Stars.

The conversion was created and tested with a llama.cpp implementation of the G9v3 architecture available on the feature/g9v3-support branch. Until G9v3 support is merged upstream, standard llama.cpp builds will not load this model.

Available files

File	Format	Size	SHA-256
`G9v3-39A5B-BF16.gguf`	BF16	72.63 GiB (77,988,959,200 bytes)	`1062166296d1de8bc3cdce1200d7c90747fc705e45084bc854d0663e1db8c58e`
`G9v3-39A5B-Q8_0.gguf`	Q8_0	38.63 GiB (41,479,836,640 bytes)	`4df00c03c39b28720ce401ad5df0587987d01ae3048199d447b6b664958992fe`
`G9v3-39A5B-Q4_K_M.gguf`	Q4_K_M	21.94 GiB (23,559,222,240 bytes)	`f2ce5c7fe6663a066175de35af5c8da2e0597504630abadc3c0cd124a515d626`
`G9v3-39A5B-IQ4_XS.gguf`	IQ4_XS (imatrix)	19.44 GiB (20,871,738,560 bytes)	`bf3342d70ff602bf34aef2f77fe9626cd8d575e00758a1c3588542fbf147fee2`

The IQ4_XS file was quantized with an importance matrix. The matrix is published as G9v3-39A5B.imatrix.gguf (SHA-256 3072a061c77d31657fe4c01a528cf0a67f2abc700f329d4f569c18f6b810861d) so the quantization can be reproduced or used as a base for further IQ quantizations.

Choose the file according to your hardware and quality requirements:

Format	Description	Recommended system RAM
BF16	Highest-fidelity GGUF. Source BF16 tensors are preserved, along with required F32 tensors.	96 GiB or more
Q8_0	8.51 BPW quantization with very low quantization error; a near-lossless practical choice when memory allows.	48 GiB or more
Q4_K_M	Mixed K-quant that keeps selected tensors at higher precision; practical for local CPU inference.	32 GiB or more
IQ4_XS	Importance-matrix quantization at 4.28 BPW; the smallest published option, for tight memory budgets.	32 GiB or more

The memory recommendations include headroom for the runtime, context, and operating system. Actual requirements vary with context size and llama.cpp settings.

Model details

Property	Value
Architecture	`G9v3ForCausalLM`, sparse MoE
Parameters	Approximately 39B total / 5B active per token
Layers	38 (1 dense layer followed by 37 MoE layers)
Hidden size	2,048
Attention	32 query heads, 2 key/value heads, head dimension 128
Experts	320 routed experts, 32 selected per token, plus 1 shared expert
Expert intermediate size	512
Vocabulary	130,560 tokens
Maximum context	131,072 tokens
Languages	English and Chinese
Features	Think/No-Think modes, tool calling, long context

The base model is a preview release. Refer to the ai9stars/G9v3-39A5B model card for the original description and usage guidance.

llama.cpp setup

Build the G9v3-enabled branch:

bash

git clone --branch feature/g9v3-support --single-branch \
  https://github.com/linuxid10t/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp/build --target llama-cli llama-server -j

Command line

Both GGUF files contain the model's chat template. Start an interactive conversation with the Q4_K_M file using:

bash

./llama.cpp/build/bin/llama-cli \
  -m G9v3-39A5B-Q4_K_M.gguf \
  -c 4096 -t 32 -cnv

For BF16, use the same command with the other filename:

bash

./llama.cpp/build/bin/llama-cli \
  -m G9v3-39A5B-BF16.gguf \
  -c 4096 -t 32 -cnv

Start with a modest context such as 4,096 tokens, then increase it according to your available memory and workload.

OpenAI-compatible server

bash

./llama.cpp/build/bin/llama-server \
  -m G9v3-39A5B-Q4_K_M.gguf \
  -c 4096 -t 32 -tb 32 \
  --host 127.0.0.1 --port 8080

Replace the model filename with G9v3-39A5B-BF16.gguf to serve the BF16 version on a machine with sufficient memory.

Example request:

bash

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "G9v3-39A5B-Q4_K_M.gguf",
    "messages": [{"role": "user", "content": "Who are you?"}],
    "temperature": 0.7,
    "top_p": 0.95,
    "max_tokens": 256
  }'

Bind the server to a non-loopback address only after configuring appropriate authentication and network controls.

Conversion provenance

Converted from the 2026-08-19 revision of the source BF16 checkpoint (Hugging Face commit 70213b14ea559298507e46e36e53d9c4c76144fc, 16 safetensors shards). The GGUF files were created and quantized with:

bash

python convert_hf_to_gguf.py G9v3-39A5B \
  --outfile G9v3-39A5B-BF16.gguf \
  --outtype bf16

./build/bin/llama-quantize \
  G9v3-39A5B-BF16.gguf \
  G9v3-39A5B-Q8_0.gguf \
  Q8_0 32

./build/bin/llama-quantize \
  G9v3-39A5B-BF16.gguf \
  G9v3-39A5B-Q4_K_M.gguf \
  Q4_K_M 32

The importance matrix for IQ4_XS was computed on the BF16 GGUF over 165 chunks (2048 tokens each, ~338k tokens) of a curated calibration mix (prose, code, technical, and multilingual text, including Chinese, from the froggeric/imatrix collection):

bash

./build/bin/llama-imatrix \
  -m G9v3-39A5B-BF16.gguf \
  -f g9v3-calib.txt \
  -c 2048 --chunks 256 -t 32 \
  -o G9v3-39A5B.imatrix.gguf

./build/bin/llama-quantize \
  --imatrix G9v3-39A5B.imatrix.gguf \
  G9v3-39A5B-BF16.gguf \
  G9v3-39A5B-IQ4_XS.gguf \
  IQ4_XS 32

Conversion environment:

llama.cpp base commit: 9d57ce456 (b10436)
G9v3 support commit: 77b13da1a
GGUF version: V3
Tensor count: 568 in each file
BF16 weight size: 74,371.15 MiB, 16.01 bits per weight
Q8_0 weight size: 39,553.33 MiB, 8.51 bits per weight
Q4_K_M weight size: 22,462.90 MiB, 4.84 bits per weight
IQ4_XS weight size: 19,899.92 MiB, 4.28 bits per weight

Validation and performance

The Q4_K_M model was tested end-to-end with both llama-cli and llama-server on a CPU-only system. The performance figures below apply only to Q4_K_M.

Hardware	Context	Threads	Prompt processing	Generation
AMD Ryzen 9 9950X, 64 GiB RAM, CPU only	4,096	32	50.18–57.9 tokens/s	10.0–10.46 tokens/s
AMD Ryzen 9 9950X, 64 GiB RAM, CPU only (IQ4_XS)	2,048	32	—	11.29 tokens/s

The exact rate will vary with prompt length, thread count, memory speed, build options, and operating system.

Validation performed (2026-08-19 checkpoint revision):

Successful load and text generation from the Q4_K_M file
Successful load and text generation from the IQ4_XS file (imatrix quantization): correct reasoning and answer, 11.29 tokens/s generation
Successful /v1/chat/completions request with separate reasoning content
BF16 GGUF metadata and all 568 tensor shapes loaded successfully
BF16 file successfully used as the source for both the Q8_0 and Q4_K_M quantizations (all 568 tensors streamed without error)
Importance matrix covers all 37 MoE layers and 904 tensor entries; only three layer-24 expert tensors at 99.69% activation coverage (one rarely routed expert), all other entries fully populated
Converter dry run passed against all 16 source checkpoint shards after rebasing onto llama.cpp b10436
Synthetic G9v3 CPU architecture test: zero numerical error
GGUF serialization round-trip test: passed

The BF16 file was not generation-benchmarked on the 64 GiB validation machine; it requires a system with more memory for representative testing.

Sampling

The base model authors recommend the following settings:

Mode	Temperature	Top-p
Think	1.0	0.95
No Think	0.7	0.95

Limitations

Q4_K_M is a lossy quantization and may perform differently from the BF16 file and original checkpoint.
The BF16 GGUF preserves source precision but has much higher memory and storage requirements.
G9v3 is a preview model and may produce inaccurate, biased, or unsafe output.
Outputs should be independently reviewed before use in medical, legal, financial, safety-critical, or other high-stakes settings.
This GGUF currently depends on the linked G9v3-enabled llama.cpp branch.
Long-context capability was not benchmarked as part of this conversion test.

License and attribution

The original model is released by AI9Stars under the Apache License 2.0. This conversion retains that license. See the base model repository for its complete license terms and attribution information.

This conversion is independently produced and is not an official AI9Stars release.

linuxid10t/G9v3-39A5B-GGUF

G9v3-39A5B GGUF

Available files

Model details

llama.cpp setup

Command line

OpenAI-compatible server

Conversion provenance

Validation and performance

Sampling

Limitations

License and attribution

linuxid10t/
G9v3-39A5B-GGUF