Quant	Size	Use case
Q5_K_M	6.02 GiB	Recommended balance of quality and size
Q4_K_M	5.24 GiB	Smaller and faster; passed the same chat-routing smoke tests
Q6_K	6.85 GiB	Higher fidelity
Q8_0	8.87 GiB	Highest-fidelity quant in this repository

These are single-file, text-only models. The inherited vision components are not included because Astrea's visual behavior was not part of the release evaluation.

Run with llama.cpp

Use a current build of llama.cpp; Qwen3.5 support is recent.

bash

llama-cli \
  -hf Altworld/Astrea-R8-Chat-9B-GGUF:Q5_K_M \
  -cnv \
  --reasoning off \
  --temp 0.8 \
  --min-p 0.025 \
  --repeat-penalty 1.08

For an OpenAI-compatible local server:

bash

llama-server \
  -hf Altworld/Astrea-R8-Chat-9B-GGUF:Q5_K_M \
  --reasoning off \
  --temp 0.8 \
  --min-p 0.025 \
  --repeat-penalty 1.08

Use --temp 0.2 for factual chat. The model carries its official chat template inside the GGUF and does not require a system prompt. Reasoning is disabled in the examples because Astrea's release behavior is direct response without a visible thinking block.

Validation

Each file was loaded and generated from with llama.cpp. The checks covered:

bare greeting with no system prompt;
native Astrea/Altworld identity;
ordinary factual chat;
supportive conversation;
creative-writing routing from a plain request with no scene card;
multi-turn recall using the embedded chat template.

Q4_K_M, the most aggressive quant here, correctly answered Good afternoon. as normal chat and produced a coherent story from Write a short story about a dragon who collects spoons. Q5_K_M correctly recalled a name across turns.

The GGUFs declare the base architecture's 262,144-token context. Actual usable context depends on your hardware, llama.cpp build, and runtime settings.

Conversion details

Converted from the official merged BF16 release with llama.cpp commit 76f46ad. The source checkpoint contains no MTP tensors, so conversion used --no-mtp. Quantization used llama.cpp's Q4_K_M, Q5_K_M, Q6_K, and Q8_0 presets.

About Astrea

Astrea was trained on top of Qwen3.5-9B using multi-round LoRA supervised fine-tuning, alternating conversational and narrative rounds and checking each round against Altworldbench. See the BF16 model card for benchmarks, methodology, examples, limitations, and the evaluation protocol.

Astrea is English-first and can still hallucinate. Verify high-stakes claims independently. Engine and quantization differences can affect output, so test your own prompts and settings.

License

Apache-2.0. See LICENSE and NOTICE.