Pluto Nano's native architecture (astrai_pluto) is not yet supported upstream in llama.cpp. These files were produced by re-mapping Pluto's tensors onto the qwen2_moe architecture with dummy zero-weight shared-expert tensors (Pluto uses top-1 routing without a shared expert). See the conversion script & rationale in GGUF.md.

Outputs match the bf16 safetensors version. Quality at low quantization (Q4_K_M) may degrade slightly because top-1 MoE has higher per-token sensitivity than the top-4 MoE that qwen2_moe quantizers assume.

Files

File	Size	Bits	Use case
`pluto-nano-0.5.fp16.gguf`	2.0 GB	16-bit	reference / fine-tune base
`pluto-nano-0.5.q8_0.gguf`	1.1 GB	8.5-bit	≈ FP8, near-zero quality loss
`pluto-nano-0.5.q6_K.gguf`	963 MB	6.6-bit	small + clean
`pluto-nano-0.5.q4_K_M.gguf`	679 MB	4.8-bit	smallest practical

Usage

bash

# Download
hf download ASTRAI-labs/pluto-nano-0.5-gguf pluto-nano-0.5.q8_0.gguf

# Run
./llama-cli -m pluto-nano-0.5.q8_0.gguf \
  --prompt "<|lang_en|>\n<|user|>\nHi, who are you?\n<|im_end|>\n<|assistant|>\n" \
  -n 200 --temp 0.7

Chat template

<|lang_{en|pt|es|zh|hi}|>
<|user|>
...question...
<|im_end|>
<|assistant|>
...response...
<|im_end|>

License

ASTRAI Closed License — see main repo.