You can deploy this model and start earning money today!

noctrex/
Ling-3.0-tiny-MXFP4_MOE-GGUF

Name: Ling-3.0-tiny-MXFP4_MOE-GGUF
Availability: InStock

gguf

text-generation

inclusionAI/Ling-3.0-tiny

quantized

endpoints_compatible

conversational

Views

No views yet

These are MXFP4 quantizations of the model inclusionAI / Ling-3.0-tiny

Quick Start

Download the latest release of llama.cpp.
Download your preferred model variant from below.

Which version should I choose?

All FP4 variants use MXFP4 for the MoE (Mixture of Experts) weights to keep the model efficient.
I've included also a new type Q8_XL_MOE, that uses Q8 for MoE tensors and BF16 for everything else. The difference lies in how the remaining tensors are handled:

Variant	Quality	Performance	Size	Recommendation
Q8_XL_MOE	⭐⭐⭐⭐⭐	Variable*	8.77GiB	Maximum quality, uses Q8 instead of FP4 for the MoE weights.
BF16	⭐⭐⭐	Variable*	4.54GiB	Best for maximum accuracy; original unquantized weights.
F16	⭐⭐	Fast	4.94GiB	Great alternative if BF16 is slow on your hardware.
Q8	⭐	Fastest	4.94GiB	Balanced performance and memory usage.

Note: On some older architectures, BF16 may be slower than F16.
Check that your GPU supports native BF16

Recommended parameters from inclusionAI:

temperature=1.0
top_p=0.95
top_k=20

noctrex/Ling-3.0-tiny-MXFP4_MOE-GGUF

Quick Start

Which version should I choose?

noctrex/
Ling-3.0-tiny-MXFP4_MOE-GGUF