Unofficial GGUF conversion and importance-matrix quantizations of inclusionAI/Ling-3.0-tiny, created from immutable source revision a2ee06c0. No fine-tuning, merging, or other parameter training was performed. The original model documentation, intended use, benchmark claims, and limitations remain authoritative.

Experimental runtime requirement

As of 2026-08-11, BailingMoE3 support remains unmerged in upstream llama.cpp. These files were converted and validated with PR #26608 at exact commit d8d8625. This includes the Q-LoRA path required by Ling-3.0-tiny (q_lora_rank=256) from 517b4675 and the pinned multi-argument tool-parser fix 0266ebca. Stock or older llama.cpp binaries and other GGUF runtimes may reject this architecture or produce incorrect output until they incorporate equivalent support.

Preserved model facts

BailingMoeV3 hybrid KDA/MLA sparse MoE, 526 GGUF tensors
7,893,392,800 parameters total; approximately 1.3B active per token
24 layers; 128 routed experts, 8 selected per token, plus 1 shared expert
Q-LoRA rank 256 and KV-LoRA rank 512
Native configured context: 131,072 tokens
Embedded tokenizer and source chat template
No NEXTN/MTP layers (num_nextn_predict_layers=0)

The source identifies itself as Transformers model_type=bailing_hybrid with BailingMoeV3ForCausalLM; the pinned converter intentionally maps that model to GGUF general.architecture=bailingmoe3. This is not a model-family mismatch.

The original card's 256K command uses an external YaRN/runtime override. This release preserves the checkpoint's native 131,072-token configuration and does not claim validated 256K operation. Do not enable MTP speculative decoding for this Tiny checkpoint.

Files and recommendations

File	Quant	Size	Matrix	Suggested use
`Ling-3.0-tiny-BF16.gguf`	BF16	14.72 GiB	No	Exact GGUF reference/requantization source
`Ling-3.0-tiny-Q8_0.gguf`	Q8_0	7.83 GiB	No	Highest-fidelity quantized option
`Ling-3.0-tiny-Q6_K.gguf`	Q6_K	6.05 GiB	Yes	Quality-first practical choice
`Ling-3.0-tiny-Q5_K_M.gguf`	Q5_K_M	5.25 GiB	Yes	Recommended quality/size balance
`Ling-3.0-tiny-Q4_K_M.gguf`	Q4_K_M	4.49 GiB	Yes	Recommended lower-memory default
`Ling-3.0-tiny-Q4_K_S.gguf`	Q4_K_S	4.24 GiB	Yes	Smaller K-quant alternative
`Ling-3.0-tiny-IQ4_XS.gguf`	IQ4_XS	3.99 GiB	Yes	Most compact 4-bit option
`Ling-3.0-tiny-Q3_K_M.gguf`	Q3_K_M	3.58 GiB	Yes	Larger K-quant 3-bit tier
`Ling-3.0-tiny-IQ3_M.gguf`	IQ3_M	3.31 GiB	Yes	Smaller 3-bit tier
`Ling-3.0-tiny-IQ2_M.gguf`	IQ2_M	2.52 GiB	Yes	Extreme compression; substantial loss
`Ling-3.0-tiny-imatrix.gguf`	Auxiliary	41.98 MiB	—	Reproducing importance-aware quants

If memory permits, prefer Q6_K or Q8_0 for fidelity. Q5_K_M is the quality-oriented general recommendation; Q4_K_M is the lower-memory default. IQ3_M and IQ2_M are specialized memory-constrained choices; the measured loss at IQ2_M is large enough that it should not be a default. File size is not total runtime memory: context length, state/KV caches, backend, and GPU offload add overhead. IQ backend support varies, so use the pinned runtime until equivalent BailingMoE3 support lands elsewhere.

Checksums are in SHA256SUMS.

Download and run

bash

hf download Mike0021/Ling-3.0-tiny-GGUF \
  --include "Ling-3.0-tiny-Q5_K_M.gguf" \
  --local-dir ./models

Build the tested unmerged runtime (review the PR before running it):

bash

git clone --filter=blob:none https://github.com/ggml-org/llama.cpp.git
git -C llama.cpp fetch origin refs/pull/26608/head:pr-26608
git -C llama.cpp checkout d8d862521e9ad842f2b47f3b392b039317782aa0
cmake -S llama.cpp -B llama.cpp/build -DGGML_CUDA=ON -DGGML_NATIVE=OFF
cmake --build llama.cpp/build --config Release --parallel

For a CPU-only build, omit -DGGML_CUDA=ON. This server example deliberately starts at 8K context to keep memory moderate:

bash

./llama.cpp/build/bin/llama-server \
  -m ./models/Ling-3.0-tiny-Q5_K_M.gguf \
  --alias ling-3.0-tiny --host 127.0.0.1 --port 8080 \
  --jinja -c 8192 -ngl 999

bash

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ling-3.0-tiny",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "stream": false
  }'

These sampling settings follow the original model's recommendations. Thinking is enabled by the embedded source chat template by default.

To disable thinking in the pinned server, pass "chat_template_kwargs":{"enable_thinking":false} in the request. Keep --jinja enabled so the embedded template is applied.

The pinned runtime logs special_eos_id is not in special_eog_ids while loading this tokenizer. The raw arithmetic reference stopped on token 156895 in Transformers, and Q4_K_M server stop behavior was tested as described below, but the warning is preserved here because it has not yet been resolved upstream.

Conversion provenance

Item	Value
Source	`inclusionAI/Ling-3.0-tiny@a2ee06c0f2de5b171701aee7f73f70a1da75483b`
Source weights	32 safetensors shards, 15,787,992,416 bytes
Converter/runtime	`aetherbird/llama.cpp@d8d862521e9ad842f2b47f3b392b039317782aa0` (upstream PR #26608)
Conversion	BF16 GGUF, then every quant directly from BF16
Detailed provenance	`conversion_manifest.json`
Source shard hashes	`source-safetensors.sha256`
Core reproduction commands	`REPRODUCE.md`

Importance-matrix calibration

Importance-aware files used two complementary, pinned calibration sources. The primary corpus was lemon07r/bartowski-imatrix-v5-semantic at revision a306f203ee4323e0afe846ae02c2daafe17384d9. Its 2,075 semantic samples span 13 languages and include code, math, science, dialogue, and Q&A, which is substantially broader than English-only WikiText calibration.

An additive second pass used combined_all_micro.parquet from eaddario/imatrix-calibration at revision e87ed55dcba9d9c3a3e41539f3e728e981b1daa4. This MIT-licensed mixture adds multilingual text plus tool-use, math, and code prompts. It was added because the first pass left one routed expert unobserved in one layer; the release gate requires every routed-expert slot to have a nonzero count.

Input: bartowski-imatrix-v5-semantic.txt
SHA-256: ff879b5a748f822ef539e43c596a3f44ab922f0295ee209d4220d9f86e86a063
1,496,006 bytes; 6,318 serialized lines
Supplement parquet SHA-256: 94389921e1f67b180a99de28c3090b41ce6f1960eb13abad21b7eba7cbe11b26
Extracted supplement SHA-256: fdb2d41abf04a2fb207502741a561a5a9ab385eb0c44a450eae676c410955946 (1,008,653 bytes; 3,130 serialized lines)
Context / batch / ubatch: 4096 / 4096 / 512
Complete 4,096-token chunks processed: 162 (663,552 tokens); 5,338 trailing tokens excluded
Matrix entries: 332
Per-expert count values: 8,832
Routed-expert slots with zero observations: 0

The matrix is the modern GGUF imatrix format. It contains 69 expert-count vectors of length 128 (8,832 layer/tensor expert slots); “zero” is measured over those slots, not over 128 globally unique expert IDs. Output-tensor statistics were intentionally not collected: the pinned llama.cpp imatrix documentation says it is typically better not to use importance statistics when quantizing output.weight, and therefore defaults --process-output to false. Observed per-slot counts ranged from 16 to 326,023 (median 33,514); a distribution summary and the lowest-count slots are recorded in validation/imatrix.json.

The final matrix SHA-256 is e8b15d131f9ce294f922c5c387f7a69829c12100d6a35bb1635a2b859083c3f0. llama-quantize embeds only one quantize.imatrix.dataset scalar, so the importance-aware model files name the primary corpus even though the final matrix contains both ordered passes. The manifest is the authoritative record of the two-source lineage. It also records the absolute paths embedded by the quantizer; changing those paths can preserve tensor values while changing the GGUF file hash.

The corpus was used only to collect activation statistics. It was not used to train or fine-tune the model and is not an evaluation set.

Held-out validation

Validation used the separate WikiText-2 test file from ggml-org/ci@927b3642933080f1b0e811e2f916e14c292992f9; this file was not used for imatrix collection. Content-level uniqueness from all calibration material or from the model's original pretraining data is not asserted. The extracted wiki.test.raw SHA-256 is 173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08. PPL and BF16-relative KLD used 32 fixed sequential chunks at context/batch/ubatch 512, scoring 8,160 held-out tokens. Exact commands are in REPRODUCE.md, and machine-readable results are under validation/.

Artifact	Loads	Greedy raw vs HF BF16	PPL ± SE	ΔPPL	Mean KLD ± SE (nats)
BF16 self	Pass	Exact	11.901303 ± 0.415179	+0.033176	0.000000 ± 0.000000
Q8_0	Pass	Exact	11.812842 ± 0.410345	−0.055285	0.011688 ± 0.000329
Q6_K	Pass	Exact	11.873857 ± 0.413599	+0.005730	0.023357 ± 0.000625
Q5_K_M	Pass	Exact	12.087854 ± 0.422594	+0.219727	0.053244 ± 0.001318
Q4_K_M	Pass	Exact	12.651529 ± 0.447483	+0.783402	0.130069 ± 0.003051
Q4_K_S	Pass	Exact	12.608386 ± 0.443531	+0.740259	0.138631 ± 0.003234
IQ4_XS	Pass	Exact	12.640906 ± 0.445231	+0.772779	0.155524 ± 0.003489
Q3_K_M	Pass	Exact	13.649613 ± 0.484819	+1.781486	0.301154 ± 0.006362
IQ3_M	Pass	Exact	12.967071 ± 0.446764	+1.098944	0.312063 ± 0.006496
IQ2_M	Pass	Exact	16.362374 ± 0.564546	+4.494247	0.696147 ± 0.011718

These tests measure conversion and quantization behavior, not general model capability or safety. Results are comparable only under the documented tokenizer, context, chunk, and pinned-runtime settings. The stored BF16 reference has PPL 11.868127 ± 0.412222. BF16 self-comparison establishes the uint16 stored-log-probability/backend resolution; mean KLD rounded to 0.000000 nats in this run. Small negative ΔPPL values, such as Q8_0, are within sampling uncertainty and do not mean the quant is better than BF16.

“Loads” means the pinned runtime completed its tensor integrity/load check and a graph evaluation. “Greedy raw vs HF BF16” compares a deterministic 12-token continuation against a separately generated Transformers BF16 reference. The validator binds both runtimes to the exact same full prompt; all ten artifacts matched this one shallow case exactly. This is a conversion smoke test, not a claim that quantized logits or arbitrary generations equal BF16. All six tokenizer test cases, including Chinese, code, whitespace, multilingual text, and special tokens, matched Transformers token IDs exactly.

Q6_K contains six Q8_0 fallbacks because those narrow MLA tensors cannot use the requested block width. The 3-bit and 2-bit files likewise contain exactly six documented MLA fallbacks. Their complete tensor-type inventories are in the structure reports and manifest.

Matrix ablation

A direct Q4_K_M A/B against a temporary no-matrix quant gave mixed evidence. The matrix lowered the mean KLD point estimate from 0.131547 to 0.130069 nats and raised the same-top-token point estimate from 84.596% to 85.221%, while PPL moved from 12.357816 to 12.651529. This is not presented as a universal quality gain; the broader calibration coverage and those KLD/same-top point estimate shifts motivated retaining the matrix build. See kld-Q4_K_M-ab.json.

Fixed multiple-choice collapse screen

The pinned mmlu-validation.bin contains 1,548 four-choice tasks. A fixed seed-1 subset of 500 was used as a regression/collapse check, not as a model capability benchmark. The tool's log says “TruthfulQA,” but the supplied input is the pinned MMLU validation binary (SHA-256 470af3a74eccacfaf6f43b08aabf510f61e6c92fe20d17241ded934151e225fa).

Artifact	Accuracy ± SE
BF16	38.2% ± 2.1751%
Q5_K_M	39.0% ± 2.1835%
Q4_K_M	38.8% ± 2.1814%
Q4_K_S	39.0% ± 2.1835%
IQ4_XS	37.2% ± 2.1637%
Q3_K_M	37.8% ± 2.1707%
IQ3_M	37.8% ± 2.1707%
IQ2_M	34.8% ± 2.1324%

Random chance was 25.0% ± 1.9384%. Q8_0 and Q6_K were not run through this auxiliary screen; their held-out KLD results are the stronger fidelity evidence.

Long-context and server checks

BF16, Q4_K_M, and the most aggressive IQ2_M completed a one-chunk 32,768-token perplexity/prefill evaluation at batch 4,096: respectively 23.3709, 25.7803, and 34.6812 PPL. Other artifacts were validated at context 512. The checkpoint's native 131,072-token limit and the external 256K YaRN configuration were not exercised.

Q4_K_M was also tested through llama-server --jinja. Thinking-disabled and thinking-enabled requests both stopped normally, the latter exposed separate reasoning content, a Chinese prompt returned 巴黎, and a required tool request produced get_weather with both location=Paris and unit=celsius arguments and finish_reason=tool_calls. These server results apply to Q4_K_M; they are not generalized to every quant.

Rejected candidates

Two generated candidates were deliberately not published. IQ4_NL was only 28,606,464 bytes smaller than Q4_K_S while its KLD rose from 0.138631 to 0.149734. MXFP4_MOE passed an exact 69-tensor routed-expert whitelist, but at 4,718,248,800 bytes and 0.267021 KLD it was larger and much less faithful than Q4_K_S. On the tested RTX PRO 4500 Blackwell it improved 512-token prompt throughput by 17.9% but reduced 128-token generation throughput by 8.2%. Full measurements are in rejected-candidates.json.

As a post-hoc independent cross-check, the canonical BF16 and Q8_0 SHA-256 values exactly match bloomer010/Ling-3.0-tiny-GGUF@598201. That repository was not used as a weight source.

Limitations and attribution

Runtime support is experimental and tied to an unmerged llama.cpp revision.
Quantization can change factuality, reasoning, tool-call formatting, and multilingual behavior; validate the chosen file on your workload.
Long contexts add substantial memory and were not exhaustively exercised for every artifact.
No new safety evaluation was performed. The source model's limitations and acceptable-use considerations still apply.
This is an unofficial conversion, not endorsed by InclusionAI, Hugging Face, or llama.cpp maintainers.

The source card declares the MIT license. Original authorship belongs to InclusionAI; this repository provides an unofficial format conversion by Mike0021.