README.md

---
license: mit
library_name: gguf
pipeline_tag: text-generation
inference: false
base_model: inclusionAI/Ling-3.0-tiny
base_model_relation: quantized
model_name: Ling-3.0-tiny GGUF
quantized_by: Mike0021
tags:
  - gguf
  - llama.cpp
  - bailingmoe3
  - mixture-of-experts
  - quantized
  - reasoning
  - conversational
---

# Ling-3.0-tiny GGUF

Unofficial GGUF conversion and importance-matrix quantizations of
[inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny),
created from immutable source revision
[`a2ee06c0`](https://huggingface.co/inclusionAI/Ling-3.0-tiny/tree/a2ee06c0f2de5b171701aee7f73f70a1da75483b).
No fine-tuning, merging, or other parameter training was performed. The
original model documentation, intended use, benchmark claims, and limitations
remain authoritative.

> **Experimental runtime requirement**
>
> As of 2026-08-11, BailingMoE3 support remains unmerged in upstream
> `llama.cpp`. These files were converted and validated with
> [PR #26608](https://github.com/ggml-org/llama.cpp/pull/26608) at exact commit
> [`d8d8625`](https://github.com/aetherbird/llama.cpp/commit/d8d862521e9ad842f2b47f3b392b039317782aa0).
> This includes the Q-LoRA path required by Ling-3.0-tiny
> (`q_lora_rank=256`) from
> [`517b4675`](https://github.com/aetherbird/llama.cpp/commit/517b467544f732ddabb3f7727932f8d004ad9457)
> and the pinned multi-argument tool-parser fix
> [`0266ebca`](https://github.com/aetherbird/llama.cpp/commit/0266ebca66bd95b7a85d37b8ca08ccf9812b85cc).
> Stock or older llama.cpp binaries and other GGUF
> runtimes may reject this architecture or produce incorrect output until they
> incorporate equivalent support.

## Preserved model facts

- BailingMoeV3 hybrid KDA/MLA sparse MoE, 526 GGUF tensors
- 7,893,392,800 parameters total; approximately 1.3B active per token
- 24 layers; 128 routed experts, 8 selected per token, plus 1 shared expert
- Q-LoRA rank 256 and KV-LoRA rank 512
- Native configured context: 131,072 tokens
- Embedded tokenizer and source chat template
- No NEXTN/MTP layers (`num_nextn_predict_layers=0`)

The source identifies itself as Transformers `model_type=bailing_hybrid` with
`BailingMoeV3ForCausalLM`; the pinned converter intentionally maps that model
to GGUF `general.architecture=bailingmoe3`. This is not a model-family
mismatch.

The original card's 256K command uses an external YaRN/runtime override. This
release preserves the checkpoint's native 131,072-token configuration and does
not claim validated 256K operation. Do not enable MTP speculative decoding for
this Tiny checkpoint.

## Files and recommendations

| File | Quant | Size | Matrix | Suggested use |
|---|---:|---:|:---:|---|
| `Ling-3.0-tiny-BF16.gguf` | BF16 | 14.72 GiB | No | Exact GGUF reference/requantization source |
| `Ling-3.0-tiny-Q8_0.gguf` | Q8_0 | 7.83 GiB | No | Highest-fidelity quantized option |
| `Ling-3.0-tiny-Q6_K.gguf` | Q6_K | 6.05 GiB | Yes | Quality-first practical choice |
| `Ling-3.0-tiny-Q5_K_M.gguf` | Q5_K_M | 5.25 GiB | Yes | Recommended quality/size balance |
| `Ling-3.0-tiny-Q4_K_M.gguf` | Q4_K_M | 4.49 GiB | Yes | Recommended lower-memory default |
| `Ling-3.0-tiny-Q4_K_S.gguf` | Q4_K_S | 4.24 GiB | Yes | Smaller K-quant alternative |
| `Ling-3.0-tiny-IQ4_XS.gguf` | IQ4_XS | 3.99 GiB | Yes | Most compact 4-bit option |
| `Ling-3.0-tiny-Q3_K_M.gguf` | Q3_K_M | 3.58 GiB | Yes | Larger K-quant 3-bit tier |
| `Ling-3.0-tiny-IQ3_M.gguf` | IQ3_M | 3.31 GiB | Yes | Smaller 3-bit tier |
| `Ling-3.0-tiny-IQ2_M.gguf` | IQ2_M | 2.52 GiB | Yes | Extreme compression; substantial loss |
| `Ling-3.0-tiny-imatrix.gguf` | Auxiliary | 41.98 MiB | — | Reproducing importance-aware quants |

If memory permits, prefer Q6_K or Q8_0 for fidelity. Q5_K_M is the
quality-oriented general recommendation; Q4_K_M is the lower-memory default.
IQ3_M and IQ2_M are specialized memory-constrained choices; the measured loss
at IQ2_M is large enough that it should not be a default. File size is not
total runtime memory: context length, state/KV caches, backend, and GPU offload
add overhead. IQ backend support varies, so use the pinned runtime until
equivalent BailingMoE3 support lands elsewhere.

Checksums are in [`SHA256SUMS`](./SHA256SUMS).

## Download and run

```bash
hf download Mike0021/Ling-3.0-tiny-GGUF \
  --include "Ling-3.0-tiny-Q5_K_M.gguf" \
  --local-dir ./models
```

Build the tested unmerged runtime (review the PR before running it):

```bash
git clone --filter=blob:none https://github.com/ggml-org/llama.cpp.git
git -C llama.cpp fetch origin refs/pull/26608/head:pr-26608
git -C llama.cpp checkout d8d862521e9ad842f2b47f3b392b039317782aa0
cmake -S llama.cpp -B llama.cpp/build -DGGML_CUDA=ON -DGGML_NATIVE=OFF
cmake --build llama.cpp/build --config Release --parallel
```

For a CPU-only build, omit `-DGGML_CUDA=ON`. This server example deliberately
starts at 8K context to keep memory moderate:

```bash
./llama.cpp/build/bin/llama-server \
  -m ./models/Ling-3.0-tiny-Q5_K_M.gguf \
  --alias ling-3.0-tiny --host 127.0.0.1 --port 8080 \
  --jinja -c 8192 -ngl 999
```

```bash
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ling-3.0-tiny",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "stream": false
  }'
```

These sampling settings follow the original model's recommendations. Thinking
is enabled by the embedded source chat template by default.

To disable thinking in the pinned server, pass
`"chat_template_kwargs":{"enable_thinking":false}` in the request. Keep
`--jinja` enabled so the embedded template is applied.

The pinned runtime logs `special_eos_id is not in special_eog_ids` while
loading this tokenizer. The raw arithmetic reference stopped on token 156895
in Transformers, and Q4_K_M server stop behavior was tested as described
below, but the warning is preserved here because it has not yet been resolved
upstream.

## Conversion provenance

| Item | Value |
|---|---|
| Source | `inclusionAI/Ling-3.0-tiny@a2ee06c0f2de5b171701aee7f73f70a1da75483b` |
| Source weights | 32 safetensors shards, 15,787,992,416 bytes |
| Converter/runtime | `aetherbird/llama.cpp@d8d862521e9ad842f2b47f3b392b039317782aa0` (upstream PR #26608) |
| Conversion | BF16 GGUF, then every quant directly from BF16 |
| Detailed provenance | [`conversion_manifest.json`](./conversion_manifest.json) |
| Source shard hashes | [`source-safetensors.sha256`](./source-safetensors.sha256) |
| Core reproduction commands | [`REPRODUCE.md`](./REPRODUCE.md) |

## Importance-matrix calibration

Importance-aware files used two complementary, pinned calibration sources.
The primary corpus was
[`lemon07r/bartowski-imatrix-v5-semantic`](https://huggingface.co/datasets/lemon07r/bartowski-imatrix-v5-semantic/tree/a306f203ee4323e0afe846ae02c2daafe17384d9)
at revision `a306f203ee4323e0afe846ae02c2daafe17384d9`. Its 2,075 semantic
samples span 13 languages and include code, math, science, dialogue, and Q&A,
which is substantially broader than English-only WikiText calibration.

An additive second pass used `combined_all_micro.parquet` from
[`eaddario/imatrix-calibration`](https://huggingface.co/datasets/eaddario/imatrix-calibration/tree/e87ed55dcba9d9c3a3e41539f3e728e981b1daa4)
at revision `e87ed55dcba9d9c3a3e41539f3e728e981b1daa4`. This MIT-licensed
mixture adds multilingual text plus tool-use, math, and code prompts. It was
added because the first pass left one routed expert unobserved in one layer;
the release gate requires every routed-expert slot to have a nonzero count.

- Input: `bartowski-imatrix-v5-semantic.txt`
- SHA-256: `ff879b5a748f822ef539e43c596a3f44ab922f0295ee209d4220d9f86e86a063`
- 1,496,006 bytes; 6,318 serialized lines
- Supplement parquet SHA-256:
  `94389921e1f67b180a99de28c3090b41ce6f1960eb13abad21b7eba7cbe11b26`
- Extracted supplement SHA-256:
  `fdb2d41abf04a2fb207502741a561a5a9ab385eb0c44a450eae676c410955946`
  (1,008,653 bytes; 3,130 serialized lines)
- Context / batch / ubatch: 4096 / 4096 / 512
- Complete 4,096-token chunks processed: 162
  (663,552 tokens); 5,338 trailing tokens excluded
- Matrix entries: 332
- Per-expert count values: 8,832
- Routed-expert slots with zero observations: 0

The matrix is the modern GGUF imatrix format. It contains 69 expert-count
vectors of length 128 (8,832 layer/tensor expert slots); “zero” is measured
over those slots, not over 128 globally unique expert IDs. Output-tensor
statistics were intentionally not collected: the pinned llama.cpp imatrix
documentation says it is typically better not to use importance statistics
when quantizing `output.weight`, and therefore defaults `--process-output` to
false.
Observed per-slot counts ranged from 16 to 326,023 (median 33,514); a
distribution summary and the lowest-count slots are recorded in
`validation/imatrix.json`.

The final matrix SHA-256 is
`e8b15d131f9ce294f922c5c387f7a69829c12100d6a35bb1635a2b859083c3f0`.
`llama-quantize` embeds only one `quantize.imatrix.dataset` scalar, so the
importance-aware model files name the primary corpus even though the final
matrix contains both ordered passes. The manifest is the authoritative record
of the two-source lineage. It also records the absolute paths embedded by the
quantizer; changing those paths can preserve tensor values while changing the
GGUF file hash.

The corpus was used only to collect activation statistics. It was not used to
train or fine-tune the model and is not an evaluation set.

## Held-out validation

Validation used the separate WikiText-2 test file from
`ggml-org/ci@927b3642933080f1b0e811e2f916e14c292992f9`; this file was not
used for imatrix collection. Content-level uniqueness from all calibration
material or from the model's original pretraining data is not asserted. The
extracted `wiki.test.raw` SHA-256 is
`173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08`.
PPL and BF16-relative KLD used 32 fixed sequential chunks at
context/batch/ubatch 512, scoring
8,160 held-out tokens. Exact commands are in
[`REPRODUCE.md`](./REPRODUCE.md), and machine-readable results are under
[`validation/`](./validation/).

| Artifact | Loads | Greedy raw vs HF BF16 | PPL ± SE | ΔPPL | Mean KLD ± SE (nats) |
|---|:---:|:---:|---:|---:|---:|
| BF16 self | Pass | Exact | 11.901303 ± 0.415179 | +0.033176 | 0.000000 ± 0.000000 |
| Q8_0 | Pass | Exact | 11.812842 ± 0.410345 | −0.055285 | 0.011688 ± 0.000329 |
| Q6_K | Pass | Exact | 11.873857 ± 0.413599 | +0.005730 | 0.023357 ± 0.000625 |
| Q5_K_M | Pass | Exact | 12.087854 ± 0.422594 | +0.219727 | 0.053244 ± 0.001318 |
| Q4_K_M | Pass | Exact | 12.651529 ± 0.447483 | +0.783402 | 0.130069 ± 0.003051 |
| Q4_K_S | Pass | Exact | 12.608386 ± 0.443531 | +0.740259 | 0.138631 ± 0.003234 |
| IQ4_XS | Pass | Exact | 12.640906 ± 0.445231 | +0.772779 | 0.155524 ± 0.003489 |
| Q3_K_M | Pass | Exact | 13.649613 ± 0.484819 | +1.781486 | 0.301154 ± 0.006362 |
| IQ3_M | Pass | Exact | 12.967071 ± 0.446764 | +1.098944 | 0.312063 ± 0.006496 |
| IQ2_M | Pass | Exact | 16.362374 ± 0.564546 | +4.494247 | 0.696147 ± 0.011718 |

These tests measure conversion and quantization behavior, not general model
capability or safety. Results are comparable only under the documented
tokenizer, context, chunk, and pinned-runtime settings. The stored BF16
reference has PPL 11.868127 ± 0.412222. BF16 self-comparison establishes the
uint16 stored-log-probability/backend resolution; mean KLD rounded to 0.000000
nats in this run. Small negative ΔPPL values, such as Q8_0, are within sampling
uncertainty and do not mean the quant is better than BF16.

“Loads” means the pinned runtime completed its tensor integrity/load check and
a graph evaluation. “Greedy raw vs HF BF16” compares a deterministic 12-token
continuation against a separately generated Transformers BF16 reference. The
validator binds both runtimes to the exact same full prompt; all ten artifacts
matched this one shallow case exactly. This is a conversion smoke test, not a
claim that quantized logits or arbitrary generations equal BF16. All six
tokenizer test cases, including Chinese, code, whitespace, multilingual text,
and special tokens, matched Transformers token IDs exactly.

Q6_K contains six Q8_0 fallbacks because those narrow MLA tensors cannot use
the requested block width. The 3-bit and 2-bit files likewise contain exactly
six documented MLA fallbacks. Their complete tensor-type inventories are in
the structure reports and manifest.

### Matrix ablation

A direct Q4_K_M A/B against a temporary no-matrix quant gave mixed evidence.
The matrix lowered the mean KLD point estimate from 0.131547 to 0.130069 nats
and raised the same-top-token point estimate from 84.596% to 85.221%, while
PPL moved from 12.357816 to 12.651529. This is not presented as a universal
quality gain; the broader calibration coverage and those KLD/same-top point
estimate shifts motivated retaining the matrix build. See
[`kld-Q4_K_M-ab.json`](./validation/kld-Q4_K_M-ab.json).

### Fixed multiple-choice collapse screen

The pinned `mmlu-validation.bin` contains 1,548 four-choice tasks. A fixed
seed-1 subset of 500 was used as a regression/collapse check, not as a model
capability benchmark. The tool's log says “TruthfulQA,” but the supplied input
is the pinned MMLU validation binary (SHA-256
`470af3a74eccacfaf6f43b08aabf510f61e6c92fe20d17241ded934151e225fa`).

| Artifact | Accuracy ± SE |
|---|---:|
| BF16 | 38.2% ± 2.1751% |
| Q5_K_M | 39.0% ± 2.1835% |
| Q4_K_M | 38.8% ± 2.1814% |
| Q4_K_S | 39.0% ± 2.1835% |
| IQ4_XS | 37.2% ± 2.1637% |
| Q3_K_M | 37.8% ± 2.1707% |
| IQ3_M | 37.8% ± 2.1707% |
| IQ2_M | 34.8% ± 2.1324% |

Random chance was 25.0% ± 1.9384%. Q8_0 and Q6_K were not run through this
auxiliary screen; their held-out KLD results are the stronger fidelity evidence.

### Long-context and server checks

BF16, Q4_K_M, and the most aggressive IQ2_M completed a one-chunk 32,768-token
perplexity/prefill evaluation at batch 4,096: respectively 23.3709, 25.7803,
and 34.6812 PPL. Other artifacts were validated at context 512. The checkpoint's
native 131,072-token limit and the external 256K YaRN configuration were not
exercised.

Q4_K_M was also tested through `llama-server --jinja`. Thinking-disabled and
thinking-enabled requests both stopped normally, the latter exposed separate
reasoning content, a Chinese prompt returned `巴黎`, and a required
tool request produced `get_weather` with both `location=Paris` and
`unit=celsius` arguments and `finish_reason=tool_calls`. These server results
apply to Q4_K_M; they are not generalized to every quant.

### Rejected candidates

Two generated candidates were deliberately not published. IQ4_NL was only
28,606,464 bytes smaller than Q4_K_S while its KLD rose from 0.138631 to
0.149734. MXFP4_MOE passed an exact 69-tensor routed-expert whitelist, but at
4,718,248,800 bytes and 0.267021 KLD it was larger and much less faithful than
Q4_K_S. On the tested RTX PRO 4500 Blackwell it improved 512-token prompt
throughput by 17.9% but reduced 128-token generation throughput by 8.2%.
Full measurements are in
[`rejected-candidates.json`](./validation/rejected-candidates.json).

As a post-hoc independent cross-check, the canonical BF16 and Q8_0 SHA-256
values exactly match
[`bloomer010/Ling-3.0-tiny-GGUF@598201`](https://huggingface.co/bloomer010/Ling-3.0-tiny-GGUF/tree/59820116411687d44e1333816609afca8c93aa0b).
That repository was not used as a weight source.

## Limitations and attribution

- Runtime support is experimental and tied to an unmerged llama.cpp revision.
- Quantization can change factuality, reasoning, tool-call formatting, and
  multilingual behavior; validate the chosen file on your workload.
- Long contexts add substantial memory and were not exhaustively exercised for
  every artifact.
- No new safety evaluation was performed. The source model's limitations and
  acceptable-use considerations still apply.
- This is an unofficial conversion, not endorsed by InclusionAI, Hugging Face,
  or llama.cpp maintainers.

The source card declares the MIT license. Original authorship belongs to
InclusionAI; this repository provides an unofficial format conversion by
Mike0021.