SC117/Ling-3.0-tiny-abliterated-APEX-GGUF

APEX Abliterated 7.9B MoE

Ling-3.0-tiny

English | 📖 中文文档

Abliterated APEX GGUF quants of inclusionAI/Ling-3.0-tiny — 7.9B total / 1.3B active hybrid-reasoning MoE

🔓 Abliteration

Refusal direction removed with abliterix using LoRA steering search on a bnb-4bit loaded model. Over 130 candidate steering configurations were explored; the Pareto-selected recipe was merged into the BF16 weights.

85% fewer refusals (15/100 vs 98/100 baseline) at 0.0677 KL divergence. Post-abliteration capability spot-checks (math, logic, coding, translation, knowledge) all pass with no degradation observed.

💡 What is APEX?

These GGUF files are quantized using APEX, a novel MoE-aware mixed-precision quantization technique that outperforms standard quantization methods while being significantly smaller.

APEX beats Q8_0 perplexity at half the size — and even beats F16.

APEX classifies every tensor by its role — routed expert, shared expert, or attention — and applies a layer-wise precision gradient, giving the most sensitive edge layers higher precision and compressing the redundant middle layers more aggressively. Ling-3.0-tiny's 128 routed experts (only 8 active per token) make it an ideal candidate.

📦 APEX Quantization Tiers

File	Size	BPW	Best For
*-APEX-I-Quality.gguf	5.77 GB	5.84	Highest quality, best accuracy
*-APEX-I-Balanced.gguf	5.96 GB	6.03	Best all-rounder, recommended
*-APEX-I-Compact.gguf	3.99 GB	4.10	Best quality/size ratio, 8 GB GPUs
*-APEX-Mini.gguf	3.41 GB	3.45	Smallest viable, 6 GB GPUs, long context

All tiers quantized from a single BF16 source (15.07 GB) with the same diverse imatrix. Expert tensor layout: edge layers (L0–4, L19–23) keep higher precision than middle layers (L10–13); shared experts stay at Q8_0; the router is never quantized.

📐 I-Variant: Diverse Imatrix Calibration

All tiers use a diverse calibration dataset spanning chat, code, reasoning, and tool-calling — no Wikipedia (500 chunks). This produces higher accuracy on real-world benchmarks, lower KL divergence, and only a tiny perplexity increase on wikitext.

🧠 Model Details

Architecture	BailingMoeV3 — hybrid KDA/MLA linear-attention MoE
Parameters	7.9B total, 1.3B active per token
Experts	128 routed experts + 1 shared expert, 8 routed active per token
Layers	24 layers, 3:1 KDA–MLA stacking
Context	131,072 tokens native
Reasoning	Native hybrid reasoning (thinking mode on by default)
Abliteration	abliterix LoRA steering (85% fewer refusals, 0.0677 KL)

🚀 Usage

llama.cpp

hf download SC117/Ling-3.0-tiny-abliterated-APEX-GGUF --include "*.gguf" --local-dir ./models ./llama-server -m ./models/Ling-3.0-tiny-abliterated-APEX-I-Balanced.gguf -ngl 99 -c 32768 --flash-attn on --jinja

⚠️ bailingmoe3 architecture support: these files require a llama.cpp build that includes PR #26608 (BailingMoE3). Standard releases may not load them yet — check your build version first.

Ollama

echo 'FROM ./Ling-3.0-tiny-abliterated-APEX-I-Balanced.gguf' > Modelfile ollama create ling-tiny-abliterated -f Modelfile && ollama run ling-tiny-abliterated

🎛️ Recommended Settings

Mode	Parameters
Thinking (default)	temp=1.0, top_p=0.95, top_k=20
Fast / instruct	temp=0.7, top_p=0.8, top_k=20 (disable thinking via chat template)

Links

• Original Model: https://huggingface.co/inclusionAI/Ling-3.0-tiny
• APEX Quantization: https://github.com/mudler/apex-quant
• abliterix: https://github.com/wuwangzhang1216/abliterix
• BailingMoE3 GGUF support (llama.cpp): https://github.com/ggml-org/llama.cpp/pull/26608

Citation

@misc{ling3tiny,
title = {{Ling-3.0-tiny}: A Lightweight Hybrid Reasoning MoE Model},
url = {https://huggingface.co/inclusionAI/Ling-3.0-tiny},
author = {{inclusionAI}},
year = {2026}
}