Instructions to use kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
llama cli -hf kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
llama cli -hf kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
./llama-cli -hf kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
./build/bin/llama-cli -hf kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP

Use Docker

docker model run hf.co/kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP

LM Studio
Jan

vLLM

How to use kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP

Ollama
How to use kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF with Ollama:
```
ollama run hf.co/kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP
```

Unsloth Studio

How to use kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF to start chatting

How to use kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF with Docker Model Runner:
```
docker model run hf.co/kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP
```

Lemonade

How to use kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP

Run and chat with the model

lemonade run user.Ling-3.0-tiny-ROCmFP4-GGUF-Q4_0_ROCMFP

List all available models

lemonade list

Hermes Agent

How to use kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "kingjones777/Ling-3.0-tiny-ROCmFP4-GGUF:Q4_0_ROCMFP" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Ling-3.0-tiny-ROCmFP4-GGUF / README.md

kingjones777

card: add all-quant-variants table (4-bit / FP8 / FP8-AGENT)

529afca verified 1 day ago

preview code

Raw

History Blame Contribute Delete

9.77 kB

metadata

license: mit
base_model: inclusionAI/Ling-3.0-tiny
base_model_relation: quantized
tags:
  - rocmfpx
  - ai-max-395
  - ryzen-ai-max-395
  - amd
  - gguf
  - llama.cpp
  - rocm
  - gfx1151
  - strix-halo
  - bailingmoe3
  - moe
  - hybrid-linear-attention
language:
  - en
pipeline_tag: text-generation

⚠️ STOCK llama.cpp WILL NOT LOAD THIS MODEL

bailingmoe3 is not merged upstream, and Ling-3.0-tiny additionally needs the Q-LoRA attention path (q_lora_rank: 256) that flash-era builds lack. Ignore the auto-generated "Use this model" commands above — use the patch in patches/.

🚀 101.08 tok/s on AMD Ryzen AI MAX+ 395 (gfx1151 / Strix Halo) — 4.30 GiB, smaller than Q4_K_M (4.49 GiB). Verified on two independent machines.

✅ The patch you need is in this repo

patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch — applies to charlie12345/ROCmFPX at commit 2809dc5, verified with git apply --check.

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout 2809dc5
git apply patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch
cmake -B build -S . -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

Full build notes, per-architecture details and licence: patches/README.md in this repo.

⚠️ If you add files under src/models/, re-run cmake -B build -S . — the models/*.cpp GLOB is configure-time, so cmake --build alone will not link them.

Ling-3.0-tiny — ROCmFP4 (tier 102 `COHERENT`) GGUF

A 4-bit ROCmFP4 quantization of inclusionAI/Ling-3.0-tiny, built for AMD gfx1151 (Ryzen AI MAX+ 395 / Strix Halo) with per-tensor protection of the LM head, token embeddings, shared experts, router, and the hybrid model's recurrent state.


File	`Ling-3.0-tiny-Q4_0_ROCMFP4_COHERENT.gguf`
Size	4.2987 GiB (4,615,656,288 bytes)
BPW	4.676
ftype	`Q4_0_ROCMFP4_COHERENT` (102)
Source	BF16 GGUF (14.72 GiB) — lossless source, not a requantization
sha256	`fd9af463569509aca718f6ccacab388c74d8abe108511d5b6e9c00e3243b42b4`

⛔ REQUIRES A PATCHED llama.cpp — STOCK WILL NOT LOAD THIS

Two independent reasons, both unavoidable:

bailingmoe3 is not in upstream llama.cpp. Support is still open in PR #26608 (unmerged at time of writing).
Ling-3.0-tiny needs the Q-LoRA attention path. Its config sets q_lora_rank: 256 (q_a_proj → q_a_layernorm → q_b_proj). Several existing bailingmoe3 implementations were written against Ling-3.0-flash, which has q_lora_rank: null and therefore no query compression. On such a build, every GGUF of tiny fails — including the BF16 and Q4_K_M ones — typically at missing tensor 'blk.0.ssm_f.weight', before the Q-LoRA gap is even reached.

You need a build with both bailingmoe3 and its Q-LoRA path. The reference implementation is the branch behind PR #26608 (aetherbird/llama.cpp, branch bailingmoe3-support). The ROCmFP4 quant types additionally require a fork that implements them; upstream llama.cpp does not have Q4_0_ROCMFP4_*.

If your build loads Ling-3.0-flash but not tiny, you are missing the Q-LoRA path specifically.

⚠️ `strings` is not a capability check

We tested a second gfx1151 machine whose libllama.so contained bailingmoe3 (60 matches), ssm_f_a, and attn_q_a — it looked fully capable. It still failed with the exact same missing tensor 'blk.0.ssm_f.weight'.

Those symbols live in the tensor-name table. The fallback logic that maps ssm_f → ssm_f_a, and the Q-LoRA branch itself, are separate code. Grepping the binary tells you nothing — attempt the load.

All quant variants

Three builds of this model, all measured in one session on one box with one binary (Ryzen AI MAX+ 395, gfx1151, ROCm 7.2.4, ROCmFPX-2809dc5) — so these rows are directly comparable. Median of 3, warm-up discarded, otherwise-idle box.

variant	ftype	size	bpw	decode (median)	range	repo
4-bit COHERENT	102	4.30 GiB	4.67	104.04	104.00 – 104.24	Ling-3.0-tiny-ROCmFP4-GGUF
8-bit AGENT	115	7.72 GiB	8.40	88.82	88.80 – 88.83	Ling-3.0-tiny-ROCmFPX-Q8_0-AGENT-GGUF
8-bit plain	111	7.62 GiB	8.28	89.51	89.48 – 89.51	Ling-3.0-tiny-ROCmFPX-Q8_0-GGUF

⚠️ The 4-bit build is faster (104.04 vs ~89 tok/s) and 44% smaller. These 8-bit builds exist for accuracy headroom, not speed — pick them only if you need the extra precision.

What AGENT actually changes: it keeps far more tensors at true Q8_0 instead of the packed 8-bit type — measured in these files, 135 tensors vs 2 tensors. On models with an MTP draft head that raises draft acceptance and wins ~6%; these two models have no MTP head, and here the two 8-bit builds are within noise of each other.

Measured results

Verified on two independent gfx1151 machines, using the model's official sampling (temperature 0.6, top_p 0.95, top_k 20).

	machine A	machine B
SoC	Ryzen AI MAX+ 395 (gfx1151)	Ryzen AI MAX+ 395 (gfx1151)
memory	128 GB unified	125 GB unified
ROCm	7.2.4	7.13.0
flags	`-ngl 99 -c 4096 -fa on`	`-ngl 999 -c 32768 -fa on -fit off --no-mmap`
loads	✅ `arch=bailingmoe3`, 526 tensors	✅
17 × 23	✅ `391`	✅ `391`
capital of Japan	✅ `Tokyo`	✅ `Tokyo`
days in 2024	✅ `366`	✅ `366`
reasoning separation	✅ clean, in `reasoning_content`	✅
decode speed	97.64 tok/s	101.08 tok/s

⭐ The build is portable across ROCm minor versions

The binaries were compiled against ROCm 7.2.4 and run unmodified on a ROCm 7.13.0 host — all 9 Q4_0_ROCMFP4_* quant types still enumerated, model loads, 101 tok/s. Both hosts are gfx1151. Useful if you have one build box and several Strix Halo machines: you can copy llama-server + lib*.so* rather than rebuilding per host (set LD_LIBRARY_PATH to the directory you copied them into).

Serving configuration that works

Long-running deployment on machine B (systemd, always-hot):

-dev ROCm0 -ngl 999 -fa on -c 32768 -fit off -np 1 --no-mmap --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20

plus LimitMEMLOCK=infinity, HSA_OVERRIDE_GFX_VERSION=11.5.1, GGML_HIP_ENABLE_UNIFIED_MEMORY=1.

Verified on the running process, not just at launch: memlock unlimited (the 8 MB default will hobble the model), n_ctx = 32768 actually granted — --fit is on by default and can silently shrink context or push tensors to CPU, so -fit off and then confirm the number — and zero "tensor override to CPU" lines in the log.

Per-tensor protection (audited in the finished file)

tensor class	type
`output.weight` (LM head)	Q6_K
`token_embd.weight`	Q6_K
`*_shexp` shared experts (69)	Q8_0
`ffn_gate_inp` router (23)	F32
`ssm_a`, `ssm_dt.bias` (36)	F32
`ssm_conv1d_{q,k,v}` (54)	F32
norms (79)	F32
routed experts, attention projections	4-bit

Why this matters. Tier _STRIX (105) protects attention K/V but not the LM head — on this model's 157,184-token vocabulary that leaves every logit passing through a 4-bit tensor. Tier 102 COHERENT carries Q6_K token embeddings, and the head/shared-expert protections above were applied explicitly. Shared experts matter because they are dense — they process every token, so their error is systematic rather than averaged across the 128 routed experts.

The recurrent/linear-attention state (ssm_a, ssm_dt, conv1d) is kept at F32: these are float32 in the source model, and quantizing hybrid state is a known way to produce a model that loads, runs, and emits fluent nonsense.

Size comparison (same source, same machine)

build	size
BF16	14.72 GiB
Q4_K_M	4.4926 GiB
this build	4.2987 GiB

What was NOT measured

Stated plainly so you can judge fitness for your use case:

No perplexity run, and no quality A/B against Q4_K_M or BF16. The correctness checks above are memorized-fact prompts — they are necessary but not sufficient, and a damaged model can pass them.
No long-context testing. All generations were short. The 32,768-token context was granted and confirmed at load on machine B, but nothing exercised rope/KV behaviour at depth, and nothing was run near the model's 131,072 ceiling.
No tool-calling evaluation.
MTP / speculative decoding untested — Ling-3.0-tiny has num_nextn_predict_layers: 0, so it has no MTP layer to exercise.

Model

BailingMoeV3ForCausalLM / bailing_hybrid, GGUF arch bailingmoe3. 24 layers in a 3:1 stack of KDA (Kimi Delta Attention, 18 layers) and MLA (Multi-head Latent Attention, 6 layers) · hidden 1536 · 128 routed experts, 8 active · shared experts · vocab 157,184 · q_lora_rank 256 · kv_lora_rank 512 · context 131,072.

Base model licence: MIT (inherited). All credit for the model itself goes to inclusionAI.

kingjones777
/

Ling-3.0-tiny-ROCmFP4-GGUF

⚠️ STOCK `llama.cpp` WILL NOT LOAD THIS MODEL

✅ The patch you need is in this repo

Ling-3.0-tiny — ROCmFP4 (tier 102 `COHERENT`) GGUF

⛔ REQUIRES A PATCHED llama.cpp — STOCK WILL NOT LOAD THIS

⚠️ `strings` is not a capability check

All quant variants

Measured results

⭐ The build is portable across ROCm minor versions

Serving configuration that works

Per-tensor protection (audited in the finished file)

Size comparison (same source, same machine)

What was NOT measured

Model

⚠️ STOCK llama.cpp WILL NOT LOAD THIS MODEL

✅ The patch you need is in this repo

Ling-3.0-tiny — ROCmFP4 (tier 102 COHERENT) GGUF

⛔ REQUIRES A PATCHED llama.cpp — STOCK WILL NOT LOAD THIS

⚠️ strings is not a capability check

All quant variants

Measured results

⭐ The build is portable across ROCm minor versions

Serving configuration that works

Per-tensor protection (audited in the finished file)

Size comparison (same source, same machine)

What was NOT measured

Model

⚠️ STOCK `llama.cpp` WILL NOT LOAD THIS MODEL

Ling-3.0-tiny — ROCmFP4 (tier 102 `COHERENT`) GGUF

⚠️ `strings` is not a capability check