Instructions to use HelloSun/flint_cpu_opt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use HelloSun/flint_cpu_opt with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Use Docker

docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL

Llama new
LM Studio
Jan

vLLM

How to use HelloSun/flint_cpu_opt with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HelloSun/flint_cpu_opt"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HelloSun/flint_cpu_opt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL

Ollama
How to use HelloSun/flint_cpu_opt with Ollama:
```
ollama run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
```
Unsloth Desktop

How to use HelloSun/flint_cpu_opt with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "HelloSun/flint_cpu_opt:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use HelloSun/flint_cpu_opt with Docker Model Runner:
```
docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
```

Lemonade

How to use HelloSun/flint_cpu_opt with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull HelloSun/flint_cpu_opt:UD-Q4_K_XL

Run and chat with the model

lemonade run user.flint_cpu_opt-UD-Q4_K_XL

List all available models

lemonade list

Hermes Agent

How to use HelloSun/flint_cpu_opt with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default HelloSun/flint_cpu_opt:UD-Q4_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use HelloSun/flint_cpu_opt with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "HelloSun/flint_cpu_opt:UD-Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

flint_cpu_opt / benchmarks /summary.md

HelloSun

KLD: add stock-vs-stock control showing flint noise floor; add graph accuracy test

7163b59 verified 10 days ago

preview code

Raw History Blame Contribute Delete

3.29 kB

# flint-UD-Q4_K_XL CPU optimization (Xeon Platinum 8559C, 16 threads)

`llama-bench -t 16 -fa on -lm mlock -b 1024 -ub 1024`, 3 interleaved rounds (stock / Qwen3.6 patch / this patch per round), median t/s.
Raw: `ab-interleaved-flint.csv`.

| variant | pp256 | pp2048 | tg128 |
|---|---|---|---|
| stock llama.cpp (de5122d) | 231.0 | 274.2 | 38.2 |
| Qwen3.6 CPU-OPT patch, unchanged | 264.4 | 307.8 | 42.1 |
| **this patch** | **295.8** | **306.7** | **42.8** |

Versus stock: pp256 1.28x, pp2048 1.12x, tg128 1.12x. Versus the Qwen patch alone, only pp256 (+12%) is clearly better;
decode (+2%) and pp2048 are within noise. Most of the gain comes from the Qwen3.6 patch's kernels also applying to flint.

## What is new (flint-specific)
flint-UD-Q4_K_XL is Q5_K 11.6 GiB + IQ4_XS 9.0 GiB + Q8_0 2.0 GiB (+ small Q4_K/Q6_K/IQ3_S).
1. IQ4_XS x Q8_K `vec_dot`: AVX512-VNNI (the x86 path had only AVX2). Unsigned LUT (+128) with `vpdpbusd`, bias removed
   through `bsums`, L2 prefetch (`GGML_IQ4XS_PF`, default 40). Decode MUL_MAT_ID iq4_xs reaches ~97 GB/s.
2. Q5_K x Q8_K `vec_dot`: 5th bit merged with `vptestmb` + masked add instead of shift/and/or.
   Bit-identical results to the previous VNNI version (KLD unchanged on Qwen3.6 as well).
Other contents are the Qwen3.6 patch (Q4_K/Q5_K/Q6_K VNNI, AMX tweaks, tiled MoE group 32). Its Gated-DeltaNet changes
do not apply to flint.
## Accuracy (wikitext-2 test, 32 x 512, vs stock logits)
| | KLD | same top-1 | PPL |
|---|---|---|---|
| stock vs stock (-t 16 vs -t 8) | 0.0000 | 100 % | 18.363 |
| Qwen3.6 patch unchanged | 0.0664 +- 0.0049 | 92.2 % | - |
| this patch | 0.0666 +- 0.0049 | ~92 % | **18.277** +- 0.787 |
PPL is not worse than stock (18.363 +- 0.795).
KLD is much larger than on Qwen3.6 (0.003-0.006), but **flint is numerically chaotic**: stock llama.cpp compared with itself
under a different numeric path gives the same level of KLD (control experiments, same baseline logits):
| stock, changed numeric path | KLD | same top-1 |
|---|---|---|
| `-fa off` | 0.0762 +- 0.0055 | 91.7 % |
| `-ub 128 -b 128` | 0.0700 +- 0.0051 | 92.0 % |
| this patch | 0.0666 +- 0.0049 | ~92 % |
So 0.067 is inside the noise floor of stock itself and does not indicate a defect in the kernels. Supporting evidence:
a graph-level test (`mul_mat` and `mul_mat_id`, K=2048, N=512, random weights) gives the same relative L2 error versus a float
reference for stock and this patch to 4 digits (Q4_K/Q5_K/Q6_K/IQ4_XS ~7e-3, Q8_0 ~5.4e-3, dominated by Q8_K activation quantization).
Bisect (`kld-bisect.log`, KLD with one VNNI kernel disabled): Q4_K 0.0670, Q5_K 0.0524, Q6_K 0.0685, IQ4_XS 0.0664;
removing all of `quants.c` gives 0.0162. Why tiny differences get amplified (e.g. top-6-of-256 expert routing ties) was not verified.
The earlier flint_cpu_opt README reported KLD 0.2058, which was the same effect measured on a different baseline.

## Profile (decode, 64 tokens)
MUL_MAT q8_0 41 % (attention, shared expert, lm_head; close to bandwidth bound), MUL_MAT_ID q5_K 22 %,
MUL_MAT q5_K 8 %, MUL_MAT_ID iq4_xs 8 %. Prefill: FLASH_ATTN_EXT 24 %, MUL_MAT_ID q5_K 21 %, q8_0 17 %, iq4_xs 15 %.
Prefetch distance, thread count and `--poll` sweeps showed no gain beyond noise.
Not done: flash-attention kernel work, Q8_0 decode kernel.

	# flint-UD-Q4_K_XL CPU optimization (Xeon Platinum 8559C, 16 threads)

	`llama-bench -t 16 -fa on -lm mlock -b 1024 -ub 1024`, 3 interleaved rounds (stock / Qwen3.6 patch / this patch per round), median t/s.
	Raw: `ab-interleaved-flint.csv`.

	\| variant \| pp256 \| pp2048 \| tg128 \|
	\|---\|---\|---\|---\|
	\| stock llama.cpp (de5122d) \| 231.0 \| 274.2 \| 38.2 \|
	\| Qwen3.6 CPU-OPT patch, unchanged \| 264.4 \| 307.8 \| 42.1 \|
	\| this patch \| 295.8 \| 306.7 \| 42.8 \|

	Versus stock: pp256 1.28x, pp2048 1.12x, tg128 1.12x. Versus the Qwen patch alone, only pp256 (+12%) is clearly better;
	decode (+2%) and pp2048 are within noise. Most of the gain comes from the Qwen3.6 patch's kernels also applying to flint.

	## What is new (flint-specific)
	flint-UD-Q4_K_XL is Q5_K 11.6 GiB + IQ4_XS 9.0 GiB + Q8_0 2.0 GiB (+ small Q4_K/Q6_K/IQ3_S).
	1. IQ4_XS x Q8_K `vec_dot`: AVX512-VNNI (the x86 path had only AVX2). Unsigned LUT (+128) with `vpdpbusd`, bias removed
	through `bsums`, L2 prefetch (`GGML_IQ4XS_PF`, default 40). Decode MUL_MAT_ID iq4_xs reaches ~97 GB/s.
	2. Q5_K x Q8_K `vec_dot`: 5th bit merged with `vptestmb` + masked add instead of shift/and/or.
	Bit-identical results to the previous VNNI version (KLD unchanged on Qwen3.6 as well).
	Other contents are the Qwen3.6 patch (Q4_K/Q5_K/Q6_K VNNI, AMX tweaks, tiled MoE group 32). Its Gated-DeltaNet changes
	do not apply to flint.

	## Accuracy (wikitext-2 test, 32 x 512, vs stock logits)
	\| \| KLD \| same top-1 \| PPL \|
	\|---\|---\|---\|---\|
	\| stock vs stock (-t 16 vs -t 8) \| 0.0000 \| 100 % \| 18.363 \|
	\| Qwen3.6 patch unchanged \| 0.0664 +- 0.0049 \| 92.2 % \| - \|
	\| this patch \| 0.0666 +- 0.0049 \| ~92 % \| 18.277 +- 0.787 \|

	PPL is not worse than stock (18.363 +- 0.795).

	KLD is much larger than on Qwen3.6 (0.003-0.006), but flint is numerically chaotic: stock llama.cpp compared with itself
	under a different numeric path gives the same level of KLD (control experiments, same baseline logits):

	\| stock, changed numeric path \| KLD \| same top-1 \|
	\|---\|---\|---\|
	\| `-fa off` \| 0.0762 +- 0.0055 \| 91.7 % \|
	\| `-ub 128 -b 128` \| 0.0700 +- 0.0051 \| 92.0 % \|
	\| this patch \| 0.0666 +- 0.0049 \| ~92 % \|

	So 0.067 is inside the noise floor of stock itself and does not indicate a defect in the kernels. Supporting evidence:
	a graph-level test (`mul_mat` and `mul_mat_id`, K=2048, N=512, random weights) gives the same relative L2 error versus a float
	reference for stock and this patch to 4 digits (Q4_K/Q5_K/Q6_K/IQ4_XS ~7e-3, Q8_0 ~5.4e-3, dominated by Q8_K activation quantization).
	Bisect (`kld-bisect.log`, KLD with one VNNI kernel disabled): Q4_K 0.0670, Q5_K 0.0524, Q6_K 0.0685, IQ4_XS 0.0664;
	removing all of `quants.c` gives 0.0162. Why tiny differences get amplified (e.g. top-6-of-256 expert routing ties) was not verified.
	The earlier flint_cpu_opt README reported KLD 0.2058, which was the same effect measured on a different baseline.

	## Profile (decode, 64 tokens)
	MUL_MAT q8_0 41 % (attention, shared expert, lm_head; close to bandwidth bound), MUL_MAT_ID q5_K 22 %,
	MUL_MAT q5_K 8 %, MUL_MAT_ID iq4_xs 8 %. Prefill: FLASH_ATTN_EXT 24 %, MUL_MAT_ID q5_K 21 %, q8_0 17 %, iq4_xs 15 %.
	Prefetch distance, thread count and `--poll` sweeps showed no gain beyond noise.
	Not done: flash-attention kernel work, Q8_0 decode kernel.