Instructions to use HelloSun/flint_cpu_opt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use HelloSun/flint_cpu_opt with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Use Docker

docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL

Llama new
LM Studio
Jan

vLLM

How to use HelloSun/flint_cpu_opt with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HelloSun/flint_cpu_opt"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HelloSun/flint_cpu_opt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL

Ollama
How to use HelloSun/flint_cpu_opt with Ollama:
```
ollama run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
```
Unsloth Desktop

How to use HelloSun/flint_cpu_opt with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "HelloSun/flint_cpu_opt:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use HelloSun/flint_cpu_opt with Docker Model Runner:
```
docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
```

Lemonade

How to use HelloSun/flint_cpu_opt with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull HelloSun/flint_cpu_opt:UD-Q4_K_XL

Run and chat with the model

lemonade run user.flint_cpu_opt-UD-Q4_K_XL

List all available models

lemonade list

Hermes Agent

How to use HelloSun/flint_cpu_opt with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default HelloSun/flint_cpu_opt:UD-Q4_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use HelloSun/flint_cpu_opt with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "HelloSun/flint_cpu_opt:UD-Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

flint_cpu_opt / benchmarks /summary.md

HelloSun

KLD: add stock-vs-stock control showing flint noise floor; add graph accuracy test

7163b59 verified 9 days ago

preview code

Raw History Blame Contribute Delete

3.29 kB

flint-UD-Q4_K_XL CPU optimization (Xeon Platinum 8559C, 16 threads)

llama-bench -t 16 -fa on -lm mlock -b 1024 -ub 1024, 3 interleaved rounds (stock / Qwen3.6 patch / this patch per round), median t/s. Raw: ab-interleaved-flint.csv.

variant	pp256	pp2048	tg128
stock llama.cpp (de5122d)	231.0	274.2	38.2
Qwen3.6 CPU-OPT patch, unchanged	264.4	307.8	42.1
this patch	295.8	306.7	42.8

Versus stock: pp256 1.28x, pp2048 1.12x, tg128 1.12x. Versus the Qwen patch alone, only pp256 (+12%) is clearly better; decode (+2%) and pp2048 are within noise. Most of the gain comes from the Qwen3.6 patch's kernels also applying to flint.

What is new (flint-specific)

flint-UD-Q4_K_XL is Q5_K 11.6 GiB + IQ4_XS 9.0 GiB + Q8_0 2.0 GiB (+ small Q4_K/Q6_K/IQ3_S).

IQ4_XS x Q8_K vec_dot: AVX512-VNNI (the x86 path had only AVX2). Unsigned LUT (+128) with vpdpbusd, bias removed through bsums, L2 prefetch (GGML_IQ4XS_PF, default 40). Decode MUL_MAT_ID iq4_xs reaches ~97 GB/s.
Q5_K x Q8_K vec_dot: 5th bit merged with vptestmb + masked add instead of shift/and/or. Bit-identical results to the previous VNNI version (KLD unchanged on Qwen3.6 as well). Other contents are the Qwen3.6 patch (Q4_K/Q5_K/Q6_K VNNI, AMX tweaks, tiled MoE group 32). Its Gated-DeltaNet changes do not apply to flint.

Accuracy (wikitext-2 test, 32 x 512, vs stock logits)

	KLD	same top-1	PPL
stock vs stock (-t 16 vs -t 8)	0.0000	100 %	18.363
Qwen3.6 patch unchanged	0.0664 +- 0.0049	92.2 %	-
this patch	0.0666 +- 0.0049	~92 %	18.277 +- 0.787

PPL is not worse than stock (18.363 +- 0.795).

KLD is much larger than on Qwen3.6 (0.003-0.006), but flint is numerically chaotic: stock llama.cpp compared with itself under a different numeric path gives the same level of KLD (control experiments, same baseline logits):

stock, changed numeric path	KLD	same top-1
`-fa off`	0.0762 +- 0.0055	91.7 %
`-ub 128 -b 128`	0.0700 +- 0.0051	92.0 %
this patch	0.0666 +- 0.0049	~92 %

So 0.067 is inside the noise floor of stock itself and does not indicate a defect in the kernels. Supporting evidence: a graph-level test (mul_mat and mul_mat_id, K=2048, N=512, random weights) gives the same relative L2 error versus a float reference for stock and this patch to 4 digits (Q4_K/Q5_K/Q6_K/IQ4_XS ~7e-3, Q8_0 ~5.4e-3, dominated by Q8_K activation quantization). Bisect (kld-bisect.log, KLD with one VNNI kernel disabled): Q4_K 0.0670, Q5_K 0.0524, Q6_K 0.0685, IQ4_XS 0.0664; removing all of quants.c gives 0.0162. Why tiny differences get amplified (e.g. top-6-of-256 expert routing ties) was not verified. The earlier flint_cpu_opt README reported KLD 0.2058, which was the same effect measured on a different baseline.

Profile (decode, 64 tokens)

MUL_MAT q8_0 41 % (attention, shared expert, lm_head; close to bandwidth bound), MUL_MAT_ID q5_K 22 %, MUL_MAT q5_K 8 %, MUL_MAT_ID iq4_xs 8 %. Prefill: FLASH_ATTN_EXT 24 %, MUL_MAT_ID q5_K 21 %, q8_0 17 %, iq4_xs 15 %. Prefetch distance, thread count and --poll sweeps showed no gain beyond noise. Not done: flash-attention kernel work, Q8_0 decode kernel.