Instructions to use HelloSun/flint_cpu_opt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use HelloSun/flint_cpu_opt with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Use Docker

docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL

Llama new
LM Studio
Jan

vLLM

How to use HelloSun/flint_cpu_opt with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HelloSun/flint_cpu_opt"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HelloSun/flint_cpu_opt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL

Ollama
How to use HelloSun/flint_cpu_opt with Ollama:
```
ollama run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
```
Unsloth Desktop

How to use HelloSun/flint_cpu_opt with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "HelloSun/flint_cpu_opt:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use HelloSun/flint_cpu_opt with Docker Model Runner:
```
docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
```

Lemonade

How to use HelloSun/flint_cpu_opt with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull HelloSun/flint_cpu_opt:UD-Q4_K_XL

Run and chat with the model

lemonade run user.flint_cpu_opt-UD-Q4_K_XL

List all available models

lemonade list

Hermes Agent

How to use HelloSun/flint_cpu_opt with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default HelloSun/flint_cpu_opt:UD-Q4_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use HelloSun/flint_cpu_opt with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "HelloSun/flint_cpu_opt:UD-Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

flint_cpu_opt / benchmarks /nvfp4 /README.md

HelloSun

Add NVFP4 CPU kernels results, integrate model card, add handoff

45f1d0b verified 4 days ago

preview code

Raw History Blame Contribute Delete

3.83 kB

RecursiveSI/flint-nvfp4-gguf (flint-nvfp4.gguf, 22.9 GB) with the CPU patch

NVFP4 weights (485 tensors, 20.7 GiB) plus F32 per-tensor .scale tensors and BF16 token_embd. Only the NVFP4 file was measured here; the Q4_1 and Q4_K_XL numbers in README.md / q41-summary.md were not re-run.

What was added to the patch (`patches/flint-cpu-opt.patch`)

AVX512-VNNI vec_dot for NVFP4 x Q8_0 (decode, ggml/src/ggml-cpu/arch/x86/quants.c). Weight codes are |w| (0..12, unsigned operand of vpdpbusd), the sign of w is moved onto the int8 activations (_mm256_sign_epi8). The 64-element block is split into 4 subblocks of 16 with per-subblock UE4M3 scales.
Tiled VNNI GEMM for NVFP4 (prefill, ggml/src/ggml-cpu/tiled/). Same framework as the Q4_1 path. Weights are stored as LUT + 12 (unsigned, 0..24) with one f32 scale per 16-wide subblock (sd16). The bias is removed exactly with the per-16 activation code sums (bsums): sum(w*a) = sum((w+12)*a) - 12*sum(a). Activations use Q8_0 with one scale per 32-block. tiled_unpk_fp4_64 is an AVX2 unpacker (scalar fallback). Enabled only in VNNI builds (__AVX512VNNI__ && __AVX512VL__).

Integer dot products are exact; only the float accumulation order changes.

Results

Xeon Platinum 8559C, taskset -c 0-15, -t 16 -fa 1 -b 1024 -ub 1024 -lm mmap+mlock -p 256,2048 -n 128 -r 3. Three interleaved rounds per build, median of the per-round means (tokens/s). Raw CSV: bin_*_r{1,2,3}.csv, flat list: summary.csv.

build	pp256	pp2048	tg128
before: stock llama.cpp (de5122d87), no patch	83.7	83.6	30.6
patch (Q4_1 kernels), NVFP4 path unchanged	82.3	84.4	31.8
after: patch + NVFP4 VNNI vec_dot + tiled VNNI GEMM	178.5	211.1	32.1

gain	pp256	pp2048	tg128
after vs stock	2.13x	2.52x	1.05x (noise)
after vs patch-before	2.17x	2.50x	1.01x (noise)

Per-round values: pp256 after = 153.1 / 178.5 / 181.6, pp2048 after = 206.7 / 218.6 / 211.1, tg128 after = 32.1 / 29.8 / 32.4.

Decode (tg128) did not change measurably. The VNNI vec_dot was about 1 % faster in an earlier single-run check, which is within noise. Per-op profiling (GGML_PROF) showed that prefill MUL_MAT/MUL_MAT_ID with 256 rows dominated the run time.

Correctness

vec_dot AVX512 vs generic reference (random blocks, n = 1920..12352): relative error <= 1.3e-6 (float ordering only).
Dense mul_mat, tiled (M=64) vs vec_dot (M=1): relative L2 1.4e-7 .. 2.5e-7 for K in {1024, 2048, 4096, 8192} and N in {8, 16, 24, 100, 256, 512} (covers ragged row counts and the narrow path). Versus f32 reference: about 3.7e-3 (same order as Q4_1 in the HANDOFF).
wikitext-2 test, 16 chunks x 512, llama-perplexity:

build	PPL
stock	24.4165 +- 1.566
patch + NVFP4 opt	24.4135 +- 1.568

Notes

The host was shared and heavily loaded during these runs (load average 55..63). Single runs ranged widely (stock tg128: 14.1 / 32.3 / 30.6), so use the medians. Absolute numbers are lower than the Q4_1 reference figures in q41-summary.md, which were measured on a quieter host. Compare only within this table.
After the patch, NVFP4 prefill (pp2048 211) is still below Q4_1 (pp2048 297 in q41-summary.md). Decode is about 32 t/s versus 55.7 t/s for Q4_1 on the reference host, which is not a fair comparison because the hosts differ.
Reproduce: bench_run.sh runs three copies of llama-bench (stock, patch before, patch after) from the directories set in LD_LIBRARY_PATH/paths in the script. Those copies live under /tmp/opencode and are not kept. Rebuild them from de5122d87 (stock) and from build.sh (patched). test_nvfp4.c and tiled_test.c are the kernel checks above.

RecursiveSI/flint-nvfp4-gguf (flint-nvfp4.gguf, 22.9 GB) with the CPU patch

What was added to the patch (patches/flint-cpu-opt.patch)

Results

Correctness

Notes

What was added to the patch (`patches/flint-cpu-opt.patch`)