Instructions to use HelloSun/flint_cpu_opt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use HelloSun/flint_cpu_opt with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Use Docker

docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL

Llama new
LM Studio
Jan

vLLM

How to use HelloSun/flint_cpu_opt with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HelloSun/flint_cpu_opt"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HelloSun/flint_cpu_opt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL

Ollama
How to use HelloSun/flint_cpu_opt with Ollama:
```
ollama run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
```
Unsloth Desktop

How to use HelloSun/flint_cpu_opt with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "HelloSun/flint_cpu_opt:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use HelloSun/flint_cpu_opt with Docker Model Runner:
```
docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
```

Lemonade

How to use HelloSun/flint_cpu_opt with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull HelloSun/flint_cpu_opt:UD-Q4_K_XL

Run and chat with the model

lemonade run user.flint_cpu_opt-UD-Q4_K_XL

List all available models

lemonade list

Hermes Agent

How to use HelloSun/flint_cpu_opt with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default HelloSun/flint_cpu_opt:UD-Q4_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use HelloSun/flint_cpu_opt with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "HelloSun/flint_cpu_opt:UD-Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

flint_cpu_opt / benchmarks /profile-decode.txt

HelloSun

flint CPU opt: IQ4_XS AVX512-VNNI vec_dot, Q5_K mask merge, rebased on de5122d; re-measured benchmarks and KLD

d35f8cb verified 9 days ago

Raw History Blame Contribute Delete

5.82 kB

| model                          |       size |     params | backend    | threads |  fa |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --: | ---------: | --------------: | -------------------: |
| flint ?B Q4_K - Medium         |  23.33 GiB |    39.79 B | CPU        |      16 |   1 |      mlock |            tg64 |         37.72 ± 0.00 |

[GGML_PROF] total_us=1678629
[GGML_PROF] GET_ROWS             f32   <q8_0   [  2048,    1,   1]      0.45 ms   0.03% calls=65 us/call=6.9
[GGML_PROF] RMS_NORM             f32   <f32    [  2048,    1,   1]     18.00 ms   1.07% calls=6435 us/call=2.8
[GGML_PROF] MUL_MAT              f32   <q8_0   [  4096,    1,   1]    171.92 ms  10.24% calls=2730 us/call=63.0
[GGML_PROF] RMS_NORM             f32   <f32    [   128,   32,   1]      6.32 ms   0.38% calls=3185 us/call=2.0
[GGML_PROF] MUL_MAT              f32   <q8_0   [   512,    1,   1]     62.51 ms   3.72% calls=5460 us/call=11.4
[GGML_PROF] RMS_NORM             f32   <f32    [   128,    4,   1]      7.42 ms   0.44% calls=3185 us/call=2.3
[GGML_PROF] SET_ROWS             f16   <f32    [   512,  256,   1]      7.33 ms   0.44% calls=6370 us/call=1.2
[GGML_PROF] FLASH_ATTN_EXT       f32   <f32    [   128,   32,   1]     29.72 ms   1.77% calls=3185 us/call=9.3
[GGML_PROF] MUL_MAT              f32   <q8_0   [  2048,    1,   1]    218.00 ms  12.99% calls=5070 us/call=43.0
[GGML_PROF] ADD                  f32   <f32    [  2048,    1,   1]     34.85 ms   2.08% calls=24700 us/call=1.4
[GGML_PROF] MUL_MAT              f32   <q8_0   [  6144,    1,   1]     22.79 ms   1.36% calls=260 us/call=87.6
[GGML_PROF] GLU                  f32   <f32    [  6144,    1,   1]      0.63 ms   0.04% calls=130 us/call=4.8
[GGML_PROF] ROPE                 f32   <f32    [   128,   32,   1]      5.86 ms   0.35% calls=2275 us/call=2.6
[GGML_PROF] ROPE                 f32   <f32    [   128,    4,   1]      2.98 ms   0.18% calls=2275 us/call=1.3
[GGML_PROF] MUL_MAT              f32   <bf16   [   256,    1,   1]     43.35 ms   2.58% calls=3055 us/call=14.2
[GGML_PROF] UNARY                f32   <f32    [   256,    1,   1]      4.08 ms   0.24% calls=3055 us/call=1.3
[GGML_PROF] ARGSORT              i32   <f32    [   256,    1,   1]     36.86 ms   2.20% calls=3055 us/call=12.1
[GGML_PROF] GET_ROWS             f32   <f32    [     1,    6,   1]      4.56 ms   0.27% calls=3055 us/call=1.5
[GGML_PROF] SUM_ROWS             f32   <f32    [     1,    1,   1]      3.27 ms   0.19% calls=3055 us/call=1.1
[GGML_PROF] CLAMP                f32   <f32    [     1,    1,   1]      3.43 ms   0.20% calls=3055 us/call=1.1
[GGML_PROF] DIV                  f32   <f32    [     6,    1,   1]      3.41 ms   0.20% calls=3055 us/call=1.1
[GGML_PROF] MUL_MAT_ID           f32   <iq3_s  [   512,    6,   1]     18.62 ms   1.11% calls=130 us/call=143.2
[GGML_PROF] GLU                  f32   <f32    [   512,    6,   1]      4.30 ms   0.26% calls=3055 us/call=1.4
[GGML_PROF] UNARY                f32   <f32    [   512,    6,   1]      3.62 ms   0.22% calls=3055 us/call=1.2
[GGML_PROF] POOL_1D              f32   <f32    [     1,    6,   1]     13.12 ms   0.78% calls=3055 us/call=4.3
[GGML_PROF] SCALE                f32   <f32    [     1,    6,   1]     10.41 ms   0.62% calls=9165 us/call=1.1
[GGML_PROF] LOG                  f32   <f32    [     1,    6,   1]      3.64 ms   0.22% calls=3055 us/call=1.2
[GGML_PROF] UNARY                f32   <f32    [     1,    6,   1]      7.11 ms   0.42% calls=6110 us/call=1.2
[GGML_PROF] CLAMP                f32   <f32    [     1,    6,   1]      3.29 ms   0.20% calls=3055 us/call=1.1
[GGML_PROF] DIV                  f32   <f32    [   512,    6,   1]      5.20 ms   0.31% calls=3055 us/call=1.7
[GGML_PROF] MUL_MAT_ID           f32   <q5_K   [  2048,    6,   1]    141.87 ms   8.45% calls=1885 us/call=75.3
[GGML_PROF] MUL                  f32   <f32    [  2048,    6,   1]     30.55 ms   1.82% calls=6110 us/call=5.0
[GGML_PROF] MUL_MAT              f32   <q8_0   [  1024,    1,   1]     82.51 ms   4.92% calls=4420 us/call=18.7
[GGML_PROF] GLU                  f32   <f32    [  1024,    1,   1]      3.47 ms   0.21% calls=3055 us/call=1.1
[GGML_PROF] MUL_MAT_ID           f32   <q5_K   [   512,    6,   1]    220.48 ms  13.13% calls=2340 us/call=94.2
[GGML_PROF] MUL_MAT_ID           f32   <q8_0   [  2048,    6,   1]     12.57 ms   0.75% calls=65 us/call=193.4
[GGML_PROF] MUL_MAT_ID           f32   <q6_K   [  2048,    6,   1]     10.04 ms   0.60% calls=65 us/call=154.5
[GGML_PROF] MUL_MAT              f32   <q5_K   [  4096,    1,   1]     33.90 ms   2.02% calls=455 us/call=74.5
[GGML_PROF] MUL_MAT              f32   <q5_K   [   512,    1,   1]     18.08 ms   1.08% calls=910 us/call=19.9
[GGML_PROF] MUL_MAT              f32   <q5_K   [  2048,    1,   1]     52.65 ms   3.14% calls=1235 us/call=42.6
[GGML_PROF] MUL_MAT_ID           f32   <iq4_xs [   512,    6,   1]    102.81 ms   6.12% calls=3250 us/call=31.6
[GGML_PROF] MUL_MAT_ID           f32   <iq4_xs [  2048,    6,   1]     35.32 ms   2.10% calls=1040 us/call=34.0
[GGML_PROF] MUL_MAT_ID           f32   <q4_K   [   512,    6,   1]     12.47 ms   0.74% calls=390 us/call=32.0
[GGML_PROF] MUL_MAT              f32   <q5_K   [  1024,    1,   1]     36.01 ms   2.14% calls=1560 us/call=23.1
[GGML_PROF] MUL_MAT              f32   <iq4_xs [  1024,    1,   1]      2.38 ms   0.14% calls=130 us/call=18.3
[GGML_PROF] MUL_MAT              f32   <iq4_xs [  2048,    1,   1]      1.02 ms   0.06% calls=65 us/call=15.7
[GGML_PROF] GET_ROWS             f32   <f32    [  2048,    1,   1]      0.19 ms   0.01% calls=130 us/call=1.5
[GGML_PROF] MUL_MAT              f32   <q8_0   [150144,    1,   1]    125.28 ms   7.46% calls=65 us/call=1927.3

build: de5122d87 (1)

	\| model \| size \| params \| backend \| threads \| fa \| lm \| test \| t/s \|
	\| ------------------------------ \| ---------: \| ---------: \| ---------- \| ------: \| --: \| ---------: \| --------------: \| -------------------: \|
	\| flint ?B Q4_K - Medium \| 23.33 GiB \| 39.79 B \| CPU \| 16 \| 1 \| mlock \| tg64 \| 37.72 ± 0.00 \|

	[GGML_PROF] total_us=1678629
	[GGML_PROF] GET_ROWS f32 <q8_0 [ 2048, 1, 1] 0.45 ms 0.03% calls=65 us/call=6.9
	[GGML_PROF] RMS_NORM f32 <f32 [ 2048, 1, 1] 18.00 ms 1.07% calls=6435 us/call=2.8
	[GGML_PROF] MUL_MAT f32 <q8_0 [ 4096, 1, 1] 171.92 ms 10.24% calls=2730 us/call=63.0
	[GGML_PROF] RMS_NORM f32 <f32 [ 128, 32, 1] 6.32 ms 0.38% calls=3185 us/call=2.0
	[GGML_PROF] MUL_MAT f32 <q8_0 [ 512, 1, 1] 62.51 ms 3.72% calls=5460 us/call=11.4
	[GGML_PROF] RMS_NORM f32 <f32 [ 128, 4, 1] 7.42 ms 0.44% calls=3185 us/call=2.3
	[GGML_PROF] SET_ROWS f16 <f32 [ 512, 256, 1] 7.33 ms 0.44% calls=6370 us/call=1.2
	[GGML_PROF] FLASH_ATTN_EXT f32 <f32 [ 128, 32, 1] 29.72 ms 1.77% calls=3185 us/call=9.3
	[GGML_PROF] MUL_MAT f32 <q8_0 [ 2048, 1, 1] 218.00 ms 12.99% calls=5070 us/call=43.0
	[GGML_PROF] ADD f32 <f32 [ 2048, 1, 1] 34.85 ms 2.08% calls=24700 us/call=1.4
	[GGML_PROF] MUL_MAT f32 <q8_0 [ 6144, 1, 1] 22.79 ms 1.36% calls=260 us/call=87.6
	[GGML_PROF] GLU f32 <f32 [ 6144, 1, 1] 0.63 ms 0.04% calls=130 us/call=4.8
	[GGML_PROF] ROPE f32 <f32 [ 128, 32, 1] 5.86 ms 0.35% calls=2275 us/call=2.6
	[GGML_PROF] ROPE f32 <f32 [ 128, 4, 1] 2.98 ms 0.18% calls=2275 us/call=1.3
	[GGML_PROF] MUL_MAT f32 <bf16 [ 256, 1, 1] 43.35 ms 2.58% calls=3055 us/call=14.2
	[GGML_PROF] UNARY f32 <f32 [ 256, 1, 1] 4.08 ms 0.24% calls=3055 us/call=1.3
	[GGML_PROF] ARGSORT i32 <f32 [ 256, 1, 1] 36.86 ms 2.20% calls=3055 us/call=12.1
	[GGML_PROF] GET_ROWS f32 <f32 [ 1, 6, 1] 4.56 ms 0.27% calls=3055 us/call=1.5
	[GGML_PROF] SUM_ROWS f32 <f32 [ 1, 1, 1] 3.27 ms 0.19% calls=3055 us/call=1.1
	[GGML_PROF] CLAMP f32 <f32 [ 1, 1, 1] 3.43 ms 0.20% calls=3055 us/call=1.1
	[GGML_PROF] DIV f32 <f32 [ 6, 1, 1] 3.41 ms 0.20% calls=3055 us/call=1.1
	[GGML_PROF] MUL_MAT_ID f32 <iq3_s [ 512, 6, 1] 18.62 ms 1.11% calls=130 us/call=143.2
	[GGML_PROF] GLU f32 <f32 [ 512, 6, 1] 4.30 ms 0.26% calls=3055 us/call=1.4
	[GGML_PROF] UNARY f32 <f32 [ 512, 6, 1] 3.62 ms 0.22% calls=3055 us/call=1.2
	[GGML_PROF] POOL_1D f32 <f32 [ 1, 6, 1] 13.12 ms 0.78% calls=3055 us/call=4.3
	[GGML_PROF] SCALE f32 <f32 [ 1, 6, 1] 10.41 ms 0.62% calls=9165 us/call=1.1
	[GGML_PROF] LOG f32 <f32 [ 1, 6, 1] 3.64 ms 0.22% calls=3055 us/call=1.2
	[GGML_PROF] UNARY f32 <f32 [ 1, 6, 1] 7.11 ms 0.42% calls=6110 us/call=1.2
	[GGML_PROF] CLAMP f32 <f32 [ 1, 6, 1] 3.29 ms 0.20% calls=3055 us/call=1.1
	[GGML_PROF] DIV f32 <f32 [ 512, 6, 1] 5.20 ms 0.31% calls=3055 us/call=1.7
	[GGML_PROF] MUL_MAT_ID f32 <q5_K [ 2048, 6, 1] 141.87 ms 8.45% calls=1885 us/call=75.3
	[GGML_PROF] MUL f32 <f32 [ 2048, 6, 1] 30.55 ms 1.82% calls=6110 us/call=5.0
	[GGML_PROF] MUL_MAT f32 <q8_0 [ 1024, 1, 1] 82.51 ms 4.92% calls=4420 us/call=18.7
	[GGML_PROF] GLU f32 <f32 [ 1024, 1, 1] 3.47 ms 0.21% calls=3055 us/call=1.1
	[GGML_PROF] MUL_MAT_ID f32 <q5_K [ 512, 6, 1] 220.48 ms 13.13% calls=2340 us/call=94.2
	[GGML_PROF] MUL_MAT_ID f32 <q8_0 [ 2048, 6, 1] 12.57 ms 0.75% calls=65 us/call=193.4
	[GGML_PROF] MUL_MAT_ID f32 <q6_K [ 2048, 6, 1] 10.04 ms 0.60% calls=65 us/call=154.5
	[GGML_PROF] MUL_MAT f32 <q5_K [ 4096, 1, 1] 33.90 ms 2.02% calls=455 us/call=74.5
	[GGML_PROF] MUL_MAT f32 <q5_K [ 512, 1, 1] 18.08 ms 1.08% calls=910 us/call=19.9
	[GGML_PROF] MUL_MAT f32 <q5_K [ 2048, 1, 1] 52.65 ms 3.14% calls=1235 us/call=42.6
	[GGML_PROF] MUL_MAT_ID f32 <iq4_xs [ 512, 6, 1] 102.81 ms 6.12% calls=3250 us/call=31.6
	[GGML_PROF] MUL_MAT_ID f32 <iq4_xs [ 2048, 6, 1] 35.32 ms 2.10% calls=1040 us/call=34.0
	[GGML_PROF] MUL_MAT_ID f32 <q4_K [ 512, 6, 1] 12.47 ms 0.74% calls=390 us/call=32.0
	[GGML_PROF] MUL_MAT f32 <q5_K [ 1024, 1, 1] 36.01 ms 2.14% calls=1560 us/call=23.1
	[GGML_PROF] MUL_MAT f32 <iq4_xs [ 1024, 1, 1] 2.38 ms 0.14% calls=130 us/call=18.3
	[GGML_PROF] MUL_MAT f32 <iq4_xs [ 2048, 1, 1] 1.02 ms 0.06% calls=65 us/call=15.7
	[GGML_PROF] GET_ROWS f32 <f32 [ 2048, 1, 1] 0.19 ms 0.01% calls=130 us/call=1.5
	[GGML_PROF] MUL_MAT f32 <q8_0 [150144, 1, 1] 125.28 ms 7.46% calls=65 us/call=1927.3

	build: de5122d87 (1)