Add auto arena mode (80% of available memory, capped at all experts); patch 369ef971

2873e19 verified 14 days ago

3.84 kB

---
library_name: llama.cpp
license: mit
base_model: RecursiveSI/flint
tags:
- flint
- llama.cpp
- moe
- expert-offloading
- cpu-inference
---
# Flint EPQ for llama.cpp

CPU expert paging for `unsloth/flint-GGUF` (`flint-UD-Q4_K_XL.gguf`). Routed expert weights stay on disk and are loaded into a RAM arena on demand. Full results: [docs/REPORT.md](docs/REPORT.md).

## Install

Requires a CMake build of llama.cpp at commit `970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc` with `flint-epq.patch` applied (sha256 `369ef9710e507f5f8caeae398382399b43dd9adeeb006dfd27ce5e84bc041db4`).

```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc
curl -L 'https://huggingface.co/HelloSun/fint-epqv2/resolve/main/flint-epq.patch' -o flint-epq.patch
git apply flint-epq.patch
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DGGML_NATIVE=OFF \
  -DGGML_AVX2=ON -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON \
  -DGGML_AMX_INT8=OFF -DGGML_AMX_TILE=OFF -DGGML_AMX_BF16=OFF
cmake --build build -j 8 --target llama-cli
hf download unsloth/flint-GGUF flint-UD-Q4_K_XL.gguf --local-dir models
```

## Run

Examples use the 4 GiB memory tier (`SDQ_ARENA_MB=1024`, a 1 GiB expert arena). Prompt: `Explain in one sentence why the sky is blue.`

**native** (unmodified llama.cpp MoE path, needs about 24 GiB RAM, so it does not fit in 4 GiB):

```bash
SDQ_PAGER=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
  -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
```

**v1** (expert paging, mode C). Preload and row-parallel default to on, so turn them off explicitly:

```bash
SDQ_PAGER=1 SDQ_ARENA_MB=1024 SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0 \
  build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
  -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
```

**v2** (default settings, preload and row-parallel on). At 4 GiB the arena cannot hold all experts, so v2 runs the same path as v1 here:

```bash
SDQ_PAGER=1 SDQ_ARENA_MB=1024 \
  build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
  -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
```

**auto** (no `SDQ_ARENA_MB`): the arena is 80 % of available memory, capped at the size of all experts. Available memory is the smaller of `MemAvailable` and the cgroup headroom (`memory.max - memory.current`). With at least about 36 GiB available the cap is reached, so preload and row-parallel turn on automatically (the same behavior as v2):

```bash
SDQ_PAGER=1 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
  -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
```

The remaining 20 % must cover the non-expert weights and compute buffers, which load after the arena is sized. The auto estimate uses the cgroup's current usage, including reclaimable page cache, so it is conservative. Small limits have not been tested in auto mode; set `SDQ_ARENA_MB` explicitly there.

v2 changes behavior only when the arena holds all 12,032 experts (≥ 28.6 GiB, for example `SDQ_ARENA_MB=29400`).

Use 16 threads (`-t 16`). Row-parallel needs more threads than the number of experts used per step (6), so at least 7.

## Notes

- Memory at 4 GiB tier: peak 2.76 GiB. The process needs about 1.8 GiB plus the arena. Under a 2 GiB limit a 128 MiB arena works (peak 1.89 GiB); 256 MiB does not.
- Generation speed, 4 GiB tier, 16 threads (median): v1 4.6 t/s, v2 4.6 t/s (same path). At 32 GiB with full arena: native 19.4 t/s, v1 5.5 t/s, v2 13.7 t/s. Details are in the report.
- Output matches native greedy decoding at 8 threads; at 16 threads it diverges after a few tokens (float reordering). See the report.

	---
	library_name: llama.cpp
	license: mit
	base_model: RecursiveSI/flint
	tags:
	- flint
	- llama.cpp
	- moe
	- expert-offloading
	- cpu-inference
	---

	# Flint EPQ for llama.cpp

	CPU expert paging for `unsloth/flint-GGUF` (`flint-UD-Q4_K_XL.gguf`). Routed expert weights stay on disk and are loaded into a RAM arena on demand. Full results: [docs/REPORT.md](docs/REPORT.md).

	## Install

	Requires a CMake build of llama.cpp at commit `970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc` with `flint-epq.patch` applied (sha256 `369ef9710e507f5f8caeae398382399b43dd9adeeb006dfd27ce5e84bc041db4`).

	```bash
	git clone https://github.com/ggml-org/llama.cpp
	cd llama.cpp
	git checkout 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc
	curl -L 'https://huggingface.co/HelloSun/fint-epqv2/resolve/main/flint-epq.patch' -o flint-epq.patch
	git apply flint-epq.patch
	cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DGGML_NATIVE=OFF \
	-DGGML_AVX2=ON -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON \
	-DGGML_AMX_INT8=OFF -DGGML_AMX_TILE=OFF -DGGML_AMX_BF16=OFF
	cmake --build build -j 8 --target llama-cli

	hf download unsloth/flint-GGUF flint-UD-Q4_K_XL.gguf --local-dir models
	```

	## Run

	Examples use the 4 GiB memory tier (`SDQ_ARENA_MB=1024`, a 1 GiB expert arena). Prompt: `Explain in one sentence why the sky is blue.`

	native (unmodified llama.cpp MoE path, needs about 24 GiB RAM, so it does not fit in 4 GiB):

	```bash
	SDQ_PAGER=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
	-p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
	```

	v1 (expert paging, mode C). Preload and row-parallel default to on, so turn them off explicitly:

	```bash
	SDQ_PAGER=1 SDQ_ARENA_MB=1024 SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0 \
	build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
	-p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
	```

	v2 (default settings, preload and row-parallel on). At 4 GiB the arena cannot hold all experts, so v2 runs the same path as v1 here:

	```bash
	SDQ_PAGER=1 SDQ_ARENA_MB=1024 \
	build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
	-p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
	```

	auto (no `SDQ_ARENA_MB`): the arena is 80 % of available memory, capped at the size of all experts. Available memory is the smaller of `MemAvailable` and the cgroup headroom (`memory.max - memory.current`). With at least about 36 GiB available the cap is reached, so preload and row-parallel turn on automatically (the same behavior as v2):

	```bash
	SDQ_PAGER=1 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
	-p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
	```

	The remaining 20 % must cover the non-expert weights and compute buffers, which load after the arena is sized. The auto estimate uses the cgroup's current usage, including reclaimable page cache, so it is conservative. Small limits have not been tested in auto mode; set `SDQ_ARENA_MB` explicitly there.

	v2 changes behavior only when the arena holds all 12,032 experts (≥ 28.6 GiB, for example `SDQ_ARENA_MB=29400`).

	Use 16 threads (`-t 16`). Row-parallel needs more threads than the number of experts used per step (6), so at least 7.

	## Notes

	- Memory at 4 GiB tier: peak 2.76 GiB. The process needs about 1.8 GiB plus the arena. Under a 2 GiB limit a 128 MiB arena works (peak 1.89 GiB); 256 MiB does not.
	- Generation speed, 4 GiB tier, 16 threads (median): v1 4.6 t/s, v2 4.6 t/s (same path). At 32 GiB with full arena: native 19.4 t/s, v1 5.5 t/s, v2 13.7 t/s. Details are in the report.
	- Output matches native greedy decoding at 8 threads; at 16 threads it diverges after a few tokens (float reordering). See the report.