AILOFlow
A local runtime for open-weight LLMs, with dynamic weight streaming from storage.
Run models larger than your RAM by treating the weights as a dataset spread across every SSD in the machine — and measure honestly what that actually costs.
Source code: https://github.com/xxrickyxx/AiloFlow
The one rule
Every number shown is measured.
Where a value cannot be measured on the machine, the interface prints n/a
instead of a plausible-looking placeholder. Estimates are labelled as estimates.
A storage benchmark that was served by the OS page cache says so out loud, rather
than reporting a SATA SSD at 9 GB/s.
This is not a demo that simulates a runtime. Nothing renders a number that did not come from a real measurement.
What it does
Manages its own inference engine. Picks between CUDA, HIP/ROCm, Vulkan, Metal and CPU based on the GPUs actually present, downloads the matching llama.cpp build from the official releases, verifies it and runs it as a child process. No external daemon required. Ollama is supported as an optional engine, and models already downloaded through Ollama run even with its daemon stopped — they are ordinary GGUF files.
Speaks two protocols on one port, so IDE assistants can target it directly:
| Protocol | Base URL | Endpoints |
|---|---|---|
| OpenAI | http://127.0.0.1:11500/v1 |
/models, /chat/completions, /completions |
| Ollama | http://127.0.0.1:11500 |
/api/tags, /api/chat, /api/generate, /api/show, /api/ps |
Tested with Cline and Continue. Reasoning models keep their chain of thought in
thinking / reasoning_content, separate from the answer, so it never
contaminates generated code.
Cancelling a request stops the engine: a disconnected client whose generation
keeps running would hold the GPU for minutes producing tokens nobody reads. For
clients that stop reading without closing the socket, a new request supersedes
the old one and POST /v1/generation/stop frees the GPU on demand.
Whatever base URL you configure works — .../v1, .../api or the bare host —
because providers append /api/chat to whatever you typed and the duplicated
prefix is normalised instead of returning an HTML 404 into the IDE.
For agent use, pick an Instruct model, not a Thinking one. A Thinking
variant reasons before every answer and cannot be told not to: enable_thinking,
reasoning_budget: 0 and /no_think are all ignored by a model trained to
always reason. At 19 tok/s that is minutes of deliberation per step, and the
reasoning eats the context the agent needs for its own instructions.
Reads the hardware instead of guessing it. CPU instruction sets come from
IsProcessorFeaturePresent / /proc/cpuinfo / sysctl, so an i9-13900K is
correctly reported as AVX2 rather than AVX-512. GPU VRAM comes from the driver
registry key, because WMI's AdapterRAM is 32-bit and reports 4 GB for anything
larger. GPU utilisation comes from performance counters that work on AMD, Intel
and NVIDIA alike. Volumes are joined to physical disks through the partition
table, so a drive is never mislabelled by array position.
Distributes a model across drives into a .sflow container, validates that
the shards on disk match the manifest, and measures the streaming pipeline —
bandwidth, per-layer latency, cache hit rate, and how many tokens per second
storage alone could sustain.
Estimates a model before you download it, from the machine's measured bandwidths.
Measured, on a real machine
i9-13900K · 32 GB DDR5-6000 · Radeon RX 6750 XT · 1 NVMe + 4 SATA SSD
| SATA SSD sequential read (cache-cold) | 371–462 MB/s, 220 µs |
| RAM bandwidth (single-threaded memcpy) | 35.7 GB/s |
| Qwen3-1.7B, own Vulkan engine | 235–240 tok/s |
| Qwen3-Next-80B-A3B, own Vulkan engine | 19.4 tok/s |
| 80B sharded across 5 SSDs, full layer sweep | 44.97 GB in 95 s |
The 80B sweep is genuinely cache-cold: the shard set is larger than RAM.
What it will not do
Decode is bandwidth-bound — one token requires reading every active weight — and the arithmetic is unforgiving:
| Model | Bytes per token | Estimated on the machine above |
|---|---|---|
| 300B dense, Q4 | 169 GB | 0.13 tok/s |
| 300B MoE, 32B active | 18 GB | 1.2 tok/s |
| gpt-oss-120b, 5.1B active | 2.9 GB | 12.6 tok/s |
A dense 300B streamed from disk would need 1345 GB/s to reach 10 tok/s — two orders of magnitude beyond any consumer array. Mixture-of-Experts models, which touch a fraction of their weights per token, are the ones this architecture can genuinely serve.
Generation itself runs on llama.cpp's compute kernels; rewriting those was never the goal. Chatting with a model therefore does not exercise the streaming pipeline — llama.cpp loads weights its own way. The useful question is not "AILOFlow versus llama.cpp on tokens/s" but can storage feed the rate the engine achieves?
Catalog
40 curated open-weight models — Kimi K2, DeepSeek R1/V3, Qwen3 235B/480B, Llama 4, GLM-4.6, MiniMax-M2, gpt-oss and more — filterable by size and capability, plus free search across Hugging Face. Every repository id is verified against the API in the test suite.
Downloads belong to the runtime rather than to a browser tab: they survive navigation, page reloads and server restarts, retry automatically on network failures with exponential backoff, and resume from the exact byte. Models published as split file sets are handled as a single model.
Install
git clone https://github.com/xxrickyxx/AiloFlow.git
cd AiloFlow
npm install
npm run server # API on 127.0.0.1:11500
npm run gui:dev # interface on localhost:3000
Then: Settings → Install engine, Catalog → download a model, Models → load, Chat.
Interface available in English and Italian.
Privacy
No prompt and no model leaves the machine. The API listens on 127.0.0.1 only,
and there is no remote telemetry of any kind.
Licence
Dual-licensed:
- Non-commercial use — CC BY-NC-SA 4.0
- Commercial use — separate written agreement required
llama.cpp is a separate work under the MIT License and is not redistributed here; AILOFlow fetches it at runtime from the official releases. Models downloaded through the catalog carry their own licences, which the catalog displays.
Author: Riccardo Sparacino — LinkedIn