runtime/README.md

# DecisionNode Edge runtime

The official runtime for the DecisionNode Edge models from Bynn Intelligence. Load a model, send a request in the DecisionNode API format, and receive typed, calibrated decisions. It runs on NVIDIA GPUs, Apple silicon and CPUs, entirely offline.

Models: [DecisionNode Edge](https://huggingface.co/bynn-intelligence-com/decisionnode-edge) and [DecisionNode Edge Mini](https://huggingface.co/bynn-intelligence-com/decisionnode-edge-mini) on Hugging Face.

## Install

```bash
pip install decisionnode-edge              # torch, transformers, safetensors
pip install "decisionnode-edge[images]"    # adds Pillow and torchvision, for pictures
```

The model folders carry a copy of this package in `runtime/` for machines without network access: `pip install ./runtime` (or `"./runtime[images]"`) from the model folder installs the same thing.

Optional, for speed on NVIDIA GPUs with the PyTorch backend: `flash-linear-attention` and `causal-conv1d`. transformers uses them when they are installed. Without them the model gives the same answers about twice as slowly on long inputs.

### The fastest install for your device

The runtime reads the model with PyTorch unless a faster backend is installed. Install the one for your device:

| Device | Install | Backend |
|---|---|---|
| NVIDIA GPU (Linux) | `pip install "decisionnode-edge[vllm]"` | `vllm` |
| x86 CPU | `pip install "decisionnode-edge[openvino]"`, then build the OpenVINO files once ([below](#building-what-a-backend-loads)) | `openvino` |
| Apple silicon | `pip install "decisionnode-edge[mlx]"` | `mlx` |
| Any device llama.cpp supports, including NVIDIA Jetson | llama.cpp's `llama-server`; the model folder ships the 8-bit GGUF file ([below](#building-what-a-backend-loads)) | `llamacpp` |

Every backend agrees with the PyTorch reference on at least 98% of top answers in our tests, and the remaining differences fall on near ties. A backend only computes the model's last hidden state; the runtime then applies the model's own answer table (`readout.safetensors`) and calibration (`calibration.json`) in float32, as it does on PyTorch.

Measured with one `truth` question per request, one request at a time, median time of the whole `decide` call:

| Device (as measured) | Backend | Edge, about 500 / 2,000 tokens | Edge Mini | Answers kept (Edge, Edge Mini) | PyTorch on the same device (Edge; Edge Mini) |
|---|---|---|---|---|---|
| NVIDIA RTX PRO 6000 Blackwell | `vllm` | 9.5 / 23 ms | 7.3 / 15 ms | 99.7%, 99.2% | 14 / 28 ms; 14 / 16 ms (with the optional kernels) |
| AMD Ryzen 7 9800X3D (8 cores) | `openvino` | 0.46 / 1.7 s | 0.24 / 0.90 s | 98.6%, 98.4% | bfloat16: 0.49 / 2.6 s; 0.28 / 1.4 s. float32, the default: 1.6 / 7.8 s; 0.66 / 4.0 s |
| Apple M5 Max | `mlx` | 35 / 127 ms | 17 / 65 ms | 99.5%, 99.3% | `device="mps"`: 0.30 / 1.3 s; 0.28 / 1.2 s |
| Apple M5 Max | `llamacpp` (Metal) | 55 / 212 ms | 30 / 112 ms | 99.4%, 99.0% | |
| NVIDIA RTX PRO 6000 Blackwell | `llamacpp` (CUDA) | 14 / 44 ms | 10.5 / 30 ms | 99.2%, 98.4% | |
| AMD Ryzen 7 9800X3D (8 cores) | `llamacpp` (CPU) | 1.1 / 5.1 s | 0.58 / 2.7 s | 99.8%, 98.1% (412 questions up to 512 tokens) | |

At about 8,000 tokens: `vllm` 87 ms (Edge) and 50 ms (Edge Mini), `mlx` 625 and 297 ms, `llamacpp` 218 and 160 ms on CUDA and 0.92 and 0.55 s on Metal. OpenVINO hands prompts of that length to PyTorch.

"Answers kept" is how often the backend's top answer equals the PyTorch reference (bfloat16 on an NVIDIA GPU) on a fixed set of 2,056 held-out questions; for OpenVINO, on the 1,936 of them up to 2,048 tokens, the longest it is built for.

The GPU is an NVIDIA RTX PRO 6000 Blackwell workstation card, the CPU an AMD Ryzen 7 9800X3D (8 cores, 16 threads), the Mac an Apple M5 Max. Versions: vLLM 0.32.0, OpenVINO 2026.4.1, MLX 0.32.4 with mlx-lm 0.32.2, llama.cpp at commit f174553, PyTorch 2.13 and 2.14, transformers 5.18 and 5.19. All backends ran in bfloat16 except llama.cpp, which ran the 8-bit (Q8_0) file.

## Use

```python
from decisionnode_edge import EdgeModel

model = EdgeModel.load("path/to/decisionnode-edge")   # CUDA if present, else the CPU

response = model.decide({
    "state": {"ticket": "I was charged twice and nobody has replied for 3 days."},
    "questions": {
        "route": {
            "type": "choice",
            "instructions": "Where should `ticket` go?",
            "criteria": {"billing": "money", "bug": "something broken", "account": "login and access"},
        },
        "urgency": {
            "type": "score",
            "instructions": "How urgent is `ticket`?",
            "criteria": ["routine", "today", "urgent", "critical"],
        },
        "escalate": {"type": "truth", "instructions": "Should a human take over now?"},
        "days": {"type": "number", "instructions": "How many days has the customer waited?", "min": 0, "max": 30},
    },
})

response["answers"]["route"]       # {"type": "choice", "choice": ..., "confidence": ..., "probabilities": {...}}
response["answers"]["escalate"]    # {"type": "truth", "truth": ...}
```

`model.predict(request)` gives the unrounded calibrated probabilities per question instead: one per option or level in request order, `[yes, no]` for a truth question, one per grid value for a number question. Point and box questions are answered by `decide` only.

A response names the model that answered: `decisionnode-edge-0.9.1` for DecisionNode Edge, `decisionnode-edge-mini-0.9.1` for DecisionNode Edge Mini (the `name` in the model directory's `decisionnode_edge.json`).

From the command line:

```bash
python -m decisionnode_edge path/to/decisionnode-edge request.json
cat request.json | python -m decisionnode_edge path/to/decisionnode-edge -
```

### Serving over HTTP

```bash
python -m decisionnode_edge serve path/to/decisionnode-edge --host 127.0.0.1 --port 8000
python -m decisionnode_edge serve path/to/decisionnode-edge --api-key key.txt     # callers send Authorization: Bearer <key>
```

A small server on Python's standard library, with nothing more to install. It loads the model with the backend `EdgeModel.load` picks (or `--backend`, `--artifacts`, `--device`, `--dtype`), answers one request at a time, and listens on 127.0.0.1 unless `--host` says otherwise.

| Route | What |
|---|---|
| `POST /v1/systemone` | the `/v1/systemone` format used by public decision-model leaderboards: `{"model", "state", "questions", "assets"}` with `noul`, `choice`, `score` and `box` questions. A `noul` is answered `{"type": "noul", "noul": p}`, a choice and a score as on `/v1/decide` with probabilities that sum to exactly 1, a box as below. `model` is required, and any name is accepted: a local server has no model catalogue, and the answer names the model that answered. Number and point questions go to `/v1/decide` |
| `POST /v1/decide` | the request and answers of this runtime ([below](#the-request)), `number`, `truth`, `point` and `box` included |
| `GET /v1/models` | `{"models": [{"name", "description", "release_date"}]}` |
| `GET /healthz` | `{"status": "ok", "model", "backend"}`, no key needed |

Every answer carries `usage`: `input_tokens` and `output_tokens` (always 0). Pictures go as base64: the runtime's `images` with `data` (a local `path` is refused over HTTP, so a caller can never make the server read a file), or, on `/v1/systemone`, the format's own:

- an `assets` map, `{"<ref>": {"type": "image", "format": "png", "data_base64": "..."}}` with format `png`, `jpg` or `jpeg` (the bytes decide when they say otherwise). Wherever a reference appears in the state, an instruction or a criterion, the text reads `Picture <n>` and the asset becomes picture n (`image` when it is the only one, else `image_<n>`), numbered in the order the references first appear. An asset no text refers to is refused. A score's `legend` shows its levels as sent, references included.
- `image_url` content parts (`{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}`) in the same places, and `images` as a list of data URLs.

This is the 0.9.2 behaviour, the same as the hosted `/v1/systemone`. In 0.9.1 each asset became a picture whose id was its reference, and the text stayed as sent. `/v1/decide` reads pictures as before.

A `box` question on `/v1/systemone` is `{"type": "box", "instructions": "..."}`. Its answer is `{"type": "box", "box": {"top", "bottom", "left", "right"}, "confidence"}`, the edges as fractions of the picture's height and width. When the request holds several pictures, the box looks in the one a path in backticks names in the state (`` `photo` ``, `` `cams.door` ``), else in the first picture of the state; with the runtime's own `images`, name it with `image`, as on `/v1/decide`. A box with `max` keeps the list answer of `/v1/decide`.

`confidence` is the probability that the box is right (an intersection over union of 0.5 or more with the true box) when the model folder holds a box calibration, `box_calibration.json`. Without one it is `present`, the probability that the thing is in the picture. The server reads the file once at start and refuses to start on a file it cannot check. `--box-calibration FILE` uses another file and `--box-calibration none` uses none; `--box-calibration-sha256` (or `box_calibration_sha256` in `decisionnode_edge.json`) pins the file's sha256. `/v1/decide` never reads it.

Errors come as `{"detail": {"error_type": ..., "message": ...}}` (a 422 as a list of field errors). A request longer than the model's limit is a 400 whose message says "maximum context length"; a choice with more than 255 options is a 400 that says "options per choice". On `/v1/systemone` these limits come in the hosted route's bodies, which benchmark harnesses read to record a request as beyond the model:

| Limit | `/v1/systemone` body |
|---|---|
| longer than the model's limit | `{"detail": {"error_type": "max_tokens_exceeded", "message": "The request does not fit the model's maximum context length (...)."}}` |
| more than 255 options | `{"detail": "Too many choices. Must have at most 255 choices (255 options per choice)."}` |
| more than 10 levels | `{"detail": "Too many score levels. Must have at most 10 levels (a score takes 2 to 10 levels; one level is answered without the model)."}` |

### Options

`EdgeModel.load(path, device=None, dtype=None, backend=None, artifacts=None, options=None)`:

- `device`: `"cuda"`, `"cuda:1"`, `"cpu"` or `"mps"`. Default: CUDA when present, else the CPU. It also sets the device class the backend is chosen for.
- `dtype`: `"bfloat16"` or `"float32"`, for PyTorch. Default: bfloat16 on a GPU (the reference precision) and float32 on a CPU. On CPUs with bfloat16 support (recent AMD and Intel parts) `dtype="bfloat16"` was about three times faster in our tests, with answers as close to the GPU's as float32 gives.
- `backend`: `"auto"` (the default), `"torch"`, `"vllm"`, `"openvino"`, `"mlx"` or `"llamacpp"`.
- `artifacts`: the directory the build tool wrote, for `openvino` and `llamacpp`.
- `options`: the backend's settings ([below](#backend-options)).

### Choosing a backend

By default `EdgeModel.load` takes the fastest backend that is installed and can run on the machine:

| Device class | Order |
|---|---|
| NVIDIA GPU | `vllm`, `llamacpp`, `torch` |
| Apple silicon | `mlx`, `llamacpp`, `torch` |
| Any other CPU | `openvino`, `llamacpp`, `torch` |

`openvino` counts only when `artifacts` holds an OpenVINO build, and `llamacpp` only when `artifacts` holds a GGUF build and `llama-server` is found (or `options` names a running server). Asked for by name, `openvino` and `llamacpp` also find a build in the model folder itself, such as the GGUF file that ships there. `model.backend` says which one was taken.

To choose yourself:

```python
model = EdgeModel.load("path/to/decisionnode-edge", backend="torch")
model = EdgeModel.load("path/to/decisionnode-edge", backend="openvino", artifacts="path/to/edge-openvino")
model = EdgeModel.load("path/to/decisionnode-edge", backend="llamacpp", options={"url": "http://127.0.0.1:8080"})
```

The same from the environment: `DECISIONNODE_EDGE_BACKEND`, `DECISIONNODE_EDGE_ARTIFACTS`, and `DECISIONNODE_EDGE_OPTIONS` (a JSON object). Arguments win over the environment. On the command line: `python -m decisionnode_edge path/to/decisionnode-edge request.json --backend openvino --artifacts path/to/edge-openvino`.

The faster backends read text. PyTorch reads what they do not take: pictures, and prompts longer than the backend's limit (an OpenVINO build's longest length, 2,048 tokens by default; a llama.cpp context, 16,384 tokens by default). It is loaded the first time it is needed, on `device` in `dtype`, and stays loaded. If most of your requests carry pictures, use `backend="torch"`.

### Building what a backend loads

The build tools read the model directory and write a new directory. They never change the model directory.

**OpenVINO.** One file per static input length, 512 and 2,048 tokens by default:

```bash
pip install "decisionnode-edge[openvino]"
python -m decisionnode_edge.build openvino path/to/decisionnode-edge --out path/to/edge-openvino
python -m decisionnode_edge.build openvino path/to/decisionnode-edge --out path/to/edge-openvino --lengths 512,1024,2048
```

The export runs on the GPU when there is one (`--device cuda`, then the default) and on the CPU otherwise. On our machine it took 14 minutes for 512 tokens and 19 minutes for 2,048, per length. The files keep the weights in float32 (exact copies of the bfloat16 weights) and OpenVINO runs them in bfloat16, so the build takes 6.2 GB for Edge Mini and 15.2 GB for Edge. Before it ends, the tool runs each file once against PyTorch.

A prompt is padded to the next length it fits, so a 600-token prompt costs about what a 2,048-token one does. Add a length (for example `--lengths 512,1024,2048`) if many of your prompts fall between two lengths. Lengths above a few thousand tokens take hours to export; longer prompts go to PyTorch.

**GGUF for llama.cpp.** The model folder ships the 8-bit file (`decisionnode-edge-0.9.1-q8_0.gguf`, `decisionnode-edge-mini-0.9.1-q8_0.gguf`) with its build record, so `EdgeModel.load("path/to/decisionnode-edge", backend="llamacpp")` uses it with no build step. To make it yourself (8-bit keeps the answers; 4-bit does not, so the tool does not offer it):

```bash
git clone https://github.com/ggml-org/llama.cpp
cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release        # add -DGGML_CUDA=ON on an NVIDIA GPU or a Jetson
cmake --build llama.cpp/build --target llama-server -j
pip install -r llama.cpp/requirements/requirements-convert_hf_to_gguf.txt
python -m decisionnode_edge.build gguf path/to/decisionnode-edge --out path/to/edge-gguf --llama-cpp llama.cpp
```

The conversion takes seconds and writes one file: 2.0 GB for Edge, 0.8 GB for Edge Mini. The same file serves a CPU, an NVIDIA GPU, a Jetson and a Mac; llama.cpp itself is built for each.

The runtime then starts `llama-server` on 127.0.0.1 itself (from your `PATH`, or `options={"server": "path/to/llama-server"}`) and stops it with the model. To run the server yourself, start it with the embedding endpoint and the last-position pooling, with a batch as large as the context:

```bash
llama-server -m edge-gguf/model-q8_0.gguf --embeddings --pooling last -c 16384 -b 16384 -ub 16384 -ngl 999 --port 8080
```

and pass `options={"url": "http://127.0.0.1:8080"}`. The runtime sends token ids, so llama.cpp's own tokenizer is never used.

### Backend options

Options can be flat (`{"threads": 16}`) or under a backend's name (`{"openvino": {"threads": 16}}`), so one set of options can serve whichever backend is taken.

| Backend | Option | Default |
|---|---|---|
| `vllm` | any vLLM engine argument, such as `gpu_memory_utilization` or `enforce_eager` | a 2 GiB KV cache; the share of the GPU that covers the weights, that cache and 4 GiB for activations |
| `openvino` | `threads` | OpenVINO's choice |
| `openvino` | `precision`: `"bf16"` or `"f32"` | `"bf16"` (OpenVINO falls back to f32 on CPUs without bfloat16) |
| `openvino` | `config`: other OpenVINO CPU properties | |
| `llamacpp` | `url`: a running `llama-server` | start one |
| `llamacpp` | `server`, `gguf`: the program and the model file | `llama-server` on `PATH`, the build's file |
| `llamacpp` | `gpu_layers`, `threads` | 999 (all layers on the GPU when there is one), llama.cpp's choice |
| `llamacpp` | `context`: the longest prompt llama.cpp takes | 16,384 |
| `mlx` | `cache_limit_gb`: how much freed GPU memory MLX keeps for reuse | 2 |

vLLM runs its engine inside your process and uses the first visible GPU (choose it with `CUDA_VISIBLE_DEVICES`). Its first load takes about half a minute while it compiles. MLX uses the Apple GPU.

### Pictures

Send pictures beside the state, up to 16, each with an `id`:

```python
{"images": [{"id": "receipt", "data": "<base64>", "media_type": "image/jpeg"}]}   # as with the hosted API
{"images": [{"id": "receipt", "path": "/data/receipt.jpg"}]}                       # a local file, Edge only
```

A picture is turned upright from its EXIF orientation, converted to RGB and scaled so its longest side is at most 2,048 pixels. Every question sees every picture. The prompt names a picture `Picture <n> (<id>)`, so instructions can refer to "Picture 2" or to "the receipt".

### Points and boxes

A `point` question asks where something is in a picture, a `box` question asks for a rectangle around it. Add `max` to get every match as a list:

```python
response = model.decide({
    "images": [{"id": "belt", "path": "/data/belt.jpg"}],
    "questions": {
        "pick": {"type": "point", "instructions": "The yellow package closest to the robot arm."},
        "damage": {"type": "box", "instructions": "The damaged package.", "units": "normalized"},
        "parcels": {"type": "box", "instructions": "Every parcel on the belt.", "max": 10},
    },
})

response["answers"]["pick"]
# {"type": "point", "x": ..., "y": ..., "present": ..., "confidence": ..., "image": "belt", "width": 1920, "height": 1080}
response["answers"]["parcels"]
# {"type": "box", "count": ..., "present": ..., "complete": ..., "boxes": [{"x1": ..., "y1": ..., "x2": ..., "y2": ..., "confidence": ...}, ...],
#  "image": "belt", "width": 1920, "height": 1080}
```

| Field | Notes |
|---|---|
| `instructions` | what to find, required: a phrase ("the yellow package"), a description with conditions, or a question ("Where is the crack?") |
| `image` | the id of the picture to look in; needed when the request has two or more pictures |
| `units` | `"pixels"` (the default): whole pixels of the picture as sent, after its EXIF orientation. `"normalized"`: fractions of the width and height with 4 decimals |
| `max` | 2 to 16: list up to that many places in place of one |

In an answer, `x` and `y` are measured from the top-left corner, `x` to the right and `y` down. A box is its top-left corner (`x1`, `y1`) and its bottom-right corner (`x2`, `y2`). `present` is the probability that the thing is in the picture at all; when it is low, the place is only the best guess. `confidence` says how sure the model is of the place: the product of the peak probabilities of its coordinate readings. It is not calibrated yet, so use it to compare answers, not as a probability. `image`, `width` and `height` name the picture and its size.

With `max`, `count` is the number of places listed, `complete` the probability that there are no more than `max`, and the places come from the left, ties from the top, each with its own `confidence`. Two boxes that overlap with an intersection over union of 0.7 or more are one thing: the less confident one is left out.

A point or box question takes a few short prompts on the same picture: whether the thing is there, then where it is across the width, then down the height at that spot, and for a box its width and height around that centre. A point takes 3 prompts, a box 5, a list of n boxes 2 + 4n. Each prompt reads the picture again, so a box costs about five times a `truth` question on the same picture. As with every question that carries pictures, PyTorch reads them whichever backend is chosen.

## The request

The request and the answers follow the DecisionNode API's `/v1/decide`:

| Field | Notes |
|---|---|
| `state` | a string (used as it is), an object or an array (compact JSON, key order kept); missing, null and `""` are the empty state |
| `questions` | id to question, at least one; the id is never shown to the model |
| `images` | optional, see above |
| `model` | ignored |

| Question type | Fields |
|---|---|
| `choice` | `instructions` (optional), `criteria`: {option: description}, 1 to 255 options; a description may be a string, an object, an array or null |
| `score` | `instructions` (optional), `criteria`: [level, ...] low to high, 1 to 10 levels |
| `truth` | `instructions` and/or `criteria`: {"true": ..., "false": ...} |
| `number` | `instructions`, `min`, `max`, `step` (defaults 0, 255, 1), up to 4,096 values; more than 256 values are answered in pages of 256 and composed |
| `point`, `box` | `instructions`, `image` (with two or more pictures), `units` (`pixels` or `normalized`), `max` (2 to 16, a list); see [Points and boxes](#points-and-boxes) |

Answers are rounded as the API rounds them: probabilities to 2 decimals (3 for number questions), with the largest entries moved by 0.01 until a list sums to 1.

A request the model cannot answer raises `decisionnode_edge.RequestError`, whose `status` is the HTTP status the hosted API gives the same fault (400 or 422). For point and box questions, `errors` lists every faulty field as the hosted API's 422 does. The runtime checks what the model needs; the hosted API's error bodies are not reproduced word for word. Requests longer than the model's limit (`limits.max_input_tokens` in `decisionnode_edge.json`, 16,384 tokens) are refused, never cut.

Not on Edge: videos, boosters, image URLs and uploads, sessions and batch jobs. These are features of the hosted API.

## How it works

Each question becomes one prompt: the state block (pictures first, then the state text), then the question block (the type, the instructions, then one line per option, each led by an answer code token), then the word `Answer` and an answer marker. The codes of a question are drawn from a hash of the question alone, so the same question always gets the same codes. The model reads the prompt in one forward pass, and the last hidden state is scored against the rows of `readout.safetensors` that belong to the question's codes (or to its yes and no tokens). `calibration.json` turns those scores into probabilities. The model never generates text.

A point or box question is a short series of such prompts. Its coordinate prompts list no options: they are answered by the 256 code tokens in order, code k meaning step k of 256 along a side of the picture, and the answer is the probability-weighted mean of the steps around the most probable one. A later prompt is given what the earlier ones read: the y is asked at the x already chosen, and a box's size around its centre.

The code is a few short modules: `template.py` (the prompt and the codes), `request.py` (reading a request), `answers.py` (calibration and the answer shapes), `places.py` (point and box questions), `backends.py` (what reads the backbone: PyTorch, vLLM, OpenVINO, MLX or llama.cpp), `build.py` (the OpenVINO and GGUF builds), `server.py` (the HTTP server) with `systemone.py` (the `/v1/systemone` format), and `model.py` tying them to the weights.

## License

The runtime code, like the model weights, is licensed under the GNU Affero General Public License, version 3 only (SPDX: `AGPL-3.0-only`). For use in closed products and services, Bynn Intelligence Inc. offers a commercial license: licensing@bynn.com.