---
license: apache-2.0
language:
- en
library_name: onnx
pipeline_tag: audio-classification
tags:
- function-calling
- tool-calling
- spoken-language-understanding
- onnx
- on-device
extra_gated_fields:
Affiliation (Company or Academic Institution): text
Role: text
---
# NeuDecide
**A tiny voice-action model. Audio and a list of tools in, a tool call out. No transcript in between.**
[GitHub](https://github.com/neuphonic/neudecide) · [PyPI](https://pypi.org/project/neudecide/) · [HF Space](https://huggingface.co/spaces/neuphonic/neudecide) · arXiv (coming soon)
NeuDecide converts speech directly into function calls. It's 43 MB, can run on a single thread via ONNX Runtime, and needs no GPU or internet connection once downloaded. Tools are passed in as JSON schemas at inference time, so you change what the model can do by changing the JSON, not by retraining.
```
"clean the bathroom" + [cleanRoom, goToBase, getBattery] → [{"name": "cleanRoom", "arguments": {"room": "bathroom"}}]
```
## Installation
```bash
pip install neudecide
```
Requires Python 3.10 – 3.13. The model weights are downloaded from the Hugging Face Hub on first use and cached after that.
## Quickstart
This example turns a recorded voice command into a call for a robot vacuum. Save a WAV recording of yourself saying something like "clean the bathroom" as `command.wav`, then run:
```python
from neudecide import NeuDecide
TOOLS = [
{
"name": "cleanRoom",
"description": "Vacuum-cleans one room.",
"parameters": {
"type": "object",
"properties": {
"room": {
"type": "string",
"enum": [
"living room",
"bathroom",
"kitchen",
"bedroom",
],
},
},
"required": ["room"],
},
},
{
"name": "goToBase",
"description": "Sends the vacuum back to its charging base.",
"parameters": {
"type": "object",
"properties": {},
},
},
{
"name": "getBattery",
"description": "Reports the vacuum's battery level.",
"parameters": {
"type": "object",
"properties": {},
},
},
]
def main():
model = NeuDecide.from_pretrained()
calls = model.generate("command.wav", TOOLS)
print(calls)
if __name__ == "__main__":
main()
```
Output:
```
[{'name': 'cleanRoom', 'arguments': {'room': 'bathroom'}}]
```
`generate` returns a list of `{"name", "arguments"}` dicts, or `[]` when no tool applies. If the answer reaches the 128-token output limit, you get the calls completed before it, with a warning.
A command-line version is in [`examples/basic_example.py`](https://github.com/neuphonic/neudecide/blob/main/examples/basic_example.py). From a clone of the GitHub repo, `python examples/basic_example.py` runs it on `examples/command.wav` with the tools in `examples/tools.json`; pass `--audio` and `--tools` to use your own.
## Usage
### Audio input
`audio` can be a path to a PCM WAV file or a NumPy array. Arrays may be mono `(samples,)` or multi-channel `(samples, channels)`; channels are averaged to mono. Integer arrays, such as the int16 samples `scipy.io.wavfile.read` returns, are scaled to [-1, 1] automatically; float arrays should already be in [-1, 1]. Audio is resampled to 16 kHz automatically, so pass `sample_rate=` if your array is at a different rate:
```python
calls = model.generate("command.wav", TOOLS) # WAV file
calls = model.generate(waveform, TOOLS, sample_rate=44100) # NumPy array
```
Only WAV is read natively. For other formats (MP3, FLAC, OGG), load the audio yourself and pass the array. Audio longer than 30 seconds is truncated.
### Tool definitions
Each tool has a `name`, a `description`, and `parameters` as a JSON Schema object. You can pass the list directly or as a JSON string. The model only knows what you tell it about each tool, so clear names and descriptions do most of the work. When an argument has a fixed set of values, like the rooms above, list them with `enum`.
## How it works
NeuDecide is a pre-trained streaming English speech encoder joined to a compact tool-calling encoder–decoder, initialised from [Needle](https://github.com/cactus-compute/needle), and trained end to end as one system. Inference runs three ONNX graphs:
| Graph | Runs | Does | Parameters | Size |
|---|---|---|---|---|
| `audio_encoder` | Once per utterance | Waveform → 12.5 Hz latent frames | 28.7M | 22.5 MB |
| `tool_encoder` | Once per utterance and tool list | Audio frames + tool tokens → cross-attention keys and values | 11.8M | 10.1 MB |
| `decoder_step` | Once per output token | Next token, reusing a cached self-attention state | 15.0M | 10.0 MB |
| | |
|---|---|
| Model width | 512 |
| Decoder layers | 8 |
| Attention | Grouped-query, 4 key/value heads of 64 dimensions |
| Vocabulary | 8,192 SentencePiece BPE tokens |
| Sample rate | 16 kHz |
| Limits | 30 s of audio, 1,536 tool tokens, 128 output tokens |
Tools are serialised exactly as Python's `json.dumps` writes them by default, matching how the model was trained.
## Results
We compared NeuDecide with five speech-recognition cascades (Parakeet 110M and 660M feeding Needle or FunctionGemma, and Whistle feeding Needle 3) and with Voxtral Mini 3B, an end-to-end speech model. Each system chooses from 10 tools. Some requests need only the right tool; the rest also need its arguments filled, and exact match counts a call as right only if the tool and every argument match. The results cover both kinds of request together, weighted by how many of each SLURP has, as in the plot. The table shows two of the cascades alongside Voxtral Mini 3B; a cascade's size counts its ASR and decision model together.
| | | NeuDecide | Whistle → Needle 3 | Parakeet 660M → FunctionGemma | Voxtral Mini 3B |
|---|---|---|---|---|---|
| **Size** | Parameters | 55.5M | 176M | 870M | 3B |
| **Size** | File size | 43 MB | 52 MB | 3.0 GB | 18.7 GB |
| **Size** | File size vs NeuDecide | 1× | 1.2× | 71× | 435× |
| SLURP | Tool accuracy | **75.5%** | 25.6% | 25.2% | 48.7% |
| SLURP | Exact match | **37.5%** | 7.1% | 1.9% | 6.9% |
On SLURP, NeuDecide beats every other system on both metrics. With 10 tools, it gets more than five times as many commands exactly right as the best cascade or Voxtral Mini 3B.
## Performance
Measured with the `q4` export:
| Device | Time to call | Load time |
|---|---|---|
| MacBook Pro M3 | 46 ms | 159 ms |
| Samsung S24+ | 82 ms | 267 ms |
| Raspberry Pi 5 | 206 ms | 499 ms |
Peak RAM ranged from 146 MB to 174 MB across these devices; it depends on the runtime and workload.
## Limitations
- **It can fire on noise.** Given 30 s of low-level noise, it still produced a call. There is no calibrated confidence score yet, so put a gate in front: voice-activity detection, or an explicit "no action" tool.
- **Keep tool lists short.** We recommend at most 10 tools. Longer lists also cost more compute than longer audio, and lists over 1,536 tokens are truncated with a warning.
- **English only.**
- **Length limits.** Inputs are capped at 30 s of audio, and outputs are capped at 128 tokens.
- **Free-form arguments are harder.** On SNIPS, where arguments are often open-ended names such as artists and playlists, transcript-based cascades still fill arguments more accurately. Use `enum` wherever an argument has a fixed set of values.
Test it on your own tools and your users' voices before deploying. NeuDecide works best next to a larger model: let it handle the fast, frequent, bounded decisions, and route anything it can't place to a bigger model or a person.
## Acknowledgements
- https://github.com/cactus-compute/needle
- https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms
## License
Apache 2.0. See [LICENSE](https://github.com/neuphonic/neudecide/blob/main/LICENSE).