ORB planner
Code · Live demo (Space) · Project page · Training data
A 5.10M-parameter decoder-only transformer that turns a one-line caption, or plain words like "rain on a tin roof with distant thunder" (since v8), into a compact, typed Spec. It does not generate audio by itself: the Spec is compiled by a deterministic DSP renderer in the ORB repository.
caption ──► this model ──► Spec ──► DSP render ──► 48 kHz WAV
Results
| Metric | Value |
|---|---|
| Held-out grammar captions (500) | 500/500 (100%), every field 100% |
| Held-out grammar captions (2,000) | 2000/2000 (100%) |
| Plain-text prompts, our set (209) | 58.4% fully right (v7 + keyword rules: 45.0%) |
| Plain-text prompts, blind set (40) | 70.0% fully right (v7 + keyword rules: 77.5%) |
| Decode failures | 0 |
| Parameters | 5,096,192 |
| Context | 1024 |
| Vocab | 395 (byte-level + structural; 47 roles, 13 textures, NOBED) |
A plain-text prompt passes only if roles, extra roles, texture, mood, intensity and duration are all right. The blind set was written by someone else; the gap to our own set shows that the free-text skill generalises less than our set suggests.
Usage
pip install "orb-audio[train,hub] @ git+https://github.com/stdevMac/orb-audio"
from huggingface_hub import hf_hub_download
from orb.api import load_planner, generate, save_wav
path = hf_hub_download("marcosmaceo/orb-planner", "planner.safetensors")
hf_hub_download("marcosmaceo/orb-planner", "config.json") # read from the same folder
tok, planner = load_planner(path)
out = generate(
"6.00-second bright cue, wind bed, with whoosh @1.00 0.90s -6dB L2 #2a",
planner=(tok, planner),
)
save_wav(out["audio"], "whoosh.wav")
print(out["spec"])
Or without this package, using the config and safetensors directly:
import json, torch
from safetensors.torch import load_file
from orb.model.planner import Planner
cfg = json.load(open("config.json"))
model = Planner(cfg["vocab_size"], **cfg["model"])
model.load_state_dict(load_file("planner.safetensors"))
model.eval()
Decoding is constrained by a grammar mask, so every output is a valid Spec
(t + dur <= duration, categorical ranges, at most 8 events).
The Spec
{
"duration": 6.0,
"texture": "wind",
"mood": "bright",
"events": [
{"role": "whoosh", "t": 1.0, "dur": 0.9, "gain_db": -6.0, "pan": -0.5, "seed": 42}
]
}
- 47 roles: the 10 core ones (whoosh, impact, ui_pop, riser, swell, click, bell, sub_drop, glitch, noise_sweep) plus 37 physically modelled natural, foley and everyday sounds (thunder, door_slam, footsteps, bird_chirp, wave_crash, glass_break, phone_ring, siren, …)
- 13 textures: room_cold, wind, city_hum, vinyl, drone, rain, fire, sci_fi, ocean, stream, forest, night, traffic, or
none - 10 moods: calm, warm, playful, hopeful, bright, tense, dark, eerie, neutral, intense
For agents
The Spec is plain JSON, so an agent can read, diff and edit a sound like any
other config. The repository ships an MCP server (orb-mcp, install with the
mcp extra) that exposes planning and rendering as tools; see
docs/agents.md.
Training
Trained from scratch on synthetic data only: v8 is warm-started from v7 and
trained for 40k steps (batch 32, AdamW, cosine schedule, one H100) on 150k
procedurally generated grammar (caption, Spec) pairs from
orb.grammar.sample_spec plus 150k synthetic plain-text (prompt, Spec) pairs
from orb.freetext. No pretrained weights, no external audio, no
human-labelled data. The published
orb-planner-data
dataset holds exactly these 300k training rows, plus the held-out sets.
Intended use
Turning structured captions into an editable sound description for the ORB renderer. Grammar captions are read exactly. Plain words are read on a best-effort basis, and only into the roles and textures above.
Limitations
- For a noun it has no sound for, the planner may invent the nearest role. Counts ("three knocks") and long multi-clause prompts are weak. Write a grammar caption or pass a Spec dict when exactness matters.
- v1 is SFX and texture only; music is not modelled.
License
Apache-2.0. See the repository for the renderer, examples and the rest of the engine.
- Downloads last month
- 33
Dataset used to train marcosmaceo/orb-planner
Space using marcosmaceo/orb-planner 1
Evaluation results
- Held-out exact Spec match (%) on ORB scores (500 held-out captions)self-reported100.000