File size: 5,689 Bytes
570b87b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 | ---
language:
- en
library_name: other
pipeline_tag: other
license: other
tags:
- ambisonics
- spatial-audio
- hoa
- spherical-harmonics
- audio-analysis
- tool
- agi-tool
- wigner-d
---
# HOA7 Spatial Field Decoder (hoa64)
A **formula-driven spatial field decoder** for **7th-order Ambisonics** — **64 channels** (Ambix **ACN + SN3D**). It encodes/decodes spherical fields, rotates them with **Wigner-D** matrices, analyzes direction-of-arrival / energy, fuses vision detections onto the sphere, and emits **spatial conditioning** for generative pipelines. Pure NumPy; **not an LLM and not a learned model** — deterministic spherical-harmonic geometry, no weights.
| Property | Choice |
|----------|--------|
| Version | `0.5.0` |
| Geometry | Ambix **ACN + SN3D** |
| Channels | 64 = (7+1)², order 7 |
| Rotation | **Wigner-D** (~150× vs dense) |
| Audio | WAV / live mic (ffmpeg·arecord) |
| Vision | Boxes / YOLO labels / optional torchvision |
| Agents | HTTP `:8765` + `spatial-report` CLI |
| Diffusion | Conditioning JSON + optional ComfyUI submit |
| Language | Python 3, NumPy only (optional torch/torchvision) |
## What it does
- **Encode** point sources / plane waves / scene mixes → HOA-7 coefficients (`encode_points`, `encode_plane_waves`, `encode_scene`).
- **Decode** coefficients → samples on the sphere, product-grid render, or a single beamform readout (`decode_directions`, `decode_grid`, `beamform`).
- **Rotate** the field in the listener frame via Wigner-D / zyz (`hoa_rotation_matrix`, `apply_hoa_rotation`, `rotate_yaw_pitch_roll`).
- **Analyze** DOA (intensity vector + peak search), directional power, field energy, per-STFT-band and per-frame reports (`analysis.py`, `report.py`).
- **Fuse vision** — object boxes / rays / YOLO detections projected onto the sphere alongside audio (`vision.py`, `detector.py`, `fuse_reports`).
- **Condition** — spatial reports → plain-text control lines for T2I/T2V prompts, structured JSON for ControlNet-style nodes, or a ComfyUI API payload (`conditioning.py`).
- **Serve** — HTTP API and iterative agent state (`server.py`, `rnn_stub.py`).
## Coordinates (always)
- **+X** front, **+Y** left, **+Z** up (Ambix listener frame).
- Azimuth 0° = front, **+90° = left**, −90° = right.
- Elevation 0° = horizon, +90° = zenith.
- W (omnidirectional HOA channel) maps to field size / POV: low W → tight / subject-focused / narrow FOV; high W → wide / environmental / immersive FOV.
## Install / run
```bash
cd spatial-hoa # this repo root
export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
python3 -m hoa64 --help
# or install the CLI on PATH:
ln -s "$PWD/scripts/spatial-report" ~/.local/bin/spatial-report
```
**Deps:** NumPy (system `ffmpeg`/`arecord` for live capture; optional torch/torchvision for the real object detector).
## CLI map (`spatial-report`)
| Command | Purpose |
|---------|---------|
| `analyze` | Ambix / mono WAV → spatial JSON |
| `demo-scene` | Synthetic multi-source audio |
| `vision` | Raw sphere boxes → report |
| `detect` | Image / YOLO / demo → report |
| `live` | Mic capture → report |
| `condition` | Report → diffusion prompt + control vector |
| `serve` | HTTP API `:8765` |
## HTTP API (`POST /v1/spatial/analyze`)
Modes: `demo_scene` · `ambix_file` · `mono_file` · `vision` · `fuse` · `detect` · `live` · `condition`
```bash
curl -s -X POST http://127.0.0.1:8765/v1/spatial/analyze \
-H 'Content-Type: application/json' \
-d '{"mode":"demo_scene","order":3}'
```
An OpenAI-compatible function schema is provided at `tools/spatial_analyze.openai.json`.
## Agent integration (Qwythos / Pi)
The HOA-7 calculator is **not** part of any LLM's weights. Agents should call it as a tool and treat the returned `one_liner` / `doa_*` / fuse fields as ground-truth geometry:
- Run the CLI: `spatial-report analyze /path/to.wav --ambix -o /tmp/spatial.json && cat /tmp/spatial.json`
- Or HTTP: `POST http://127.0.0.1:8765/v1/spatial/analyze`
- See `integrations/qwythos_system_snippet.md` for the exact agent prompt block.
## Quick test pack
```bash
python3 examples/demo_e2e_testpack.py # artifacts in /tmp/spatial_hoa_e2e/
spatial-report analyze /tmp/spatial_hoa_e2e/scene_ambix4.wav --ambix -o /tmp/a.json
spatial-report detect --demo-image /tmp/frame.png -o /tmp/v.json
spatial-report condition /tmp/spatial_hoa_e2e/fuse_report.json --prompt 'cinematic interior' -o /tmp/c.json
```
## Tests
```bash
python3 tests/test_basis.py
python3 tests/test_encode_decode.py
python3 tests/test_rotate_rnn.py
python3 tests/test_phase1_audio.py
python3 tests/test_phase2_wigner.py
python3 tests/test_phase3_vision.py
python3 tests/test_integration_extras.py
```
## Limitations
- **Deterministic geometry, not a learned model.** There are no trained weights; accuracy is bounded by the exact spherical-harmonic basis (Farina / Ambix), not by data. The `rnn_stub.py` integrator is an **explicit Euler pose/rotation loop** — learned field dynamics are not implemented yet.
- **No audio synthesis.** This decodes/analyzes spatial fields and produces conditioning; it does not generate audio content.
- **Live capture** requires system `ffmpeg`/`arecord`.
- **Object detection** is optional: the demo path uses synthetic boxes; the real detector downloads torchvision weights on first use.
- **Coordinate convention is Ambix ACN/SN3D.** Interop with other conventions (FuMa, N3D) requires explicit conversion.
## License
This repository carries no explicit license file. The code is provided as-is by the author; contact `woodfireind` for usage terms. Third-party optional deps (torch/torchvision) keep their own licenses.
|