| --- |
| license: apache-2.0 |
| task_categories: |
| - audio-to-audio |
| - text-to-audio |
| - image-to-text |
| tags: |
| - music-generation |
| - magenta |
| - magenta-rt |
| - onnx |
| - burn |
| - llama-cpp |
| - performance-rnn |
| - melody-rnn |
| - drums-rnn |
| - improv-rnn |
| - polyphony-rnn |
| - musicvae |
| - groovae |
| - piano-genie |
| - ddsp |
| - gansynth |
| - nsynth |
| - coconet |
| - music-transformer |
| - onsets-and-frames |
| - spectrostream |
| - musiccoca |
| - synesthesia |
| - directml |
| - vulkan |
| - wgpu |
| - audio |
| - midi |
| language: |
| - en |
| library_name: onnxruntime |
| base_model: |
| - unsloth/gemma-3n-E2B-it |
| - google/magenta-realtime |
| --- |
| |
| # Synesthesia — AI Music Models |
|
|
| ONNX and GGUF model weights for [Synesthesia](https://github.com/kryptodogg/synesthesia), |
| a cyber-physical synthesizer, 3D/4D signal workstation, and multi-modal music AI app. |
|
|
| Synesthesia brings together every open-weights model from **Magenta Classic** and |
| **Magenta RT** under one repo, exportable to ONNX for local inference and continuously |
| fine-tunable via free Google Colab notebooks. |
|
|
| --- |
|
|
| ## Inference Runtimes |
|
|
| | Runtime | Models | Backend | Notes | |
| |---------|--------|---------|-------| |
| | **Burn wgpu** | DDSP, GANSynth, NSynth, Piano Genie | Vulkan / DX12 | Pure Rust, no ROCm required | |
| | **ORT + DirectML** | RNN family, MusicVAE, Coconet, Onsets & Frames | DirectML | Fallback while Burn op coverage matures | |
| | **llama.cpp + Vulkan** | Gemma-3N | Vulkan | Same stack as LM Studio, GGUF format | |
| | **Magenta RT (JAX)** | Magenta RT LLM, SpectroStream, MusicCoCa | TPU / GPU | Free Colab TPU v2-8 for inference + finetuning | |
|
|
| Vulkan works on AMD without ROCm on Windows 11. All runtimes target the RX 6700 XT. |
|
|
| --- |
|
|
| ## Model Inventory |
|
|
| ### Magenta RT (Real-Time Audio Generation) |
|
|
| Magenta RT is composed of three components working as a pipeline: |
| SpectroStream (audio codec), MusicCoCa (style embeddings), and an encoder-decoder |
| transformer LLM — the only open-weights model supporting real-time continuous |
| musical audio generation. |
|
|
| It is an 800 million parameter autoregressive transformer trained on |
| ~190k hours of stock music. It uses 38% fewer parameters |
| than Stable Audio Open and 77% fewer than MusicGen Large. |
|
|
| | ID | Model | Format | Task | Synesthesia Role | |
| |----|-------|--------|------|-----------------| |
| | MRT-001 | Magenta RT LLM | JAX / ONNX | Real-time stereo audio generation | Continuous live generation engine | |
| | MRT-002 | SpectroStream Encoder | ONNX | Audio → discrete tokens (48kHz stereo, 25Hz, 64 RVQ) | Audio tokenizer | |
| | MRT-003 | SpectroStream Decoder | ONNX | Tokens → 48kHz stereo audio | Audio detokenizer | |
| | MRT-004 | MusicCoCa Text | ONNX | Text → 768-dim music embedding | Text prompt → style control | |
| | MRT-005 | MusicCoCa Audio | ONNX | Audio → 768-dim music embedding | Audio prompt → style control | |
|
|
| **Finetuning:** Free Colab TPU v2-8 via `Magenta_RT_Finetune.ipynb`. Customize to |
| your own audio catalog. Official Colab demos support live generation, |
| finetuning, and live audio injection (audio injection = mix user audio with model |
| output and feed as context for next generation chunk). |
|
|
| --- |
|
|
| ### Magenta Classic — MIDI / Symbolic |
|
|
| MusicRNN implements Magenta's LSTM-based language models: |
| MelodyRNN, DrumsRNN, ImprovRNN, and PerformanceRNN. |
|
|
| | ID | Model | Format | Task | Synesthesia Role | |
| |----|-------|--------|------|-----------------| |
| | MC-001 | Performance RNN | ONNX | Expressive MIDI performance generation | AI arpeggiator, live note generation | |
| | MC-002 | Melody RNN | ONNX | Melody continuation (LSTM) | Melody continuation tool | |
| | MC-003 | Drums RNN | ONNX | Drum pattern generation (LSTM) | Beat generation | |
| | MC-004 | Improv RNN | ONNX | Chord-conditioned melody generation | Live improv over chord progressions | |
| | MC-005 | Polyphony RNN | ONNX | Polyphonic music generation (BachBot) | Harmonic voice generation | |
| | MC-006 | MusicVAE | ONNX enc+dec | Latent music VAE — melody, drum, trio loops | Latent interpolation, style morphing | |
| | MC-007 | GrooVAE | ONNX enc+dec | Drum performance humanization | Humanize MIDI drums | |
| | MC-008 | MidiMe | ONNX | Personalize MusicVAE in-session | User-adaptive latent space | |
| | MC-009 | Music Transformer | ONNX | Long-form piano generation | Extended composition | |
| | MC-010 | Coconet | ONNX | Counterpoint by convolution — complete partial scores | Harmony / counterpoint filler | |
|
|
| --- |
|
|
| ### Magenta Classic — Audio / Timbre |
|
|
| | ID | Model | Format | Task | Synesthesia Role | |
| |----|-------|--------|------|-----------------| |
| | MA-001 | GANSynth | ONNX | GAN audio synthesis from NSynth timbres | GANHarp-style timbre instrument | |
| | MA-002 | NSynth | ONNX | WaveNet neural audio synthesis | Sample-level timbre generation | |
| | MA-003 | DDSP Encoder | ONNX | Audio → harmonic + noise params | Timbre analysis | |
| | MA-004 | DDSP Decoder | ONNX | Harmonic params → audio | Timbre resynthesis | |
| | MA-005 | Piano Genie | ONNX | 8-button → 88-key piano VQ-VAE | Accessible piano performance | |
| | MA-006 | Onsets and Frames | ONNX | Polyphonic piano transcription (audio → MIDI) | Audio → MIDI transcription | |
| | MA-007 | SPICE | ONNX | Pitch extraction from audio | Monophonic pitch tracking | |
|
|
| --- |
|
|
| ### LLM / Vision Control |
|
|
| | ID | Model | Format | Task | Synesthesia Role | |
| |----|-------|--------|------|-----------------| |
| | LV-001 | Gemma-3N e2b-it | GGUF | Vision + text → structured JSON | Camera → mood/energy/key control | |
|
|
| **Format tiers:** |
| - `q4_k_m.gguf` — default (recommended, ~1.5GB) |
| - `q2_k.gguf` — lite tier (fastest, smallest) |
| - `f16.gguf` — full quality reference |
|
|
| **Runtime:** `llama-cpp-v3` Rust crate with Vulkan backend. |
| Same stack as LM Studio — no ROCm, no CUDA needed on Windows. |
|
|
| --- |
|
|
| ## Repository Structure |
|
|
| ``` |
| Ashiedu/Synesthesia/ |
| │ |
| ├── manifest.json ← authoritative model registry |
| │ |
| ├── magenta_rt/ |
| │ ├── llm/ ← MRT-001: JAX checkpoint + ONNX export |
| │ ├── spectrostream/ |
| │ │ ├── encoder_fp32.onnx |
| │ │ ├── encoder_fp16.onnx |
| │ │ ├── decoder_fp32.onnx |
| │ │ └── decoder_fp16.onnx |
| │ └── musiccoca/ |
| │ ├── text_fp32.onnx |
| │ ├── text_fp16.onnx |
| │ ├── audio_fp32.onnx |
| │ └── audio_fp16.onnx |
| │ |
| ├── midi/ |
| │ ├── perfrnn/ ← MC-001: fp32 / fp16 / int8 |
| │ ├── melody_rnn/ ← MC-002 |
| │ ├── drums_rnn/ ← MC-003 |
| │ ├── improv_rnn/ ← MC-004 |
| │ ├── polyphony_rnn/ ← MC-005 |
| │ ├── musicvae/ ← MC-006: encoder + decoder |
| │ ├── groovae/ ← MC-007 |
| │ ├── midime/ ← MC-008 |
| │ ├── music_transformer/ ← MC-009 |
| │ └── coconet/ ← MC-010 |
| │ |
| ├── audio/ |
| │ ├── gansynth/ ← MA-001: fp32 / fp16 |
| │ ├── nsynth/ ← MA-002 |
| │ ├── ddsp/ ← MA-003+004: encoder + decoder |
| │ ├── piano_genie/ ← MA-005 |
| │ ├── onsets_and_frames/ ← MA-006 |
| │ └── spice/ ← MA-007 |
| │ |
| └── llm/ |
| └── gemma3n_e2b/ |
| ├── q4_k_m.gguf ← LV-001: default |
| ├── q2_k.gguf |
| └── f16.gguf |
| ``` |
|
|
| Each subdirectory contains a `README.md` with input/output shapes, |
| export commands, and Burn compatibility status. |
|
|
| --- |
|
|
| ## Quality Tiers (ONNX models) |
|
|
| | Tier | Suffix | VRAM est. | Use case | |
| |------|--------|-----------|----------| |
| | Full | `_fp32.onnx` | ~2–4× Half | Reference quality, CI validation | |
| | **Half** | `_fp16.onnx` | Baseline | **Default — recommended for RX 6700 XT** | |
| | Lite | `_int8.onnx` | ~0.5× Half | Lowest latency (MIDI models only) | |
|
|
| --- |
|
|
| ## Pulling Models in Rust |
|
|
| ```rust |
| use hf_hub::api::sync::Api; |
| |
| pub fn pull(repo_path: &str) -> anyhow::Result<std::path::PathBuf> { |
| let api = Api::new()?; |
| let repo = api.model("Ashiedu/Synesthesia".to_string()); |
| Ok(repo.get(repo_path)?) |
| // Cached: ~/.cache/huggingface/hub/ |
| } |
| |
| // Example |
| let path = pull("midi/perfrnn/fp16.onnx")?; |
| ``` |
|
|
| ## Pulling Models in Python |
|
|
| ```python |
| from huggingface_hub import snapshot_download, hf_hub_download |
| |
| # Pull everything |
| snapshot_download("Ashiedu/Synesthesia", local_dir="./models") |
| |
| # Pull one file |
| hf_hub_download( |
| repo_id="Ashiedu/Synesthesia", |
| filename="midi/perfrnn/fp16.onnx", |
| local_dir="./models", |
| ) |
| ``` |
|
|
| --- |
|
|
| ## Export Workflow (Colab) |
|
|
| All models are exported from Colab and pushed here. The generic workflow: |
|
|
| ```python |
| # 1. Pull existing checkpoint (if updating) |
| from huggingface_hub import snapshot_download |
| snapshot_download("Ashiedu/Synesthesia", local_dir="./models", token=HF_TOKEN) |
| |
| # 2. Clone Magenta source |
| # !git clone https://github.com/magenta/magenta |
| # !git clone https://github.com/magenta/magenta-realtime |
| |
| # 3. Export to ONNX (varies per model — see each model's README) |
| # Magenta Classic: tf2onnx |
| # Magenta RT: JAX → onnx via jax2onnx or flax export |
| # Gemma-3N: Unsloth → GGUF |
| |
| # 4. Quantize |
| from onnxruntime.quantization import quantize_dynamic, QuantType |
| import onnxconverter_common as occ, onnx |
| |
| fp32 = onnx.load("model.onnx") |
| fp16 = occ.convert_float_to_float16(fp32, keep_io_types=True) |
| onnx.save(fp16, "model_fp16.onnx") |
| quantize_dynamic("model.onnx", "model_int8.onnx", weight_type=QuantType.QInt8) |
| |
| # 5. Push to HF |
| from huggingface_hub import HfApi |
| api = HfApi(token=HF_TOKEN) # set in Colab Secrets |
| api.upload_file( |
| path_or_fileobj="model_fp16.onnx", |
| path_in_repo="midi/perfrnn/fp16.onnx", |
| repo_id="Ashiedu/Synesthesia", |
| commit_message="MC-001 Performance RNN fp16", |
| ) |
| ``` |
|
|
| **Gemini on Colab:** Point Gemini at this README and the model's subdirectory |
| README as context. Gemini can execute the export + push workflow without |
| GitHub integration — it only needs Python and your HF token in Colab Secrets. |
|
|
| --- |
|
|
| ## Burn Compatibility Tracking |
|
|
| CI weekly attempts `burn-onnx ModelGen` on each exported model. |
| Models migrate from ORT fallback to Burn as op coverage matures. |
|
|
| | Model | Burn target | ORT fallback | Last checked | |
| |-------|------------|--------------|-------------| |
| | DDSP enc/dec | ✅ | ❌ | — | |
| | GANSynth | ✅ | ❌ | — | |
| | NSynth | ✅ | ❌ | — | |
| | Piano Genie | ✅ | ❌ | — | |
| | Performance RNN | 🔄 LSTM | ✅ | — | |
| | Melody RNN | 🔄 LSTM | ✅ | — | |
| | Drums RNN | 🔄 LSTM | ✅ | — | |
| | Improv RNN | 🔄 LSTM | ✅ | — | |
| | Polyphony RNN | 🔄 LSTM | ✅ | — | |
| | MusicVAE | 🔄 BiLSTM | ✅ | — | |
| | Coconet | 🔄 Conv | ✅ | — | |
| | Music Transformer | 🔄 Attention | ✅ | — | |
| | Onsets & Frames | 🔄 Conv+LSTM | ✅ | — | |
| | SpectroStream | 🔄 Conv | ✅ | — | |
| | MusicCoCa | 🔄 ViT+Transformer | ✅ | — | |
| | Gemma-3N | N/A — llama.cpp | ❌ | — | |
|
|
| --- |
|
|
| ## Training Philosophy |
|
|
| **Train after the app works.** The interface ships first. Training data |
| is determined by what the working app actually receives as input in practice. |
| Fine-tune on your own audio and MIDI once the signal chain is wired. |
|
|
| Tentative fine-tuning order once the app is functional: |
| 1. Performance RNN — live MIDI from the Track Mixer |
| 2. MusicVAE / GrooVAE — latent interpolation between patches |
| 3. GANSynth — timbre generation from pitch + latent input |
| 4. DDSP — resynthesis of GANSynth outputs |
| 5. Magenta RT — full audio, conditioned on your own catalog |
| 6. Gemma-3N — camera → mood/energy trained on your session recordings |
|
|
| --- |
|
|
| ## License |
|
|
| - Codebase: Apache 2.0 |
| - Magenta Classic weights: Apache 2.0 |
| - Magenta RT weights: Apache 2.0 with additional [bespoke terms](https://github.com/magenta/magenta-realtime/blob/main/LICENSE) |
| - Gemma-3N: [Gemma Terms of Use](https://ai.google.dev/gemma/terms) |
|
|
| Individual model directories note any additional upstream license terms. |
|
|
| --- |
|
|
| ## Links |
|
|
| - **App:** [kryptodogg/synesthesia](https://github.com/kryptodogg/synesthesia) |
| - **Magenta RT:** [magenta/magenta-realtime](https://github.com/magenta/magenta-realtime) |
| - **Magenta Classic:** [magenta/magenta](https://github.com/magenta/magenta) |
| - **HF Model Card:** [google/magenta-realtime](https://huggingface.co/google/magenta-realtime) |
| - **Roadmap:** GitHub Issues — `lane:ml` label |
