Update README.md

README.md

@@ -1,5 +1,277 @@
----
-license: other
-license_name: collaborative-intelligence-license
-license_link: LICENSE
----
+license: collaborative-intelligence
+language:
+  - en
+tags:
+  - music
+  - audio
+  - midi
+  - onnx
+  - multimodal
+  - music-generation
+  - evolutionary-music
+  - chromatic-modes
+datasets:
+  - earthlyframes/white-training-data
+metrics:
+  - accuracy
+library_name: onnxruntime
+pipeline_tag: audio-classification
+---
+
+# Refractor
+
+Refractor is a multimodal fitness function for evolutionary music composition. It takes up to five input modalities — MIDI piano roll, audio embedding, concept text, lyric text, and artist "sounds-like" descriptions — and scores a piece of music against a chromatic concept, classifying it across three independent mode dimensions: **temporal**, **spatial**, and **ontological**.
+
+It is the scoring engine at the heart of the [White](https://github.com/brotherclone/white) AI-assisted album production system.
+
+## Model Details
+
+### What are chromatic modes?
+
+The White project encodes musical character using a colour-theory system. Each colour (Red, Orange, Yellow, Green, Blue, Indigo, Violet) maps to a unique combination of three independent categorical dimensions:
+
+| Dimension | Classes | Example |
+|-----------|---------|---------|
+| **Temporal** | Past · Present · Future | Red → Past |
+| **Spatial** | Thing · Place · Person | Red → Thing |
+| **Ontological** | Known · Imagined · Forgotten | Red → Known |
+
+Refractor learns to predict which cell in this 3×3×3 space a piece of music occupies, and how confidently it does so.
+
+### Architecture
+
+```
+Inputs
+  piano_roll       [B, 1, 128, 256]   — MIDI as a piano roll image
+  audio_emb        [B, 512]           — CLAP audio embedding
+  concept_emb      [B, 768]           — DeBERTa-v3-base concept text embedding
+  lyric_emb        [B, 768]           — DeBERTa-v3-base lyric text embedding
+  sounds_like_emb  [B, 768]           — DeBERTa-v3-base mean-pooled artist descriptions
+  has_audio        [B]                — bool mask
+  has_midi         [B]                — bool mask
+  has_lyric        [B]                — bool mask
+  has_sounds_like  [B]                — bool mask
+
+PianoRollEncoder (CNN)
+  Conv2d(1→32) → BN → ReLU → MaxPool2d
+  Conv2d(32→64) → BN → ReLU → MaxPool2d
+  Conv2d(64→128) → BN → ReLU → AdaptiveAvgPool2d(4,4)
+  Linear(2048→512) → ReLU
+  → midi_emb [B, 512]
+
+Fusion MLP
+  cat([audio 512, midi 512, concept 768, lyric 768, sounds_like 768]) = [B, 3328]
+  Linear(3328→1024) → ReLU → Dropout(0.3)
+  Linear(1024→512)  → ReLU → Dropout(0.2)
+  → fused [B, 512]
+
+Heads
+  temporal_head    Linear(512→3)  → Softmax
+  spatial_head     Linear(512→3)  → Softmax
+  ontological_head Linear(512→3)  → Softmax
+  confidence_head  Linear(512→1)  → Sigmoid
+
+Total parameters: 5,084,362
+  CNN encoder:      1,142,208
+  Fusion + heads:   3,942,154
+```
+
+Absent modalities are handled via **learned null embeddings** (one per modality, trained end-to-end). During training, **modality dropout** (p=0.15) randomly masks present modalities, forcing the model to be robust to any combination of available inputs. At inference, dropout is disabled and the null path is used for any missing modality.
+
+### Model Details
+
+- **Developed by:** Gabriel Walsh (brotherclone)
+- **Model type:** Multimodal classification (ONNX)
+- **License:** Collaborative Intelligence
+- **Repository:** [brotherclone/white](https://github.com/brotherclone/white)
+- **Training dataset:** [earthlyframes/white-training-data](https://huggingface.co/datasets/earthlyframes/white-training-data)
+
+## Uses
+
+### Primary use — evolutionary music composition
+
+Refractor is the fitness function in an evolutionary pipeline that generates music structured around chromatic concepts:
+
+1. A colour concept is selected (e.g. **Red** → `temporal=Past, spatial=Thing, ontological=Known`)
+2. ~50 MIDI candidates are generated (chord progressions, drum patterns, bass lines, melodies)
+3. Refractor scores each candidate against the concept embedding
+4. Candidates are ranked by `confidence`; low scorers are pruned
+5. The surviving candidates are promoted and the next generation begins
+
+```python
+from training.refractor import Refractor
+
+scorer = Refractor()  # loads refractor.onnx (~19 MB)
+
+# Encode concept once, reuse across the whole evolutionary batch
+concept_emb = scorer.prepare_concept(
+    "RED temporal=Past spatial=Thing ontological=Known"
+)
+
+# Score a single MIDI candidate
+result = scorer.score(midi_bytes=midi_data, concept_emb=concept_emb)
+# → {
+#     "temporal":    {"past": 0.89, "present": 0.07, "future": 0.04},
+#     "spatial":     {"thing": 0.91, "place": 0.05, "person": 0.04},
+#     "ontological": {"known": 0.88, "imagined": 0.07, "forgotten": 0.05},
+#     "confidence":  0.87
+#   }
+
+# Score a batch of 50 candidates (single ONNX call)
+candidates = [{"midi_bytes": m} for m in midi_variants]
+ranked = scorer.score_batch(candidates, concept_emb=concept_emb)
+# → list sorted by confidence descending, each with rank + original candidate
+```
+
+### With sounds-like context
+
+If you have artist aesthetic descriptions for the target sound, pass them to further condition the score:
+
+```python
+sounds_like = [
+    "Motorik rhythms, kosmische synthesizer textures, hypnotic repetition",
+    "Driving post-punk guitars, angular riffs, sardonic delivery",
+]
+result = scorer.score(
+    midi_bytes=midi_data,
+    concept_emb=concept_emb,
+    sounds_like_texts=sounds_like,
+)
+```
+
+Or pre-compute the embedding once and reuse across a batch:
+
+```python
+sl_emb = scorer.prepare_sounds_like(sounds_like)
+ranked = scorer.score_batch(candidates, concept_emb=concept_emb, sounds_like_emb=sl_emb)
+```
+
+### Using ONNX directly
+
+```python
+import onnxruntime as ort
+import numpy as np
+
+sess = ort.InferenceSession("refractor.onnx", providers=["CPUExecutionProvider"])
+
+feed = {
+    "piano_roll":       np.zeros((1, 1, 128, 256), dtype=np.float32),
+    "audio_emb":        np.zeros((1, 512),          dtype=np.float32),
+    "concept_emb":      concept_vec.reshape(1, 768),
+    "lyric_emb":        np.zeros((1, 768),          dtype=np.float32),
+    "sounds_like_emb":  sl_vec.reshape(1, 768),
+    "has_audio":        np.array([False]),
+    "has_midi":         np.array([False]),
+    "has_lyric":        np.array([False]),
+    "has_sounds_like":  np.array([True]),
+}
+temporal, spatial, ontological, confidence = sess.run(None, feed)
+```
+
+### Out-of-scope use
+
+- General-purpose music genre or mood classification (this model is calibrated to the White colour-theory system, not universal taxonomies)
+- Real-time inference on audio streams (designed for batch scoring of pre-rendered candidates)
+- Replacement for human artistic judgement (scores are a compositional signal, not ground truth)
+
+## Training Details
+
+### Training data
+
+[earthlyframes/white-training-data](https://huggingface.co/datasets/earthlyframes/white-training-data) v0.2.0
+
+- **11,605 segments** across **83 songs**, all **8 chromatic colours**
+- Audio coverage: 85.4% (9,907 segments with CLAP embeddings)
+- MIDI coverage: 44.3% (5,145 segments with piano rolls)
+- Lyric coverage: 92.7% (10,764 segments with DeBERTa lyric embeddings)
+- Sounds-like coverage: 100% (11,605 segments, 237 artists, song-level signal broadcast to segments)
+
+Labels are derived from per-song colour assignments in the White album metadata. The `None` class (3,154 segments) covers unlabelled or transitional segments and is excluded from accuracy calculations.
+
+### Preprocessing
+
+- **MIDI → piano roll**: `pretty_midi`, quantised to 128 pitches × 256 time steps, velocity-normalised to [0, 1]
+- **Audio → embedding**: [laion/larger_clap_music](https://huggingface.co/laion/larger_clap_music), 512-dim
+- **Text → embedding**: [microsoft/deberta-v3-base](https://huggingface.co/microsoft/deberta-v3-base), mean-pooled CLS token, 768-dim; applied to concept strings, lyric text, and artist description lists
+- **Sounds-like**: per-song artist descriptions mean-pooled to a single 768-dim vector, broadcast to all segments of that song
+
+### Training procedure
+
+Phase 5 fine-tunes from a Phase 3 checkpoint (audio + MIDI + concept + lyric, 2560-dim fusion) by re-initialising the first fusion layer for the expanded 3328-dim input and loading all other weights.
+
+- **Hardware:** NVIDIA A10 (23.7 GB VRAM) via Modal
+- **Epochs:** 30 (early stopping, patience=10)
+- **Best checkpoint:** epoch 14
+- **Optimizer:** AdamW, lr=1e-5 → 5e-6 (cosine decay)
+- **Batch size:** 32
+- **Label smoothing:** 0.1
+- **Modality dropout:** p=0.15 per modality during training
+- **Model selection criterion:** best mean accuracy across temporal + spatial + ontological (not val loss — loss plateaus at ~0.0002–0.0003 during fine-tuning while accuracy varies ±15%)
+
+## Evaluation
+
+### Results
+
+Evaluated on a held-out 20% split (2,321 segments), excluding `None`-labelled segments.
+
+| Dimension | Accuracy |
+|-----------|----------|
+| Temporal  | 89.3%    |
+| Spatial   | 91.6%    |
+| Ontological | 90.7%  |
+| **Mean**  | **90.5%** |
+| Confidence (sigmoid) | ~0.87 at target match |
+
+The spatial dimension historically lagged (62% in text-only Phase 4) because instrumental tracks have no lyric signal and spatial mode correlates strongly with vocal character. Adding MIDI piano rolls in Phase 3 closed the gap to 93%; the sounds-like modality further stabilises scores on instrumental passages.
+
+### Limitations
+
+- Chromatic mode labels are derived from a single artistic framework (the White project). Scores are only meaningful relative to that framework's colour → mode mapping.
+- The confidence head is a sigmoid over a single logit, not a calibrated probability. Use it for relative ranking within a batch, not as an absolute reliability score.
+- MIDI coverage is 44% of the training data; piano-roll features have weaker gradients than the text/audio paths on segments without MIDI.
+- Sounds-like embeddings are song-level averages — they cannot distinguish between sections of the same song that have different timbral character.
+
+## Technical Specifications
+
+### Compute infrastructure
+
+- Training: Modal (cloud), NVIDIA A10 GPU
+- Inference: CPU only (`CPUExecutionProvider`), tested on Apple M-series and x86 Linux
+- ONNX opset: 17
+- Inference time: ~4 ms per batch of 50 on M2 MacBook Pro (MIDI-only, no CLAP)
+
+### Software
+
+- PyTorch 2.x (training)
+- ONNX opset 17 (export)
+- onnxruntime ≥ 1.17 (inference)
+- transformers ≥ 4.40 (DeBERTa / CLAP encoders, lazy-loaded at runtime)
+- pretty_midi (piano roll preprocessing)
+
+### Files
+
+| File | Size | Description |
+|------|------|-------------|
+| `refractor.onnx` | 19.4 MB | ONNX model (all 9 inputs) |
+| `refractor.pt` | 19.4 MB | PyTorch checkpoint |
+
+## Citation
+
+```bibtex
+@misc{walsh2026refractor,
+  author       = {Gabriel Walsh},
+  title        = {Refractor: A Multimodal Fitness Function for Chromatic Music Composition},
+  year         = {2026},
+  publisher    = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/earthlyframes/refractor}},
+  note         = {Part of the White project: \url{https://github.com/brotherclone/white}}
+}
+```
+
+## Glossary
+
+- **Chromatic modes**: The three classification dimensions (temporal, spatial, ontological) derived from the White colour-theory system for music
+- **Null embedding**: A learned parameter vector substituted for any absent modality at inference time
+- **Modality dropout**: Training-time regularisation that randomly masks present modalities, making the model robust to missing inputs
+- **Confidence**: A sigmoid scalar [0, 1] indicating how strongly the fused representation matches the target chromatic concept
+- **Sounds-like**: Song-level aesthetic descriptions of reference artists, mean-pooled into a 768-dim conditioning vector