---
title: MusicSampler
emoji: 🎹
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 4.44.1
app_file: app.py
pinned: false
license: mit
short_description: Modular AudioGenerator for DAW-INVADER
---

# TransformerPrime Text-to-Audio Pipeline

**One-sentence justification:** We use the Hugging Face `text-to-audio` pipeline with **sesame/csm-1b** (1B-param conversational TTS, Llama backbone, voice cloning) as the default; it is natively supported in transformers, fits in **<8–10 GB VRAM** with 4-bit quantization, and supports `device_map="auto"`, bfloat16, and optional Flash Attention 2 for fast inference on RTX 40/50 and A100/H100/B200.

---

## Pipeline flow (ASCII)

```
┌─────────────┐     ┌──────────────────┐     ┌─────────────────────────────────────┐
│   Text      │────▶│  HF Processor /  │────▶│  AutoModelForTextToWaveform          │
│   input     │     │  Tokenizer       │     │  (CSM/Bark/SpeechT5/MusicGen/…)       │
└─────────────┘     └──────────────────┘     │  • device_map="auto"                  │
                                            │  • torch.bfloat16                     │
                                            │  • attn_implementation=flash_attn_2   │
                                            │  • optional: 4-bit (bitsandbytes)     │
                                            └──────────────────┬───────────────────┘
                                                               │
                    ┌──────────────────────────────────────────┘
                    ▼
         ┌──────────────────────┐     ┌─────────────────────┐
         │ Spectrogram models    │────▶│ Vocoder (HiFi-GAN)  │
         │ (e.g. SpeechT5)       │     │ → waveform          │
         └──────────────────────┘     └──────────┬──────────┘
                    │                             │
                    │  Waveform models (CSM, Bark, MusicGen)
                    └─────────────────────────────┼─────────────┐
                                                  ▼             │
                                         ┌───────────────┐      │
                                         │  Waveform     │◀─────┘
                                         │  (numpy/CPU)  │
                                         └───────┬───────┘
                                                 │
                    ┌────────────────────────────┼────────────────────────────┐
                    ▼                            ▼                            ▼
             stream_chunks()              save WAV (soundfile)         Gradio / CLI
             (fixed-duration slices)      + memory profile             demo
```

---

## Install

```bash
cd c:\Users\Keith\transformerprime
pip install -r requirements.txt
```

Optional: `pip install bitsandbytes` for 4-bit/8-bit (keeps VRAM <10 GB on 1B models).

Run tests (from repo root): `PYTHONPATH=. pytest tests/`

---

## Usage

### Python API

```python
from src.text_to_audio import build_pipeline, TextToAudioPipeline
import soundfile as sf

# Default: CSM-1B, bfloat16, device_map="auto", Flash Attention 2 when supported
pipe = build_pipeline(preset="csm-1b")

# Low VRAM: 4-bit quantization
pipe_low = build_pipeline(preset="csm-1b", use_4bit=True)

# Generate and profile (time, RTF, VRAM peak)
out, profile = pipe.generate_with_profile("Hello from TransformerPrime. This runs on GPU.")
sf.write("out.wav", out["audio"], out["sampling_rate"])
print(profile)  # {"time_s": ..., "rtf": ..., "vram_peak_mb": ..., "duration_s": ...}

# Streaming-style chunks (post-generation chunking for playback)
for chunk, sr in pipe.stream_chunks("Long text here.", chunk_duration_s=0.5):
    pass  # play or write chunk
```

### CLI

```bash
python demo.py --cli --text "Your sentence here." --output out.wav
python demo.py --cli --model bark-small --quantize --output bark.wav
```

### Gradio

```bash
python demo.py --model csm-1b
# Open http://localhost:7860
```

---

## Expected performance (typical GPU)

| Preset       | GPU (example) | dtype    | VRAM (approx) | RTF (approx) | Note                    |
|-------------|---------------|----------|----------------|--------------|-------------------------|
| csm-1b      | RTX 4090     | bfloat16 | ~8–12 GB      | 0.1–0.3      | 1B, voice cloning       |
| csm-1b      | RTX 4090     | 4-bit    | ~6–8 GB       | 0.15–0.4     | Same, lower VRAM        |
| bark-small  | RTX 4070     | float32  | ~4–6 GB       | 0.2–0.5      | Smaller, multilingual   |
| speecht5    | RTX 4060     | float32  | ~2–3 GB       | <0.2        | Spectrogram + vocoder   |
| musicgen-small | RTX 4090  | float32  | ~8–10 GB      | 0.3–0.6      | Music/sfx               |

*RTF = real-time factor (wall time / audio duration; <1 = faster than real time).*

---

## Tuning tips

- **Batch inference:** Pass a list of strings: `pipe.generate(["Text one.", "Text two."])`. Watch VRAM; reduce batch size or use 4-bit if OOM.
- **Longer generation:** Increase `max_new_tokens` in `generate_kwargs` (e.g. `pipe.generate(text, generate_kwargs={"max_new_tokens": 512})`). CSM/Bark/MusicGen use this; exact cap is model-specific.
- **Emotion / style:**  
  - **CSM:** Use chat template with reference audio for voice cloning and tone.  
  - **Bark:** Use semantic tokens / history if you customize inputs.  
  - **Dia (nari-labs/Dia-1.6B-0626):** Speaker tags `[S1]`/`[S2]` and non-verbal cues like `(laughs)`.
- **Stability:** Prefer `torch.bfloat16` on Ampere+; use `use_4bit=True` to stay under 8–10 GB VRAM.
- **CPU fallback:** Omit `device_map="auto"` and set `device=-1` in the pipeline if you have no GPU; generation will be slow.

---

## Error handling and memory

- **OOM:** Enable 4-bit (`use_4bit=True`) or switch to a smaller preset (e.g. `bark-small`, `speecht5`).
- **Flash Attention 2:** If loading fails with an attn backend error, the code falls back to the default attention implementation for that model.
- **VRAM check:** `generate_with_profile()` returns `vram_peak_mb` when CUDA is available; use it to size batches and model.

---

## Presets

| Preset         | Model ID              | Description                          |
|----------------|------------------------|--------------------------------------|
| csm-1b         | sesame/csm-1b         | 1B conversational TTS, voice cloning |
| bark-small     | suno/bark-small       | Multilingual, non-verbal             |
| speecht5       | microsoft/speecht5_tts| Multi-speaker (x-vector)             |
| musicgen-small | facebook/musicgen-small | Music / SFX                        |

You can pass any `model_id` supported by `transformers` text-to-audio pipeline via `build_pipeline(model_id="org/model-name")`.

---

## Code layout

- `src/text_to_audio/pipeline.py` — Pipeline wrapper, GPU opts, streaming chunks, memory profiling.
- `src/text_to_audio/__init__.py` — Exposes `build_pipeline`, `TextToAudioPipeline`, `list_presets`.
- `demo.py` — Gradio UI and CLI; `python demo.py` (Gradio) or `python demo.py --cli --text "..." --output out.wav`.
- `tests/test_pipeline.py` — Unit tests (no model download); run after `pip install -r requirements.txt`.