README.md

---
license: apache-2.0
language:
  - en
  - es
  - it
  - fr
  - de
  - vi
library_name: onnxruntime
tags:
  - text-to-speech
  - tts
  - onnx
  - coreai
  - pytorch
  - speech-synthesis
  - duration-flow
  - multilingual
  - expressive-tts
  - emotion
  - whisper
pipeline_tag: text-to-speech
---

# Scylla's Band v2

Scylla's Band v2 is a multilingual, multi-voice, expressive text-to-speech
model for local and self-hosted inference. It predicts phone durations,
generates continuous acoustic latents with rectified flow, and decodes those
latents to a 24 kHz waveform through a learned acoustic adapter and a frozen
Vocos waveform decoder.

The v2 release expands language and delivery coverage while keeping the same
managed ten-voice identity set. Its two primary runtime targets are ONNX
Runtime for general desktop, server, and native use, and Apple Core AI for
native Apple Silicon execution on iOS 27 and macOS 27.

Public resources:

- V2 model bundles and PyTorch checkpoints:
  [`spybyscript/scyllasbandv2`](https://huggingface.co/spybyscript/scyllasbandv2)
- Runtime source: [`lowkeytea/scyllasband`](https://github.com/lowkeytea/scyllasband)
- [Interactive voice, language, and affect samples](https://lowkeytea.github.io/scyllasband/)
- [Scylla's Band Discord](https://discord.gg/cNdBuM3tS)

The Hugging Face repository contains both deployable inference bundles and the
corresponding `.pt` checkpoints. Training data is not distributed.

## What Changed in V2

- French, German, and Vietnamese join English, Spanish, and Italian.
- The public language IDs are `en_us`, `en_gb`, `es`, `it`, `fr`, `de`, and
  `vi`.
- Affect axis-order version 2 replaces the v1 `questioning` overlay with a
  trainable `whisper` overlay.
- The six-axis affect contract is explicitly versioned in every bundle, so a
  v1 `questioning` value cannot be interpreted as v2 `whisper` merely because
  both occupy the sixth tensor position.
- Runtime text normalization is expanded for French, German, and Vietnamese,
  including language-specific number and punctuation handling.
- English dialect ownership is explicit. Ink, Orpheus, and Tuesday use
  `en_gb`; the other managed voices use `en_us`. Audio trained as British
  English remains labeled `en_gb` rather than being folded into `en_us`.
- Duration targets use alignment-derived timing tied to the actual audio
  files. The v2 acoustic model is trained from the aligned/eSpeak phone
  representation; the shipped phrase-level G2P is the runtime frontend, not a
  source of acoustic-training labels.
- Training includes long and chunked views, explicit punctuation silences,
  three-segment span context, stronger condition dropout, and a vocoder adapter
  trained on both oracle and generated acoustic latents.
- The model repository now ships PyTorch checkpoints with the inference
  bundles instead of placing them in a separate training-model repository.
- Apple Core AI is a primary native backend alongside ONNX. Core ML is not a
  supported v2 runtime backend.

## Intended Use

Scylla's Band v2 is intended for:

- Speech synthesis with ten managed voices.
- American or British English, Spanish, Italian, French, German, and
  Vietnamese synthesis, subject to each voice's manifest-declared English
  dialect.
- Long-form narration with automatic planning and chunking.
- Multi-voice dialogue from tagged text.
- Continuous and mixed control over calm, joy, anger, sadness, sarcasm, and
  whisper delivery.
- Cross-platform inference through ONNX Runtime.
- Native Apple Silicon inference through Core AI on iOS 27 or macOS 27.
- Research, inspection, and re-export from the included PyTorch checkpoints.

Scylla's Band is not an arbitrary-speaker cloning system. It is not intended
for impersonation, fraud, deception, or generating speech that falsely
represents a real person as speaking.

## Quick Start

Install the public runtime, then explicitly download from the v2 repository:

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip wheel setuptools
pip install -e .
pip install numpy huggingface_hub onnxruntime

python -m scyllasband download \
    --repo-id spybyscript/scyllasbandv2 \
    --runtime-bundles onnx \
    --yes
python -m scyllasband validate-bundle scyllasband/models/onnx
python -m scyllasband speak scyllasband/models/onnx \
    --backend onnx \
    --voice scylla \
    --language en_us \
    --emotion calm=0.5 \
    -o hello.wav \
    "Hello from Scylla's Band v2."
```

An expressive whisper example:

```bash
python -m scyllasband speak scyllasband/models/onnx \
    --backend onnx \
    --voice ink \
    --language en_gb \
    --emotion calm=0.25,whisper=0.8 \
    --emotion-scale 1.25 \
    --steps 8 \
    --sampler heun \
    -o whispered.wav \
    "Keep your voice down. Someone is still in the corridor."
```

The selected bundle's `manifest.json` is authoritative for available bundle
variants, voices, languages, affect axes, shapes, and backend defaults.

## Model Details

| Field | Value |
| --- | --- |
| Model family | Continuous-latent duration/flow TTS |
| Public model version | `2` |
| Primary portable backend | ONNX Runtime |
| Primary Apple backend | Core AI on iOS/macOS 27 |
| License | Apache 2.0 |
| Output sample rate | 24 kHz |
| Acoustic representation | 100 mel bins, 24-dimensional acoustic latents |
| Waveform / latent hop | 256 / 512 samples |
| Public language IDs | `en_us`, `en_gb`, `es`, `it`, `fr`, `de`, `vi` |
| Managed voices | 10 |
| Affect controls | 6 independently scored axes, axis-order version 2 |
| Default quality profile | 8-step Heun sampling |
| Fixed graph budgets | 512 G2P text tokens, 512 phone frames, 640 latent frames |
| Latent target buckets | 256, 384, 512, 640, selected by smallest fit |

Managed voices:

```text
ariadne, felix, gwen, ink, max, orpheus, rex, scylla, stone, tuesday
```

English dialect assignment:

| English language ID | Voices |
| --- | --- |
| `en_gb` | Ink, Orpheus, Tuesday |
| `en_us` | Ariadne, Felix, Gwen, Max, Rex, Scylla, Stone |

All managed voices are trained for Spanish, Italian, French, German, and
Vietnamese. The bundle manifest lists the exact supported languages and default
language for each voice.

## Architecture

```text
text
  -> spoken-text normalization and phrase planning
  -> Scylla's Band phrase-level G2P
  -> phone IDs, punctuation, and boundary/context features
  -> duration prediction
  -> duration-expanded phone conditioning
  -> rectified-flow acoustic latent estimation
  -> Scylla's Band acoustic adapter and frozen Vocos decoder
  -> 24 kHz waveform
  -> long-form assembly when needed
```

| Component | Details |
| --- | --- |
| Text frontend | Seven-language phrase-level G2P with fixed 512-token input budget |
| Duration predictor | 192 hidden size, 4 layers, 4 heads, up to 512 phone positions |
| Acoustic generator | 24-D rectified-flow latents, 512 hidden size, 12 layers, 8 heads, AdaLN conditioning, QK normalization |
| Span context | Three context segments over up to 768 phones |
| Conditioning | Voice, language, boundary/span context, and six-axis affect; selected v2 checkpoints use no reference-audio conditioning |
| Vocoder | Six-layer, 384-channel acoustic adapter into frozen `charactr/vocos-mel-24khz` |

V2 uses explicit punctuation silence targets. Host-side long-form assembly adds
clean inter-chunk silence and short boundary fades, while the duration model
controls timing inside each generated chunk.

## Affect and Whisper Controls

V2 exposes four core delivery axes and two overlays:

```text
core:     calm, joy, anger, sadness
overlays: sarcasm, whisper
```

Each axis is independently bounded in `[0, 1]` and multiple axes may be
nonzero. For example, whisper can be combined with sadness or anger instead of
being selected as a mutually exclusive speaking style.

Training ratings use a `0` to `4` scale and are normalized for inference:

| Human rating | Runtime value |
| ---: | ---: |
| 0 | 0.0 |
| 1 | 0.25 |
| 2 | 0.5 |
| 3 | 0.75 |
| 4 | 1.0 |

`--emotion-scale` controls classifier-free guidance separately from the axis
values:

```text
guided = null + scale * (conditioned - null)
```

`1` uses the requested affect vector directly. Values around `1.25` to `1.5`
are a practical first range for stronger delivery. Higher values extrapolate
beyond direct conditioning and can produce exaggerated timing, voice drift, or
distortion. Guidance scales other than `1` require both null and conditioned
model evaluation.

Whisper response varies by voice, language, wording, sampler, and strength. It
is a learned delivery overlay, not a post-processing whisper effect.

## Runtime Bundles

### ONNX

ONNX Runtime is the primary cross-platform inference path. Full-precision and
optional quantized variants use the same manifest-driven runtime contract.
The export uses opset 18, target-bucket graphs, and shared external weights.

```text
onnx/
  manifest.json
  onnx/g2p/model.onnx
  onnx/g2p/tokenizer.json
  onnx/components/duration_predictor.onnx
  onnx/components/vector_context_encoder.onnx
  onnx/components/vector_estimator_b{256,384,512,640}.onnx
  onnx/components/vocoder_b{256,384,512,640}.onnx
  onnx/components/shared_weights.bin
  assets/
```

### Apple Core AI

The Core AI bundle runs the complete inference chain through native `.aimodel`
assets. It uses shared-weight multifunction assets for frame buckets rather
than duplicating weights in separate model files.

```text
coreai/
  manifest.json
  coreai/g2p.aimodel/
  coreai/duration_predictor.aimodel/
  coreai/vector_context_encoder.aimodel/
  coreai/vector_estimator.aimodel/
  coreai/vocoder.aimodel/
  coreai/release.json
  assets/
```

Download and run it with:

```bash
python -m scyllasband download \
    --repo-id spybyscript/scyllasbandv2 \
    --runtime-bundles coreai \
    --yes
python -m scyllasband speak scyllasband/models/coreai \
    --backend coreai \
    --voice gwen \
    --language en_us \
    -o hello_coreai.wav \
    "Hello from native Core AI."
```

Core AI requires an iOS 27 or macOS 27 SDK/runtime. The first use of a model
function includes specialization cost. Preferred GPU execution permits system
fallback; no Apple Neural Engine guarantee is made. Core ML is not a v2
runtime backend.

Other experimental bundles may be published separately, but ONNX and Core AI
are the primary v2 release targets.

## PyTorch Checkpoints

The v2 Hugging Face repository keeps export and research checkpoints beside
the inference bundles:

```text
pytorch/
  duration.pt
  vector_estimator.pt
  vocoder.pt
  autoencoder.pt
  g2p.pt
  ...matching configuration and index files...
```

These files are not required for normal ONNX or Core AI inference. Components
must be used with their matching configuration, condition indexes, phone
vocabulary, G2P tokenizer, and language/affect metadata. Mixing checkpoints or
sidecars from another release is unsupported.

## Training Data and Labels

Training data is not distributed. V2 was trained primarily on synthetic speech
covering the managed voices and seven public language IDs. Text generation was
designed for broad vocabulary, difficult pronunciations, heteronyms, numbers,
acronyms, varied sentence lengths, and emotional delivery.

Prepared audio was checked with ASR, alignment, acoustic-quality detection,
and human review. Selected clips were regenerated at higher synthesis quality
when the original contained clipping, micro-stutters, chirps, unstable pitch,
or other mechanical artifacts. Duration supervision comes from forced
alignment against the actual audio rather than a duration value stored in the
source manifest.

The affect dataset is rated independently across all six axes. Multiple
nonzero values can describe one clip, allowing mixtures such as quiet anger,
sad joy, sarcastic calm, or emotional whispering. Ratings are used as
continuous supervision rather than treating the original requested emotion as
ground truth.

## Validation

Release validation should keep the same text, phones, voice, language, affect,
guidance, sampler, steps, seed, and duration scale when comparing PyTorch,
ONNX, and Core AI. The public validation tooling produces per-voice,
per-language, and per-affect ASR/WER reports with review audio and transcript
diffs.

Useful checks:

```bash
python -m scyllasband validate-bundle scyllasband/models/onnx
python -m scyllasband validate-bundle scyllasband/models/coreai
python -m scyllasband compare-metadata LEFT.json RIGHT.json
```

The selected bundle's manifest, not this prose, is authoritative for finalized
artifacts and controls.

## Limitations

- The model supports managed voices rather than arbitrary speaker cloning.
- Each voice uses one manifest-declared English dialect; requesting the other
  English dialect for that voice is unsupported.
- Affect and whisper response varies by voice, language, wording, sampler,
  steps, and CFG scale.
- Strong CFG can exaggerate timing or destabilize identity.
- Very long text is synthesized in planned chunks; unusual fragments and poor
  punctuation can still produce awkward pacing.
- Names, rare words, code, malformed input, and language-mismatched text can be
  mispronounced.
- Synthetic training data can preserve pronunciation, prosody, dialect, and
  language biases from its source systems.
- Core AI requires current Apple platform tooling and performs first-use model
  specialization. ONNX is the portable fallback.
- Core ML is not supported by the v2 release.

## Safety and Misuse

Do not use Scylla's Band to impersonate people, deceive listeners, bypass
consent, or create speech that represents a real person as saying something
they did not say. Generated audio should be disclosed as synthetic whenever
the context could otherwise confuse a listener.

## License

Scylla's Band v2 is released under Apache 2.0. See [`LICENSE`](LICENSE).

## Citation

```bibtex
@software{spybyscript_scyllasband_v2_2026,
  author = {Spybyscript},
  title = {Scylla's Band v2: Multilingual Expressive Duration-Flow Text-to-Speech},
  year = {2026},
  url = {https://huggingface.co/spybyscript/scyllasbandv2},
  license = {Apache-2.0}
}
```

## Acknowledgments

Scylla's Band uses ONNX Runtime for portable inference and Apple Core AI for
native Apple execution. Vocos provides the frozen 24 kHz waveform-decoder
backbone. Wiktionary-derived vocabulary, eSpeak phonemization, and Montreal
Forced Aligner tooling contributed to text coverage and duration supervision.