File size: 4,397 Bytes
e2aea9c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 | ---
license: other
language:
- en
- zh
pipeline_tag: text-to-speech
tags:
- audio
- speech-editing
- text-to-speech
- safetensors
---
# dots.tts.edit
**dots.tts.edit** is a continuous autoregressive model for precise,
instruction-controlled speech editing and zero-shot text-to-speech synthesis.
It supports text replacement, insertion and deletion, emotion and prosody
control, pauses, enhancement, and background-audio operations while preserving
the speaker and the acoustic context outside edited regions.
- [Code](https://github.com/studio-dots-ai/dots.tts/tree/main)
- [Paper](https://arxiv.org/abs/2608.02673)
- [Interactive Playground](https://dots-studio-dots-tts-edit.hf.space)
- [Demo Page](https://dots-studio-dots-tts-edit-demo.static.hf.space)
## Model details
The model supports English and Mandarin speech editing and zero-shot speech
synthesis. It produces 48 kHz audio. The core weights use BF16; the speaker
encoder and vocoder retain FP32 weights.
Supported editing operations include:
- Text replacement, insertion, and deletion
- Localized emotion control
- Pitch and speaking-rate control
- Pause insertion and reduction
- Speech enhancement and background-audio operations
- Compositions of multiple edit operations
## Installation
Install the latest `main` branch of
[dots.tts](https://github.com/studio-dots-ai/dots.tts) and its recommended
dependencies:
```bash
git clone --branch main --single-branch https://github.com/studio-dots-ai/dots.tts.git
cd dots.tts
python -m pip install -e . -c constraints/recommended.txt
```
## Speech editing CLI
Source audio, a tagged instruction, and the output path are required. Source
and target transcripts are optional; when omitted, both are derived from the
instruction.
```bash
dots.tts.edit \
--model-name-or-path dots-studio/dots.tts.edit \
--source-audio /path/to/source.wav \
--instruction 'Hello <sub targ="small">brave</sub> world.' \
--output edited.wav
```
Use `--source-text` and `--target-text` to override the derived transcripts.
Source-speaker guidance defaults to `auto`: it is disabled when the instruction
contains at least one operation and every operation is `emo`, `bg`, or
`enhance`, and enabled for text, pitch, rate, pause, speaker-transfer, or mixed
edits. Pass bare `--use-xvector` (or
`--use-xvector on`) to force it on, and `--use-xvector off` to force it off.
Speaker guidance uses a default scale of `1.5`; override it with
`--speaker-scale` when needed.
## Python API
```python
import soundfile as sf
from dots_tts.edit_runtime import DotsTtsEditRuntime
runtime = DotsTtsEditRuntime.from_pretrained(
"dots-studio/dots.tts.edit",
precision="bfloat16",
)
result = runtime.generate_edit(
source_audio_path="/path/to/source.wav",
instruction='Hello <sub targ="small">brave</sub> world.',
# source_text and target_text are optional overrides.
# use_xvector defaults to "auto"; pass True or False to override it.
)
sf.write(
"edited.wav",
result["audio"].float().cpu().squeeze().numpy(),
result["sample_rate"],
)
```
Supported structural tags include `<del>`, `<ins>`,
`<sub targ="replacement">`, `<emo>`, `<pitch>`, `<rate>`, `<enhance>`,
`<bg>`, `<pause/>`, and `<spk_transfer/>`. Malformed instructions and
instructions that derive an empty source or target transcript are rejected.
`DotsTtsEditRuntime` also provides `generate` and `generate_stream` for
zero-shot TTS with the same checkpoint. See the
[code repository](https://github.com/studio-dots-ai/dots.tts) for examples.
## Local Edit Playground
Build the committed frontend source with Node.js 20+ and launch the local app:
```bash
cd apps/edit_playground/frontend
npm ci
npm run build
cd ../../..
python apps/edit_playground/app.py \
--model-name-or-path dots-studio/dots.tts.edit \
--optimize
```
## Limitations and responsible use
Generated or edited speech may contain pronunciation, timing,
speaker-similarity, or instruction-following errors. Do not use the model to
impersonate people, mislead listeners, violate consent, or create unlawful
content. Users are responsible for complying with applicable laws and for
clearly disclosing synthetic or edited audio where appropriate.
## License
See the repository license metadata and the
[`dots.tts`](https://github.com/studio-dots-ai/dots.tts) project for the terms
that apply to the weights and code.
|