Instructions to use Sourajit123/SouraTTS with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Pocket-TTS
How to use Sourajit123/SouraTTS with Pocket-TTS:
from pocket_tts import TTSModel import scipy.io.wavfile tts_model = TTSModel.load_model("Sourajit123/SouraTTS") voice_state = tts_model.get_state_for_audio_prompt( "hf://kyutai/tts-voices/alba-mackenna/casual.wav" ) audio = tts_model.generate_audio(voice_state, "Hello world, this is a test.") # Audio is a 1D torch tensor containing PCM data. scipy.io.wavfile.write("output.wav", tts_model.sample_rate, audio.numpy()) - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,8 +1,161 @@
|
|
| 1 |
---
|
| 2 |
-
license: mit
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
-
|
| 6 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
pipeline_tag: text-to-speech
|
| 8 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
|
|
|
| 2 |
language:
|
| 3 |
- en
|
| 4 |
+
license: mit
|
| 5 |
+
tags:
|
| 6 |
+
- text-to-speech
|
| 7 |
+
- emotion
|
| 8 |
+
- voice-cloning
|
| 9 |
+
- pocket-tts
|
| 10 |
+
- emoshift
|
| 11 |
+
- tts
|
| 12 |
+
- speech-synthesis
|
| 13 |
pipeline_tag: text-to-speech
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
# SouraTTS v1
|
| 17 |
+
|
| 18 |
+
A lightweight, expressive, CPU-friendly Text-to-Speech engine built on top of
|
| 19 |
+
[Pocket-TTS](https://huggingface.co/kyutai/pocket-tts) by Kyutai, with emotional
|
| 20 |
+
expressiveness powered by an EmoShift-inspired activation steering layer.
|
| 21 |
+
|
| 22 |
+
**6 emotions. 8 built-in voices. 27KB of trained weights. Runs on CPU.**
|
| 23 |
+
|
| 24 |
+
## Demo
|
| 25 |
+
```python
|
| 26 |
+
from emotts import EmoTTS
|
| 27 |
+
|
| 28 |
+
tts = EmoTTS(
|
| 29 |
+
weights="emoshift_layer5.pt",
|
| 30 |
+
meta="emoshift_layer5.json"
|
| 31 |
+
)
|
| 32 |
+
|
| 33 |
+
tts.synthesize(
|
| 34 |
+
text = "I just got the job! I cannot believe it!",
|
| 35 |
+
voice = "alba",
|
| 36 |
+
emotion = "happy",
|
| 37 |
+
intensity = 1.0,
|
| 38 |
+
output = "out.wav"
|
| 39 |
+
)
|
| 40 |
+
```
|
| 41 |
+
|
| 42 |
+
## Installation
|
| 43 |
+
```bash
|
| 44 |
+
pip install pocket-tts scipy torch
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
Then download the three files from this repo:
|
| 48 |
+
- `emoshift_layer5.pt`
|
| 49 |
+
- `emoshift_layer5.json`
|
| 50 |
+
- `emotts.py`
|
| 51 |
+
|
| 52 |
+
Place all three in the same directory and run the demo above.
|
| 53 |
+
|
| 54 |
+
## Supported Emotions
|
| 55 |
+
|
| 56 |
+
| Emotion | Recommended Intensity |
|
| 57 |
+
|----------|-----------------------|
|
| 58 |
+
| neutral | 0.0 |
|
| 59 |
+
| happy | 0.8 β 1.0 |
|
| 60 |
+
| sad | 0.8 β 1.0 |
|
| 61 |
+
| angry | 0.8 β 1.0 |
|
| 62 |
+
| fear | 0.8 β 1.0 |
|
| 63 |
+
| disgust | 0.8 β 1.0 |
|
| 64 |
+
|
| 65 |
+
Intensities above 1.2 may cause generation instability on some voice and emotion
|
| 66 |
+
combinations.
|
| 67 |
+
|
| 68 |
+
## Built-in Voices
|
| 69 |
+
|
| 70 |
+
`alba` `marius` `cosette` `jean` `fantine` `eponine` `azelma` `javert`
|
| 71 |
+
|
| 72 |
+
**Alba** is the recommended default β highest quality and most stable across all emotions.
|
| 73 |
+
|
| 74 |
+
## How It Works
|
| 75 |
+
|
| 76 |
+
SouraTTS combines two ideas:
|
| 77 |
+
|
| 78 |
+
**Pocket-TTS** is a 100M parameter streaming TTS model by Kyutai, designed for
|
| 79 |
+
CPU inference with fast first-audio latency and built-in voice cloning support.
|
| 80 |
+
|
| 81 |
+
**EmoShift** is an activation steering technique inspired by the EmoShift paper.
|
| 82 |
+
Instead of fine-tuning the entire model, we learn a small steering vector per emotion
|
| 83 |
+
(shape `1024,`) and inject it into the output of transformer layer 5 during inference.
|
| 84 |
+
The entire emotion control system is 27KB β less than 1/30th of the base model size.
|
| 85 |
+
|
| 86 |
+
### Architecture
|
| 87 |
+
```
|
| 88 |
+
Text input βββ [ Pocket-TTS frozen (100M params) ] βββ Audio
|
| 89 |
+
β
|
| 90 |
+
Layer 5 output + (intensity Γ steering_vector[emotion])
|
| 91 |
+
β
|
| 92 |
+
[ EmoShift Layer (27KB, 6 Γ 1024 params) ]
|
| 93 |
+
```
|
| 94 |
+
|
| 95 |
+
### Training
|
| 96 |
+
|
| 97 |
+
The steering vectors were trained on
|
| 98 |
+
[CREMA-D](https://huggingface.co/datasets/AbstractTTS/CREMA-D), a dataset of 7,442
|
| 99 |
+
emotional speech clips from 91 actors. We used a gender-balanced subset
|
| 100 |
+
(100 male + 100 female samples per emotion) to ensure consistent quality across
|
| 101 |
+
voice types.
|
| 102 |
+
|
| 103 |
+
Training objective:
|
| 104 |
+
- **Direction loss** β steer hidden states consistently toward each emotion direction
|
| 105 |
+
- **Magnitude loss** β prevent vectors from collapsing to zero
|
| 106 |
+
- **Variance loss** β encourage emotion-specific activation patterns
|
| 107 |
+
- **Orthogonality loss** β keep emotion vectors pointing in distinct directions
|
| 108 |
+
|
| 109 |
+
Pocket-TTS weights were fully frozen throughout. Only the 6 steering vectors
|
| 110 |
+
were trained, using Adam optimizer for 5 epochs on a Kaggle T4 GPU.
|
| 111 |
+
|
| 112 |
+
## Inference Speed
|
| 113 |
+
|
| 114 |
+
Since Pocket-TTS is optimized for CPU inference, SouraTTS inherits those
|
| 115 |
+
characteristics β fast first-audio latency with no GPU required. The EmoShift
|
| 116 |
+
layer adds negligible overhead (6 Γ 1024 float32 additions per forward pass).
|
| 117 |
+
|
| 118 |
+
## Known Limitations
|
| 119 |
+
|
| 120 |
+
- The final word of longer sentences may occasionally be slightly truncated β
|
| 121 |
+
a known characteristic of autoregressive TTS generation
|
| 122 |
+
- Intensities above 1.2 may cause audio instability on some voice/emotion combinations
|
| 123 |
+
- Voice cloning (custom voice upload) requires accepting the
|
| 124 |
+
[Kyutai Pocket-TTS terms](https://huggingface.co/kyutai/pocket-tts) β
|
| 125 |
+
planned for SouraTTS v2
|
| 126 |
+
- Non-alba voices show slightly reduced emotion stability, particularly on female voices
|
| 127 |
+
at high intensities β planned improvement for v2 with expanded training data
|
| 128 |
+
|
| 129 |
+
## Roadmap
|
| 130 |
+
|
| 131 |
+
- **v2** β Voice cloning support, expanded training data for improved female voice
|
| 132 |
+
quality, additional emotions
|
| 133 |
+
- **v3** β Real-time streaming API, Gradio demo Space
|
| 134 |
+
|
| 135 |
+
## Credits
|
| 136 |
+
|
| 137 |
+
- **Pocket-TTS** by [Kyutai](https://huggingface.co/kyutai) β base TTS model,
|
| 138 |
+
MIT licensed
|
| 139 |
+
- **EmoShift** β activation steering technique for emotional expressiveness
|
| 140 |
+
- **CREMA-D** β training dataset, Open Database License
|
| 141 |
+
- Built by [@Sourajit123](https://huggingface.co/Sourajit123)
|
| 142 |
+
|
| 143 |
+
## License
|
| 144 |
+
|
| 145 |
+
MIT β same as Pocket-TTS.
|
| 146 |
+
|
| 147 |
+
Please use responsibly. Do not use this model to clone voices without explicit
|
| 148 |
+
consent from the speaker, or to generate content that misrepresents real individuals.
|
| 149 |
+
|
| 150 |
+
## Citation
|
| 151 |
+
|
| 152 |
+
If you use SouraTTS in your work, please cite:
|
| 153 |
+
```bibtex
|
| 154 |
+
@misc{souratts2026,
|
| 155 |
+
author = {Sourajit123},
|
| 156 |
+
title = {SouraTTS v1: Expressive CPU TTS with EmoShift Activation Steering},
|
| 157 |
+
year = {2026},
|
| 158 |
+
publisher = {HuggingFace},
|
| 159 |
+
url = {https://huggingface.co/Sourajit123/SouraTTS}
|
| 160 |
+
}
|
| 161 |
+
```
|