Sourajit123 commited on
Commit
bb9ef6e
Β·
verified Β·
1 Parent(s): 6cef2b0

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +157 -4
README.md CHANGED
@@ -1,8 +1,161 @@
1
  ---
2
- license: mit
3
  language:
4
  - en
5
- base_model:
6
- - kyutai/pocket-tts
 
 
 
 
 
 
 
7
  pipeline_tag: text-to-speech
8
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
 
2
  language:
3
  - en
4
+ license: mit
5
+ tags:
6
+ - text-to-speech
7
+ - emotion
8
+ - voice-cloning
9
+ - pocket-tts
10
+ - emoshift
11
+ - tts
12
+ - speech-synthesis
13
  pipeline_tag: text-to-speech
14
+ ---
15
+
16
+ # SouraTTS v1
17
+
18
+ A lightweight, expressive, CPU-friendly Text-to-Speech engine built on top of
19
+ [Pocket-TTS](https://huggingface.co/kyutai/pocket-tts) by Kyutai, with emotional
20
+ expressiveness powered by an EmoShift-inspired activation steering layer.
21
+
22
+ **6 emotions. 8 built-in voices. 27KB of trained weights. Runs on CPU.**
23
+
24
+ ## Demo
25
+ ```python
26
+ from emotts import EmoTTS
27
+
28
+ tts = EmoTTS(
29
+ weights="emoshift_layer5.pt",
30
+ meta="emoshift_layer5.json"
31
+ )
32
+
33
+ tts.synthesize(
34
+ text = "I just got the job! I cannot believe it!",
35
+ voice = "alba",
36
+ emotion = "happy",
37
+ intensity = 1.0,
38
+ output = "out.wav"
39
+ )
40
+ ```
41
+
42
+ ## Installation
43
+ ```bash
44
+ pip install pocket-tts scipy torch
45
+ ```
46
+
47
+ Then download the three files from this repo:
48
+ - `emoshift_layer5.pt`
49
+ - `emoshift_layer5.json`
50
+ - `emotts.py`
51
+
52
+ Place all three in the same directory and run the demo above.
53
+
54
+ ## Supported Emotions
55
+
56
+ | Emotion | Recommended Intensity |
57
+ |----------|-----------------------|
58
+ | neutral | 0.0 |
59
+ | happy | 0.8 – 1.0 |
60
+ | sad | 0.8 – 1.0 |
61
+ | angry | 0.8 – 1.0 |
62
+ | fear | 0.8 – 1.0 |
63
+ | disgust | 0.8 – 1.0 |
64
+
65
+ Intensities above 1.2 may cause generation instability on some voice and emotion
66
+ combinations.
67
+
68
+ ## Built-in Voices
69
+
70
+ `alba` `marius` `cosette` `jean` `fantine` `eponine` `azelma` `javert`
71
+
72
+ **Alba** is the recommended default β€” highest quality and most stable across all emotions.
73
+
74
+ ## How It Works
75
+
76
+ SouraTTS combines two ideas:
77
+
78
+ **Pocket-TTS** is a 100M parameter streaming TTS model by Kyutai, designed for
79
+ CPU inference with fast first-audio latency and built-in voice cloning support.
80
+
81
+ **EmoShift** is an activation steering technique inspired by the EmoShift paper.
82
+ Instead of fine-tuning the entire model, we learn a small steering vector per emotion
83
+ (shape `1024,`) and inject it into the output of transformer layer 5 during inference.
84
+ The entire emotion control system is 27KB β€” less than 1/30th of the base model size.
85
+
86
+ ### Architecture
87
+ ```
88
+ Text input ──→ [ Pocket-TTS frozen (100M params) ] ──→ Audio
89
+ ↑
90
+ Layer 5 output + (intensity Γ— steering_vector[emotion])
91
+ ↑
92
+ [ EmoShift Layer (27KB, 6 Γ— 1024 params) ]
93
+ ```
94
+
95
+ ### Training
96
+
97
+ The steering vectors were trained on
98
+ [CREMA-D](https://huggingface.co/datasets/AbstractTTS/CREMA-D), a dataset of 7,442
99
+ emotional speech clips from 91 actors. We used a gender-balanced subset
100
+ (100 male + 100 female samples per emotion) to ensure consistent quality across
101
+ voice types.
102
+
103
+ Training objective:
104
+ - **Direction loss** β€” steer hidden states consistently toward each emotion direction
105
+ - **Magnitude loss** β€” prevent vectors from collapsing to zero
106
+ - **Variance loss** β€” encourage emotion-specific activation patterns
107
+ - **Orthogonality loss** β€” keep emotion vectors pointing in distinct directions
108
+
109
+ Pocket-TTS weights were fully frozen throughout. Only the 6 steering vectors
110
+ were trained, using Adam optimizer for 5 epochs on a Kaggle T4 GPU.
111
+
112
+ ## Inference Speed
113
+
114
+ Since Pocket-TTS is optimized for CPU inference, SouraTTS inherits those
115
+ characteristics β€” fast first-audio latency with no GPU required. The EmoShift
116
+ layer adds negligible overhead (6 Γ— 1024 float32 additions per forward pass).
117
+
118
+ ## Known Limitations
119
+
120
+ - The final word of longer sentences may occasionally be slightly truncated β€”
121
+ a known characteristic of autoregressive TTS generation
122
+ - Intensities above 1.2 may cause audio instability on some voice/emotion combinations
123
+ - Voice cloning (custom voice upload) requires accepting the
124
+ [Kyutai Pocket-TTS terms](https://huggingface.co/kyutai/pocket-tts) β€”
125
+ planned for SouraTTS v2
126
+ - Non-alba voices show slightly reduced emotion stability, particularly on female voices
127
+ at high intensities β€” planned improvement for v2 with expanded training data
128
+
129
+ ## Roadmap
130
+
131
+ - **v2** β€” Voice cloning support, expanded training data for improved female voice
132
+ quality, additional emotions
133
+ - **v3** β€” Real-time streaming API, Gradio demo Space
134
+
135
+ ## Credits
136
+
137
+ - **Pocket-TTS** by [Kyutai](https://huggingface.co/kyutai) β€” base TTS model,
138
+ MIT licensed
139
+ - **EmoShift** β€” activation steering technique for emotional expressiveness
140
+ - **CREMA-D** β€” training dataset, Open Database License
141
+ - Built by [@Sourajit123](https://huggingface.co/Sourajit123)
142
+
143
+ ## License
144
+
145
+ MIT β€” same as Pocket-TTS.
146
+
147
+ Please use responsibly. Do not use this model to clone voices without explicit
148
+ consent from the speaker, or to generate content that misrepresents real individuals.
149
+
150
+ ## Citation
151
+
152
+ If you use SouraTTS in your work, please cite:
153
+ ```bibtex
154
+ @misc{souratts2026,
155
+ author = {Sourajit123},
156
+ title = {SouraTTS v1: Expressive CPU TTS with EmoShift Activation Steering},
157
+ year = {2026},
158
+ publisher = {HuggingFace},
159
+ url = {https://huggingface.co/Sourajit123/SouraTTS}
160
+ }
161
+ ```