add glossolalia phonetics grounding to card
#4
by akshan-main - opened
README.md
CHANGED
|
@@ -9,7 +9,8 @@ python_version: "3.11"
|
|
| 9 |
app_file: app.py
|
| 10 |
pinned: false
|
| 11 |
license: apache-2.0
|
| 12 |
-
short_description:
|
|
|
|
| 13 |
tags:
|
| 14 |
- gradio
|
| 15 |
- build-small-hackathon
|
|
@@ -27,79 +28,47 @@ tags:
|
|
| 27 |
|
| 28 |
# Glossolalia Dial
|
| 29 |
|
| 30 |
-
Type a sentence.
|
| 31 |
-
|
| 32 |
-
At **0** you hear it spoken cleanly. At **4** you hear it as wordless glossolalia: invented words that obey English sound-rules but mean nothing, in the same voice. **The middle of the dial is the point.** At 2 the sentence is half-dissolved, recognizable but slipping, not a clean cut between speech and noise.
|
| 33 |
-
|
| 34 |
-
The dial is a learned scalar conditioner. A small network maps the dial position to a vector added into F5-TTS's time embedding (the same AdaLN pathway the model uses for the diffusion timestep), co-trained with a LoRA. The naive version (appending a `tongues N` token to the prompt) failed: F5-TTS has no language-model front end, so it read the level word aloud and intelligibility moved the wrong way (Spearman -0.70). Making the conditioning a non-text scalar means the model cannot speak it, and the LoRA only has to learn one thing: the per-level audio transformation.
|
| 35 |
-
|
| 36 |
-
**Live:** turn the dial, hit *play this dial*. Or hit *dissolve* to hear the whole 0 to 4 sweep crossfaded into one take.
|
| 37 |
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
No shipped product, open or closed, gives you a *typed-input, graded, voice-locked* slide into glossolalia. Emotion and prosody sliders (Hume, ElevenLabs) move other axes and optimise *for* intelligibility. The closest research (dysarthric-speech clones, discrete lyric-swap edits) solves a different problem. The originality here is the interaction, not the model: a continuous, learned intelligibility axis on one token.
|
| 41 |
-
|
| 42 |
-
It is also not a DSP trick. Reverb, formant-shift, and vocoders act uniformly on audio that already exists. They cannot read a sentence you just invented and erode specific words into different but plausible ones while holding syllable count, stress, and voice. Only a model trained for it can, which is why this needed a fine-tune.
|
| 43 |
|
| 44 |
## Two modes
|
| 45 |
|
| 46 |
-
- **Tongues**
|
| 47 |
-
- **Ghost**
|
| 48 |
-
|
| 49 |
-
## How it was built
|
| 50 |
-
|
| 51 |
-
There is no dataset of "sentences gradually dissolving into nonsense", so we made one. This is the whole reason a single person can build this: instead of hunting for labeled data that does not exist, we manufacture the training target from plain text.
|
| 52 |
|
| 53 |
-
|
| 54 |
-
2. For each, generate five corrupted phoneme variants at substitution rates 0, 0.25, 0.5, 0.75, 1.0. The corruption keeps the English phoneme inventory, preserves syllable count and stress, and leans toward open CV syllables. This is grounded in the phonetics of real glossolalia (Samarin 1972, Goodman 1972; Link & Tomaschek 2024 measured 95.7% CV structure across 7,486 glossolalic syllables).
|
| 55 |
-
3. Have base F5-TTS read each corrupted variant in two reference voices. That gives 30,000 (audio, original-sentence, level, voice) tuples.
|
| 56 |
-
4. Fine-tune a LoRA so that, given the *original* sentence plus a `tongues N` token, it reproduces the level-N audio. The model never sees the corrupted text. It learns the mapping from the dial alone.
|
| 57 |
|
| 58 |
-
|
| 59 |
|
| 60 |
-
##
|
| 61 |
|
| 62 |
-
|
| 63 |
-
- **Whisper-WER rises across levels.** The output is meant to get less intelligible, so word-error-rate should climb monotonically with the dial. Hallucination-guarded, so invented words at the top do not score a spuriously low WER when Whisper invents coherent text from noise.
|
| 64 |
-
- **Voice preserved.** Resemblyzer speaker-embedding cosine between dial-0 and the higher levels should stay close, so the words dissolve but the speaker does not.
|
| 65 |
-
- **The middle has to exist.** Hand-listen at dial 2 for partial dissolution, not a bimodal jump from clean to gibberish.
|
| 66 |
|
| 67 |
-
The
|
| 68 |
|
| 69 |
-
## Models (all
|
| 70 |
|
| 71 |
| Model | Size | Role |
|
| 72 |
|---|---|---|
|
| 73 |
-
| F5-TTS v1 Base | ~336M | flow-matching TTS, zero-shot
|
| 74 |
-
| Glossolalia LoRA | rank-16
|
| 75 |
-
| DistilGPT-2 | ~82M | Ghost-mode word
|
| 76 |
-
| Whisper base.en | ~74M | clone-reference transcription
|
| 77 |
|
| 78 |
-
|
| 79 |
|
| 80 |
## Badges
|
| 81 |
|
| 82 |
-
- π― **Well-Tuned**: fine-tuned LoRA published at [`akshan-main/glossolalia-dial-lora`](https://huggingface.co/akshan-main/glossolalia-dial-lora).
|
| 83 |
-
- π **Off the Grid**: no cloud APIs
|
| 84 |
-
- π¨ **Off-Brand**: the dial is a hand-built circular knob
|
| 85 |
-
- π **Field Notes**:
|
| 86 |
-
|
| 87 |
-
## Use it
|
| 88 |
-
|
| 89 |
-
- **Voice:** pick one of nine presets (warm and calm, high and arch, deep and slow, plus theatrical, haunted, and storyteller character voices from public-domain LibriVox), or open *clone your own voice* to upload or record a 6-12s clip.
|
| 90 |
-
- **Background music:** drop in an instrumental and it tempo-locks, tunes the vocal toward the track's key, then mixes it over.
|
| 91 |
-
- **Hand-tune words:** click any word to change its pronunciation or stretch it.
|
| 92 |
-
- **Space (reverb):** dry to cathedral, applied live without re-running the model.
|
| 93 |
|
| 94 |
## Links
|
| 95 |
|
|
|
|
|
|
|
| 96 |
- **Model (LoRA):** https://huggingface.co/akshan-main/glossolalia-dial-lora
|
| 97 |
-
- **Dataset
|
| 98 |
- **Code:** https://github.com/akshan-main/glossolalia
|
| 99 |
-
- **Field Notes:** https://github.com/akshan-main/glossolalia/blob/main/BLOG.md
|
| 100 |
-
- **Demo video:** https://youtu.be/dDOaBNfihyo
|
| 101 |
-
- **Social post:** https://x.com/frutigeraerosol/status/2066667649338417367
|
| 102 |
-
|
| 103 |
-
## Team
|
| 104 |
-
|
| 105 |
-
- [`akshan-main`](https://huggingface.co/akshan-main)
|
|
|
|
| 9 |
app_file: app.py
|
| 10 |
pinned: false
|
| 11 |
license: apache-2.0
|
| 12 |
+
short_description: Convert clear speech to glossolalia or mondegreen
|
| 13 |
+
|
| 14 |
tags:
|
| 15 |
- gradio
|
| 16 |
- build-small-hackathon
|
|
|
|
| 28 |
|
| 29 |
# Glossolalia Dial
|
| 30 |
|
| 31 |
+
Glossolalia Dial is a text-to-speech toy with one knob. Type a sentence and it speaks it. Turn the knob up and the words come apart into wordless babble that still sounds like a language, in the same voice the whole way. The trick is the middle, where the sentence is half-dissolved instead of cutting straight from speech to noise.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
+
**Demo video:** https://youtu.be/dDOaBNfihyo
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
## Two modes
|
| 36 |
|
| 37 |
+
- **Tongues** is the words slurring into made-up pseudo-words. A LoRA and a learned dial trained into F5-TTS. This is the glossolalia.
|
| 38 |
+
- **Ghost** swaps every word for a real one that sounds close (seashells becomes seagulls), the misheard-lyric thing. Pareidolia, not glossolalia, and labeled so. No model, runs live.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
+
## Use it
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
+
Type a sentence, pick a voice (nine presets or clone your own from a short clip), turn the dial 0 to 4, and hit play. Hit dissolve to hear the whole sweep in one take. You can also mix in a backing track, hand-tune any word, or wrap it in reverb.
|
| 43 |
|
| 44 |
+
## How it works
|
| 45 |
|
| 46 |
+
There's no dataset of sentences falling apart into nonsense, so it builds one: corrupt each sentence's phonemes at five rising rates, have base F5-TTS read each, then train the dial to reproduce that slide from the clean sentence alone. The model never sees the corrupted text, it learns the slide from the dial. Ghost mode trains nothing, it searches CMUdict for close-sounding words live and reranks them with DistilGPT-2.
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
+
The corruption is built to sit inside real glossolalia's structure, not just noise: it only swaps within the English phoneme inventory (speakers reuse their native phonotactics, Samarin 1972, Goodman 1972) and keeps syllable count and stress while leaning toward open CV syllables (measured at 95.7% in glossolalic speech, Link & Tomaschek 2024). The exact sound palette is a taste choice, not a claim to match the distribution.
|
| 49 |
|
| 50 |
+
## Models (all local, all under 32B)
|
| 51 |
|
| 52 |
| Model | Size | Role |
|
| 53 |
|---|---|---|
|
| 54 |
+
| F5-TTS v1 Base | ~336M | the voice (flow-matching TTS, zero-shot clone) |
|
| 55 |
+
| Glossolalia LoRA | rank-16 | the dial |
|
| 56 |
+
| DistilGPT-2 | ~82M | Ghost-mode word reranking |
|
| 57 |
+
| Whisper base.en | ~74M | clone-reference transcription |
|
| 58 |
|
| 59 |
+
No cloud APIs. Everything runs on the Space.
|
| 60 |
|
| 61 |
## Badges
|
| 62 |
|
| 63 |
+
- π― **Well-Tuned**: the fine-tuned LoRA, published at [`akshan-main/glossolalia-dial-lora`](https://huggingface.co/akshan-main/glossolalia-dial-lora).
|
| 64 |
+
- π **Off the Grid**: no cloud APIs, zero cloud SDKs in `requirements.txt`.
|
| 65 |
+
- π¨ **Off-Brand**: the dial is a hand-built circular knob, not a default slider.
|
| 66 |
+
- π **Field Notes**: the writeup is linked below.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
|
| 68 |
## Links
|
| 69 |
|
| 70 |
+
- **Demo video:** https://youtu.be/dDOaBNfihyo
|
| 71 |
+
- **Writeup:** https://x.com/frutigeraerosol/status/2066667649338417367
|
| 72 |
- **Model (LoRA):** https://huggingface.co/akshan-main/glossolalia-dial-lora
|
| 73 |
+
- **Dataset:** https://huggingface.co/datasets/akshan-main/glossolalia-inputs
|
| 74 |
- **Code:** https://github.com/akshan-main/glossolalia
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|