add glossolalia phonetics grounding to card

#4
by akshan-main - opened
Files changed (1) hide show
  1. README.md +24 -55
README.md CHANGED
@@ -9,7 +9,8 @@ python_version: "3.11"
9
  app_file: app.py
10
  pinned: false
11
  license: apache-2.0
12
- short_description: One dial, clean speech to wordless tongues, same voice
 
13
  tags:
14
  - gradio
15
  - build-small-hackathon
@@ -27,79 +28,47 @@ tags:
27
 
28
  # Glossolalia Dial
29
 
30
- Type a sentence. Pick a voice. Turn the dial.
31
-
32
- At **0** you hear it spoken cleanly. At **4** you hear it as wordless glossolalia: invented words that obey English sound-rules but mean nothing, in the same voice. **The middle of the dial is the point.** At 2 the sentence is half-dissolved, recognizable but slipping, not a clean cut between speech and noise.
33
-
34
- The dial is a learned scalar conditioner. A small network maps the dial position to a vector added into F5-TTS's time embedding (the same AdaLN pathway the model uses for the diffusion timestep), co-trained with a LoRA. The naive version (appending a `tongues N` token to the prompt) failed: F5-TTS has no language-model front end, so it read the level word aloud and intelligibility moved the wrong way (Spearman -0.70). Making the conditioning a non-text scalar means the model cannot speak it, and the LoRA only has to learn one thing: the per-level audio transformation.
35
-
36
- **Live:** turn the dial, hit *play this dial*. Or hit *dissolve* to hear the whole 0 to 4 sweep crossfaded into one take.
37
 
38
- ## Why it is worth a look
39
-
40
- No shipped product, open or closed, gives you a *typed-input, graded, voice-locked* slide into glossolalia. Emotion and prosody sliders (Hume, ElevenLabs) move other axes and optimise *for* intelligibility. The closest research (dysarthric-speech clones, discrete lyric-swap edits) solves a different problem. The originality here is the interaction, not the model: a continuous, learned intelligibility axis on one token.
41
-
42
- It is also not a DSP trick. Reverb, formant-shift, and vocoders act uniformly on audio that already exists. They cannot read a sentence you just invented and erode specific words into different but plausible ones while holding syllable count, stress, and voice. Only a model trained for it can, which is why this needed a fine-tune.
43
 
44
  ## Two modes
45
 
46
- - **Tongues**: true glossolalia. The dial conditions the LoRA to slur the sentence into invented, pronounceable pseudo-words. `she sells seashells by the seashore` becomes something like `she'll sell sicials by the sohar` at the middle, wordless tongues at the top.
47
- - **Ghost**: mondegreen. Real English words are swapped for similar-sounding real words (`seashells` to `seagulls`), the misheard-lyric effect. More words change as the dial rises. **This is pareidolia, not glossolalia**, and is labeled as such.
48
-
49
- ## How it was built
50
-
51
- There is no dataset of "sentences gradually dissolving into nonsense", so we made one. This is the whole reason a single person can build this: instead of hunting for labeled data that does not exist, we manufacture the training target from plain text.
52
 
53
- 1. Take 3000 public-domain sentences (Project Gutenberg + LibriSpeech). Public-domain on purpose: the corpus is rights-clean and the build is reproducible end to end, which also keeps the project Off-the-Grid (no scraped or licensed text, no cloud calls).
54
- 2. For each, generate five corrupted phoneme variants at substitution rates 0, 0.25, 0.5, 0.75, 1.0. The corruption keeps the English phoneme inventory, preserves syllable count and stress, and leans toward open CV syllables. This is grounded in the phonetics of real glossolalia (Samarin 1972, Goodman 1972; Link & Tomaschek 2024 measured 95.7% CV structure across 7,486 glossolalic syllables).
55
- 3. Have base F5-TTS read each corrupted variant in two reference voices. That gives 30,000 (audio, original-sentence, level, voice) tuples.
56
- 4. Fine-tune a LoRA so that, given the *original* sentence plus a `tongues N` token, it reproduces the level-N audio. The model never sees the corrupted text. It learns the mapping from the dial alone.
57
 
58
- Training ran on Modal (A100). Ghost mode runs the mondegreen search live (CMUdict + PanPhon phonetic distance, re-ranked by DistilGPT-2 for common-word coherence), no second model trained.
59
 
60
- ## Validation gates
61
 
62
- The gates were written into the evaluation script before any GPU spend, so the dial had to earn its result:
63
- - **Whisper-WER rises across levels.** The output is meant to get less intelligible, so word-error-rate should climb monotonically with the dial. Hallucination-guarded, so invented words at the top do not score a spuriously low WER when Whisper invents coherent text from noise.
64
- - **Voice preserved.** Resemblyzer speaker-embedding cosine between dial-0 and the higher levels should stay close, so the words dissolve but the speaker does not.
65
- - **The middle has to exist.** Hand-listen at dial 2 for partial dissolution, not a bimodal jump from clean to gibberish.
66
 
67
- The honest evidence: turn the dial and the output slides from the sentence as typed at 0, to a half-dissolved version at 2 that is recognizably the same sentence but slipping, to wordless at 4, in the same voice. The exact words change run to run because it is sampling a generative model, but the graded slide is stable. The middle is real.
68
 
69
- ## Models (all under 32B, all local)
70
 
71
  | Model | Size | Role |
72
  |---|---|---|
73
- | F5-TTS v1 Base | ~336M | flow-matching TTS, zero-shot voice clone |
74
- | Glossolalia LoRA | rank-16 adapter | the dial (published, see below) |
75
- | DistilGPT-2 | ~82M | Ghost-mode word re-ranking |
76
- | Whisper base.en | ~74M | clone-reference transcription, validation |
77
 
78
- Nothing calls a cloud API. Every model runs on the Space.
79
 
80
  ## Badges
81
 
82
- - 🎯 **Well-Tuned**: fine-tuned LoRA published at [`akshan-main/glossolalia-dial-lora`](https://huggingface.co/akshan-main/glossolalia-dial-lora).
83
- - πŸ”Œ **Off the Grid**: no cloud APIs anywhere; `requirements.txt` has zero cloud SDKs.
84
- - 🎨 **Off-Brand**: the dial is a hand-built circular knob (CSS conic-gradient rim, pointer-drag JS, arc indicator, injected HTML) driving the model, not a default Gradio slider.
85
- - πŸ““ **Field Notes**: full write-up in [`BLOG.md`](https://github.com/akshan-main/glossolalia/blob/main/BLOG.md).
86
-
87
- ## Use it
88
-
89
- - **Voice:** pick one of nine presets (warm and calm, high and arch, deep and slow, plus theatrical, haunted, and storyteller character voices from public-domain LibriVox), or open *clone your own voice* to upload or record a 6-12s clip.
90
- - **Background music:** drop in an instrumental and it tempo-locks, tunes the vocal toward the track's key, then mixes it over.
91
- - **Hand-tune words:** click any word to change its pronunciation or stretch it.
92
- - **Space (reverb):** dry to cathedral, applied live without re-running the model.
93
 
94
  ## Links
95
 
 
 
96
  - **Model (LoRA):** https://huggingface.co/akshan-main/glossolalia-dial-lora
97
- - **Dataset (inputs):** https://huggingface.co/datasets/akshan-main/glossolalia-inputs
98
  - **Code:** https://github.com/akshan-main/glossolalia
99
- - **Field Notes:** https://github.com/akshan-main/glossolalia/blob/main/BLOG.md
100
- - **Demo video:** https://youtu.be/dDOaBNfihyo
101
- - **Social post:** https://x.com/frutigeraerosol/status/2066667649338417367
102
-
103
- ## Team
104
-
105
- - [`akshan-main`](https://huggingface.co/akshan-main)
 
9
  app_file: app.py
10
  pinned: false
11
  license: apache-2.0
12
+ short_description: Convert clear speech to glossolalia or mondegreen
13
+
14
  tags:
15
  - gradio
16
  - build-small-hackathon
 
28
 
29
  # Glossolalia Dial
30
 
31
+ Glossolalia Dial is a text-to-speech toy with one knob. Type a sentence and it speaks it. Turn the knob up and the words come apart into wordless babble that still sounds like a language, in the same voice the whole way. The trick is the middle, where the sentence is half-dissolved instead of cutting straight from speech to noise.
 
 
 
 
 
 
32
 
33
+ **Demo video:** https://youtu.be/dDOaBNfihyo
 
 
 
 
34
 
35
  ## Two modes
36
 
37
+ - **Tongues** is the words slurring into made-up pseudo-words. A LoRA and a learned dial trained into F5-TTS. This is the glossolalia.
38
+ - **Ghost** swaps every word for a real one that sounds close (seashells becomes seagulls), the misheard-lyric thing. Pareidolia, not glossolalia, and labeled so. No model, runs live.
 
 
 
 
39
 
40
+ ## Use it
 
 
 
41
 
42
+ Type a sentence, pick a voice (nine presets or clone your own from a short clip), turn the dial 0 to 4, and hit play. Hit dissolve to hear the whole sweep in one take. You can also mix in a backing track, hand-tune any word, or wrap it in reverb.
43
 
44
+ ## How it works
45
 
46
+ There's no dataset of sentences falling apart into nonsense, so it builds one: corrupt each sentence's phonemes at five rising rates, have base F5-TTS read each, then train the dial to reproduce that slide from the clean sentence alone. The model never sees the corrupted text, it learns the slide from the dial. Ghost mode trains nothing, it searches CMUdict for close-sounding words live and reranks them with DistilGPT-2.
 
 
 
47
 
48
+ The corruption is built to sit inside real glossolalia's structure, not just noise: it only swaps within the English phoneme inventory (speakers reuse their native phonotactics, Samarin 1972, Goodman 1972) and keeps syllable count and stress while leaning toward open CV syllables (measured at 95.7% in glossolalic speech, Link & Tomaschek 2024). The exact sound palette is a taste choice, not a claim to match the distribution.
49
 
50
+ ## Models (all local, all under 32B)
51
 
52
  | Model | Size | Role |
53
  |---|---|---|
54
+ | F5-TTS v1 Base | ~336M | the voice (flow-matching TTS, zero-shot clone) |
55
+ | Glossolalia LoRA | rank-16 | the dial |
56
+ | DistilGPT-2 | ~82M | Ghost-mode word reranking |
57
+ | Whisper base.en | ~74M | clone-reference transcription |
58
 
59
+ No cloud APIs. Everything runs on the Space.
60
 
61
  ## Badges
62
 
63
+ - 🎯 **Well-Tuned**: the fine-tuned LoRA, published at [`akshan-main/glossolalia-dial-lora`](https://huggingface.co/akshan-main/glossolalia-dial-lora).
64
+ - πŸ”Œ **Off the Grid**: no cloud APIs, zero cloud SDKs in `requirements.txt`.
65
+ - 🎨 **Off-Brand**: the dial is a hand-built circular knob, not a default slider.
66
+ - πŸ““ **Field Notes**: the writeup is linked below.
 
 
 
 
 
 
 
67
 
68
  ## Links
69
 
70
+ - **Demo video:** https://youtu.be/dDOaBNfihyo
71
+ - **Writeup:** https://x.com/frutigeraerosol/status/2066667649338417367
72
  - **Model (LoRA):** https://huggingface.co/akshan-main/glossolalia-dial-lora
73
+ - **Dataset:** https://huggingface.co/datasets/akshan-main/glossolalia-inputs
74
  - **Code:** https://github.com/akshan-main/glossolalia