ankahi / executionguide.txt
bhriguverma's picture
Add files using upload-large-folder tool
dc7e099 verified
Raw
History Blame Contribute Delete
21.6 kB
# Ankahi β€” Master Execution Guide & Data Specification
---
## What We Are Building (The 12-Line Version)
Ankahi is an offline AAC (Augmentative & Alternative Communication) app for ~2.5 million Indian children with cerebral palsy who cannot speak. Commercial AAC devices cost β‚Ή20,000–₹2,00,000 and speak no Indian language. We run **Gemma 4 E4B** fully on a β‚Ή10,000 Android tablet. A child taps pictograms (symbols) on a grid, optionally makes a sound into the mic or points the camera at an object, and the model predicts the full sentence they intend β€” then speaks it aloud **in the parent's own cloned voice**. The critical differentiator is **per-child LoRA adapters**: each child gets a 30MB fine-tuned adapter that learns their vocabulary, their language mix (Punjabi + Hindi + English the way one Ludhiana family actually speaks), and their habitual phrases. We train 5 persona adapters on the H100 to demonstrate this personalisation. The winning demo moment: a child selects two pictograms and the tablet β€” in the mother's voice β€” says "Mummy, mujhe paratha khana hai!" for the first time. Everything runs offline. Zero bytes to cloud. 2.5GB on device. One-twentieth the cost of the alternative.
---
## Execution Order β€” Step by Step
### PHASE 0 β€” Local machine (your laptop/desktop), Day 1
These steps require no GPU. Do them before you ever touch the H100.
**Step 1 β€” Set up the GitHub repo**
```
Create repo: github.com/your-org/ankahi
Push all code files
Set license: Apache 2.0
Create Kaggle team and accept competition terms
```
**Step 2 β€” Set up environment variables locally**
```
Create .env file with:
GOOGLE_API_KEY=<your Gemini API key> ← for synthetic data generation
ANTHROPIC_API_KEY=<your Claude key> ← backup for data generation
WANDB_API_KEY=<your wandb key> ← for training curves
```
**Step 3 β€” Generate persona JSON files (no GPU, runs in seconds)**
```
python scripts/05_generate_personas.py
```
Output: `data/synth/persona_ananya.json`, `persona_arjun.json`, etc. (5 files)
**Step 4 β€” Download pictogram datasets (no GPU, ~1–2 hours, rate-limited)**
```
python scripts/01_download_arasaac.py ← ~5,000 PNGs from ARASAAC REST API
python scripts/02_download_mulberry.py ← ~3,500 SVGs, converted to PNG
```
Output: `data/processed/symbols_canonical.parquet` (~8,000 symbols)
**Step 5 β€” Sign up for academic audio corpora (do this Day 1, takes 2–5 days to approve)**
```
TORGO: http://www.cs.toronto.edu/~complingweb/data/TORGO/torgo.html
UA-Speech: https://speechlab.engr.uic.edu/CMU_DOC.php
```
While waiting for approval, generate synthetic audio:
```
python scripts/03_download_audio_corpora.py --synth-only --n-synth 500
```
**Step 6 β€” Generate all synthetic training data (no GPU, ~3–5 hours, API calls)**
```
python scripts/04_generate_synth_dialogues.py --api gemini --all
```
This calls Gemini 1.5 Pro API in batches of 50. Costs roughly $15–25 total.
Output: 5 persona JSONL files Γ— 3,000 records each + 1,500 base records = 16,500 pairs
**Step 7 β€” Build evaluation sets (no GPU, runs in seconds)**
```
python scripts/06_build_eval_sets.py
```
Output: 4 eval JSONL files in `data/eval/`
---
### PHASE 1 β€” H100 setup (Day 7–8)
SSH into the H100 machine. Clone the repo. Then:
**Step 8 β€” Install everything**
```
bash scripts/00_env_setup.sh
```
This installs: PyTorch 2.5.1 (CUDA 12.4), flash-attn, Unsloth, HuggingFace stack, Coqui TTS, AI4Bharat Indic-TTS. Takes 20–40 minutes. Do NOT skip.
**Step 9 β€” MANDATORY sanity check before any training**
```
python scripts/07_overfit_sanity_check.py
```
Expected: loss drops from ~5.0 to <0.5 in 200 steps.
If loss stays above 5.0: STOP. Fix the environment. Do not proceed to Stage 1.
If loss is NaN: learning rate is too high or dtype mismatch. Fix before continuing.
This check costs 30 minutes of H100 time and saves you from wasting 12 hours.
---
### PHASE 2 β€” Training on H100 (Days 8–14, ~53 hours total)
Run these sequentially. Each one produces an artifact that the next one uses.
**Step 10 β€” Stage 1: Base multimodal SFT (~12 hours)**
```
python src/ankahi/training/stage1_base.py --config configs/train_stage1_base.yaml
```
Trains on all 16,500 records. Rank-16 LoRA. Vision layers ON.
Output: `artifacts/stage1/base-gemma4-e4b-ankahi-v1/`
Monitor: wandb dashboard. Loss should sit in range 2.0–5.0. Above 15 = bug. NaN = stop.
**Step 11 β€” Stage 1 evaluation (1–2 hours)**
```
python src/ankahi/eval/run_eval.py --axis generation --no-simulate
```
Run this before persona training to confirm the base adapter actually improved over vanilla Gemma 4.
Output: `writeup/figures/eval_generation.json`
**Step 12 β€” Stage 2: Train all 5 persona LoRA adapters (~10 hours total, 2h each)**
```
python src/ankahi/training/stage2_persona.py --all --base-adapter artifacts/stage1/base-gemma4-e4b-ankahi-v1
```
Or run each separately if you want to monitor per-persona:
```
python src/ankahi/training/stage2_persona.py --persona ananya
python src/ankahi/training/stage2_persona.py --persona arjun
... (repeat for priya, rohan, zara)
```
Critical: rank is hardcoded to 8. Do not change. MediaPipe Android only supports rank [4, 8].
Output: `artifacts/stage2/arjun/persona-arjun.lora/` etc. (5 folders)
**Step 13 β€” Adapter specificity evaluation (~2 hours)**
```
python src/ankahi/eval/specificity.py --adapters-dir artifacts/stage2
```
This is the KEY figure for the writeup. Produces the 5Γ—5 heatmap.
Output: `writeup/figures/adapter_specificity_heatmap.png` + `.json`
What to look for: diagonal values close to 1.0, off-diagonal values 0.45–0.65. If off-diagonal is also close to 1.0, the adapters are not personalising β€” something is wrong with the data.
**Step 14 β€” Stage 3: Audio dysarthric adapter (~10 hours)**
```
python src/ankahi/training/stage3_audio.py --config configs/train_stage3_audio.yaml
```
Requires audio data in `data/processed/audio_manifest.jsonl`. If TORGO/UA-Speech not approved yet, this runs on synthetic audio only (~500 clips). Results will be weaker but demonstrable.
Output: `artifacts/stage3/audio-dysarthria.lora/`
**Step 15 β€” Stage 4: Safety tuning (~4 hours)**
```
python src/ankahi/training/stage4_safety.py --config configs/train_stage4_safety.yaml
```
Small dataset (~200 examples). Teaches the model to refuse harmful outputs and redirect.
Output: `artifacts/stage4/safety.lora/`
**Step 16 β€” Run ablations (~3 hours, run overnight alongside other work)**
```
python src/ankahi/eval/run_eval.py --axis ablation --no-simulate
```
Produces the rank/vision/data-scaling tables for the writeup.
---
### PHASE 3 β€” Deployment preparation on H100 (Day 15)
**Step 17 β€” Merge all adapters and quantise to INT8 (~2 hours)**
```
python src/ankahi/deploy/merge_and_quantize.py \
--base-model unsloth/gemma-4-E4B-it \
--stage1-adapter artifacts/stage1/base-gemma4-e4b-ankahi-v1 \
--stage3-adapter artifacts/stage3/audio-dysarthria.lora \
--stage4-adapter artifacts/stage4/safety.lora \
--output-dir artifacts/deploy
```
Note: Persona adapters are NOT merged here. They stay separate (30MB each, loaded at runtime).
Output: `artifacts/deploy/ankahi-gemma4-e4b-int8/` (~5GB)
**Step 18 β€” Convert to .litertlm for Android (~2 hours)**
```
python src/ankahi/deploy/convert_to_litertlm.py \
--input-dir artifacts/deploy/ankahi-gemma4-e4b-int8 \
--output-dir artifacts/litertlm
```
Output: `artifacts/litertlm/ankahi-gemma4-e4b-v1.litertlm` (~2.5GB)
---
### PHASE 4 β€” TTS voice cloning on H100 (Day 15, ~3 hours)
**Step 19 β€” Clone parent voice for each demo persona**
```
python src/ankahi/tts/clone_parent_voice.py \
--audio recordings/arjun_mother_voice.wav \
--persona arjun \
--language hi
python src/ankahi/tts/clone_parent_voice.py \
--audio recordings/ananya_mother_voice.wav \
--persona ananya \
--language ta
```
Do this for 3 demo personas. Requires a 60-second WAV recording per caregiver.
If recruiting real families, record this at their home. Otherwise use team members' voices and disclose clearly.
Output: `artifacts/tts/voice-arjun/`, `voice-ananya/`, etc.
---
### PHASE 5 β€” Android app and benchmarking (Days 16–21, local)
**Step 20 β€” Copy model files to Android assets**
Transfer from H100 to your machine, then to the Android device:
```
scp h100:/path/to/artifacts/litertlm/ankahi-gemma4-e4b-v1.litertlm ./mobile/
scp -r h100:/path/to/artifacts/deploy/persona_adapters ./mobile/
```
Place the .litertlm file in the Android app's `models/` directory on-device.
**Step 21 β€” Build and run the Flutter app**
```
cd mobile/ankahi_flutter
flutter pub get
flutter run --device-id <your_android_device_id>
```
**Step 22 β€” On-device benchmarking**
```
python src/ankahi/deploy/benchmark_device.py --tier flagship
python src/ankahi/deploy/benchmark_device.py --tier mid
python src/ankahi/deploy/benchmark_device.py --tier budget
```
Or use `--simulate` to generate estimates for the writeup before you have all devices.
---
### PHASE 6 β€” Writeup, video, submission (Days 22–31, local)
**Step 23 β€” Fill in real numbers in technical_report.md**
Replace all simulated values in Section 5, 6, 8, 10 with actual training results from wandb and eval JSONs.
**Step 24 β€” Run unit tests**
```
pytest tests/
```
All 3 test files must pass before submission.
**Step 25 β€” Film demo video**
Follow `demo/demo_script.md`. Target runtime: 1:45. Hard stop at 2:00.
**Step 26 β€” Submit to Kaggle**
Upload to the Kaggle competition by May 17 23:00 IST (24 hours before deadline).
---
## Synthetic Data Specification β€” Exact Detail
This is the most important section. The model is only as good as this data.
---
### Dataset 1 β€” Base Shared Corpus
**What it is:** Generic Indian-child AAC utterances, not tied to any specific persona. Teaches the model the core task: pictogram sequence β†’ natural multilingual sentence.
**How many:** 1,500 records
**Format:** JSONL, one JSON object per line
```json
{
"seq": ["water", "want"],
"target": "Mujhe pani chahiye",
"alternatives": ["Paani do please", "I want water"],
"negative": "The water is on the table",
"context": "afternoon",
"persona_id": "base"
}
```
**Field definitions:**
- `seq`: list of 1–4 pictogram concept strings in English (always English, since ARASAAC labels are English). Example: `["mother", "food", "want"]`
- `target`: the PRIMARY sentence the child intends. This is what the model is trained to predict as option 1. Must be in the primary language or natural code-switch. NOT English-only unless the persona is English-primary.
- `alternatives`: exactly 2 alternative phrasings of the same intent, ranked by likelihood. Less common phrasing or different language register.
- `negative`: a plausible-sounding but WRONG output β€” either wrong intent, inappropriate register, or adult-level vocabulary. Used for safety calibration.
- `context`: one of these values only: `breakfast`, `lunch`, `dinner`, `school`, `bedtime`, `morning`, `physiotherapy`, `playing`, `watching_TV`, `temple_visit`, `shopping`, `bath_time`, `doctor_visit`, `talking_to_grandparent`
**Distribution across contexts:** roughly equal (about 100 records per context across the 1,500)
**Language distribution:** 40% Hindi, 15% Bengali, 15% Tamil, 10% Telugu, 10% Marathi, 10% English-heavy code-switch
---
### Dataset 2 β€” Persona Corpus (5 datasets, one per child)
**What it is:** Per-child dialogue pairs that capture THAT child's vocabulary, language mix, family references, and favourite things. This is the data the persona LoRA adapter is trained on. It must be different from the base corpus β€” more specific, more personal.
**How many:** 3,000 records per persona = 15,000 total
**Format:** identical to base corpus JSONL, but with persona_id filled:
```json
{
"seq": ["mother", "food", "want"],
"target": "Mummy, mujhe paratha khana hai!",
"alternatives": ["Mama, bhookh lagi hai", "I want food please Mummy"],
"negative": "The food is ready on the table",
"context": "breakfast",
"persona_id": "arjun"
}
```
**What makes each persona corpus DIFFERENT β€” the specific details to encode:**
**Arjun (Ludhiana, 9yo, Punjabi+Hindi+English, dyskinetic CP)**
- Language: Punjabi pronouns ("mujhe" often becomes "mainu"), Hindi verbs ("khana", "jaana"), English nouns for school/technology ("school", "tablet", "TV", "homework")
- Family: "Mummy" for mother (NOT "Mama" or "Amma"), "Papa" for father, "Dadi" for grandmother (she lives with them)
- Food references: paratha, lassi, aloo sabzi, kadhi, dal makhani. NOT dosa or idli (those are Ananya's)
- Activities: cricket (he watches on TV), toy cars, going to the Gurudwara on Sunday mornings
- Time anchors: physiotherapy at 3pm (he often resists it), school at 9am, evening walk with Papa
- Sentence length: medium (7–12 words). He is 9 and has higher vocabulary than the 4yo Priya
- Sentence register: assertive, sometimes demanding. He is older and knows what he wants.
- Typical code-switch pattern: `[Punjabi subject/pronoun] + [Hindi verb phrase] + [English object noun]`
- Example: "Mainu cricket dekhna hai TV te" / "Mummy, mujhe aaj school nahi jaana" / "Dadi, mera tablet kahan hai?"
**Ananya (Chennai, 6yo, Tamil+English, spastic quadriplegia, eye-gaze input)**
- Language: Tamil base ("amma" not "mummy", "appa" not "papa", "paati" for grandmother)
- She uses eye-gaze β€” sentences tend to be SHORT (4–7 words) because selecting more than 2 symbols is physically tiring
- Food: idli, dosa, mango juice, curd rice. Never wheat-based foods (family is South Indian)
- Activities: listening to music (Tamil film songs), Chuchu TV, being read to by Amma
- Sentence register: soft, often ends in a Tamil particle. "Amma, thaagam aagudhu" (Amma, I'm thirsty)
- She does not use English much at home β€” it appears only for school subjects
- Do NOT generate long sentences. Maximum 6 words in the primary language.
**Priya (Kolkata, 4yo, Bengali+English, spastic CP + CVI β€” cortical visual impairment)**
- She is 4. Sentences are VERY short (3–5 words). She cannot process complex grammar yet.
- Language: Bengali base. English only appears for words she learned from Dora the Explorer.
- Food: mishti doi (sweet yoghurt), sandesh, rice with dal, banana. Very simple foods.
- CVI means she struggles with visually complex pictograms β€” she uses primarily 1 or 2 symbols
- Key reference: "Didu" (her grandmother) is extremely important. She asks for Didu constantly.
- Sentences: "Didu aasho" (Didu come), "Khudhho peye-chi" (I am hungry), "Na na na" (no)
- Do NOT generate 3-symbol sequences for Priya β€” she uses 1 or 2 symbols only. Most sequences should be `["single_symbol"]` or `["symbol1", "symbol2"]`
**Rohan (Delhi, 11yo, Hindi+English, athetoid CP)**
- He is 11 and has the highest vocabulary and most complex sentences of all personas
- Language: Hindi base with heavy English mixing. He is in school and thinks in Hindi-English
- Food: chole bhature, Maggi, mango, chips. Older child tastes.
- Activities: YouTube (science videos), chess, listening to Bollywood music, talking to friends
- He has had an AAC app before β€” he is faster and more efficient at communication
- Sentences can be longer (8–15 words). He can express complex needs.
- He sometimes expresses opinions, not just needs: "Mujhe woh movie boring lagi" (I found that movie boring)
- He uses school vocabulary: "teacher ne homework diya hai" (teacher gave homework)
**Zara (Pune, 7yo, Marathi+English+Hindi, spastic CP mild)**
- Marathi base, but Hindi appears from TV and Hindi films (she loves Bollywood)
- English appears at school
- Food: puran poli, misal pav, shrikhand, vada pav. Marathi festival foods appear in context.
- Father is primary caregiver (not mother). "Baba" is who she calls first, not "Aai"
- She is mildly affected β€” she has more physical capability than other personas. Can select 3–4 symbols.
- She has a younger sister she plays with β€” sister is named Riya in the data.
- Activities: drawing (she is creative), Marathi songs, playing with Riya
**Context distribution per persona corpus:**
Each 3,000-record corpus should cover ALL 14 contexts listed above. Roughly 200 records per context. Do NOT generate 3,000 breakfast records and 0 bedtime records.
---
### Dataset 3 β€” Audio Disambiguation Dataset
**What it is:** Pairs of (pictogram sequence, audio transcript hint) β†’ target sentence where the audio CHANGES the meaning from what the pictogram alone would predict.
**How many:** 200 records (eval only β€” this is not a training set, it is for measuring the audio adapter's performance)
**Format:**
```json
{
"persona_id": "arjun",
"pictogram_sequence": ["water", "want"],
"audio_transcript_hint": "thanda",
"target_sentence": "Mujhe thanda pani chahiye",
"without_audio_prediction": "Mujhe pani chahiye",
"intent_class": "request_food_drink"
}
```
**Key principle:** The `audio_transcript_hint` is a partial word or syllable that the child vocalises. It must be something that CHANGES the sentence. "Thanda" (cold) changes "I want water" to "I want cold water". If the audio hint doesn't change the prediction, it's not a useful eval example.
**Intent classes distribution (40 records each):**
- `request_food_drink` β€” "thanda" / "zyada" (more) / "nahi" (not this one)
- `report_discomfort` β€” "pet" (stomach) / "sir" (head) / locator sounds
- `request_activity` β€” "bahar" (outside) / "TV" / "Didi" (sister)
- `social_emotional` β€” "pyaar" / "hug" / caregiver name sound
- `personal_care` β€” "jaldi" (quickly) / "nahi" (not now)
---
### Dataset 4 β€” Safety Refusal Dataset
**What it is:** Pairs of (bad pictogram sequence, correct refusal/redirect response). Training data for Stage 4 safety tuning.
**How many:** 200 refusal examples + 1,000 normal (safe) examples to prevent over-refusal
**Format:**
```json
{
"seq": ["hurt", "self"],
"target": "It sounds like you might be in pain. Let me get someone to help you right away.",
"is_refusal": true,
"refusal_category": "self_harm_risk"
}
```
**Refusal categories (roughly 40 records each):**
- `self_harm_risk` β€” sequences involving hurt + self
- `secrecy_facilitation` β€” sequences that suggest "don't tell" patterns
- `inappropriate_adult` β€” accidental adult-content sequences from motor errors
- `dangerous_request` β€” asking for sharp objects, medicines without context
- `coercive_toward_others` β€” sequences that could be used to manipulate caregivers
**Normal examples (1,000 records):** Valid expressions of frustration, anger, sadness, pain β€” things a real child would need to communicate that a naive safety system might over-refuse. Examples: "No stop", "I hate this", "I am angry", "That hurts", "I don't want to". All of these must produce helpful predicted sentences, not refusals.
**Critical balance:** For every 1 refusal example, include 5 normal examples. This prevents the model becoming a "refusal machine" that blocks legitimate communication.
---
### Dataset 5 β€” Adapter Specificity Eval Set
**What it is:** 100 FIXED prompts applied to ALL 5 persona adapters to measure cross-persona style leakage.
**How many:** 100 records (eval only, not training)
**Format:**
```json
{
"prompt_id": "spec_042",
"pictogram_sequence": ["mother", "want"],
"context": "morning",
"expected_persona_styles": {
"ananya": "Tamil, short, 'Amma' not 'Mummy'",
"arjun": "Punjabi-Hindi-English mix, 'Mummy'",
"priya": "Bengali, very short, 4yo register",
"rohan": "Hindi-English, longer sentence, 11yo",
"zara": "Marathi, 'Aai' or 'Baba'"
}
}
```
These 100 prompts are identical for all 5 adapters. The test is: does Arjun's adapter generate "Mummy, mainu kuch khana chahida" while Ananya's adapter generates "Amma, enakku enna vendum"? If both generate the same English sentence, the adapters are not working.
---
### Summary Table β€” All Data
| Dataset | Records | Used for | Generated by |
|---|---|---|---|
| Base shared corpus | 1,500 | Stage 1 training | Gemini API |
| Ananya persona | 3,000 | Stage 2 training | Gemini API |
| Arjun persona | 3,000 | Stage 2 training | Gemini API |
| Priya persona | 3,000 | Stage 2 training | Gemini API |
| Rohan persona | 3,000 | Stage 2 training | Gemini API |
| Zara persona | 3,000 | Stage 2 training | Gemini API |
| Pictogram eval (held-out) | 500 | Eval only | Subsampled from above |
| Audio disambiguation | 200 | Eval only | Manual + synthetic |
| Visual grounding | 150 | Eval only | Manual |
| Safety refusals | 200 | Stage 4 training | Hand-crafted |
| Normal (anti-over-refusal) | 1,000 | Stage 4 training | Gemini API |
| Adapter specificity | 100 | Eval only | Hand-crafted |
| **TOTAL training records** | **~16,700** | | |
| **TOTAL eval records** | **~950** | | |
---
## One-Line Description Per Training Stage
| Stage | What it does | Hours on H100 |
|---|---|---|
| Stage 0: Sanity check | Verify pipeline works on 32 toy examples | 0.5h |
| Stage 1: Base SFT | Teach Gemma 4 the AAC task across all languages and personas | 12h |
| Stage 2: Persona LoRA Γ—5 | Give each child their own vocabulary and language style | 10h |
| Stage 3: Audio adapter | Teach the model to understand CP vocalizations | 10h |
| Stage 4: Safety tuning | Teach refusals; prevent over-refusal | 4h |
| Stage 5: Merge + quantise | Compress everything into a 2.5GB on-device file | 4h |
| Stage 6: TTS voice cloning | Clone 3 parent voices from 60s recordings | 3h |
| **Total** | | **~43–53h** |