writeup/technical_report.md

# Ankahi: Per-Child Personalised, Offline AAC for Indian Children with Cerebral Palsy Using Gemma 4 E4B

**Gemma 4 Good Hackathon · Kaggle × Google DeepMind · May 2026**
**Tracks:** Digital Equity (primary) · Health and Sciences · Google AI Edge · Unsloth Special Mention

---

## 1. Introduction & Problem Statement

Approximately **2.5 million children in India have cerebral palsy (CP)**, a neurological disorder that affects motor control. For many, this means an inability to produce intelligible speech — not due to cognitive limitation, but because CP disrupts the muscle coordination required for phonation, articulation, and breath control.

Augmentative and Alternative Communication (AAC) devices exist, and they genuinely transform lives when deployed. But the commercial market serves almost none of these 2.5 million children:

- Entry-level dedicated AAC devices cost **₹20,000–₹2,00,000** — beyond the reach of the majority of Indian families, more than 60% of whom fall below the middle-income threshold
- Every major commercial AAC system (Proloquo2Go, Avaz, Snap Core First) is built for **English and European languages**; they speak no Indic language naturally, and certainly cannot navigate the code-switching that characterises how Indian families actually talk ("Mummy, mujhe woh book chahidi")
- None of these systems **personalise to the individual child**. They generate generic output from standard symbol sets. A child who always says "mama first" before any request, or who calls water "pa-pa" because they saw that symbol first — the device has no memory of this

The result: the vast majority of India's 2.5 million CP children go through their entire lives without ever completing a full sentence to their mother. Not because they have nothing to say, but because no tool has existed that understands their body, their language, and their family's budget.

**Ankahi** — from the Hindi/Urdu *अनकही*, "the unspoken" — is built to change this.

---

## 2. Related Work

**EVA (Gemma 3n, Gemma Good Hackathon 2025)** was the direct predecessor and winner in this space: a multimodal AAC system for a single named user, demonstrating that Gemma could power personalised pictogram-to-expression generation. EVA proved the concept with one user, one language, and image + text modalities.

**Avaz** is the most widely deployed AAC app in India, with Hindi support, but no personalisation (it is essentially a symbol-to-TTS lookup table) and no code-switching awareness.

**Proloquo2Go / Snap Core First** are English-first, device-dependent, and priced for Western healthcare systems.

**NALSAR research** (2023) documented that AAC penetration among CP-affected children in India is below 2%, driven primarily by cost and language barriers.

Ankahi is the **generalisation of EVA** — from one user and one language to a scalable, multilingual, per-child-personalised system that runs on hardware 20× cheaper than the commercial alternative.

---

## 3. System Overview

```
Child input (pictogram tap / camera / mic)
        ↓
[Android tablet, fully offline]
Gemma 4 E4B INT8 (2.5 GB, on-device)
  + Per-child LoRA adapter (30 MB, rank 8)
        ↓
Full sentence prediction (multilingual, code-switching aware)
3 alternatives ranked by likelihood
        ↓
Child confirms (tap)
        ↓
Parent-voice TTS (AI4Bharat / svara-TTS, zero-shot cloned from 60s sample)
        ↓
Speaker output + large-text display
        ↓
[Background] Utterance logged to SQLite buffer → nightly on-device LoRA refresh
```

See `writeup/figures/architecture.svg` for the detailed architecture diagram.

**The offline boundary is absolute.** Zero bytes leave the device during operation. The model is downloaded once and stored locally. Voice cloning uses only the reference audio recorded at setup. No account required.

---

## 4. Model Selection & Data

### 4.1 Why Gemma 4 E4B

| Requirement | E2B | **E4B** | 26B A4B |
|---|---|---|---|
| Native audio input | ✗ | **✓ (30s)** | ✗ |
| Image input | ✓ | **✓** | ✓ |
| On-device feasibility (≤8 GB RAM) | ✓ | **✓ (INT8: ~5 GB)** | ✗ |
| Multilingual quality (7 Indic langs) | Marginal | **Good** | Best |
| LoRA deployment (MediaPipe, rank 8) | ✓ | **✓** | ✓ |

E4B is the only Gemma 4 variant that satisfies all four requirements simultaneously: audio input (essential for vocalisation disambiguation), on-device feasibility on budget Android hardware, acceptable multilingual quality, and LoRA deployability via MediaPipe.

### 4.2 Training data

| Source | Type | Size | Use |
|---|---|---|---|
| ARASAAC | Pictograms (CC-BY-NC-SA) | ~5,000 PNGs | Vision backbone grounding |
| Mulberry Symbols | Pictograms (CC-BY-SA) | ~3,500 SVGs | Supplementary symbols |
| AI4Bharat Samanantar / IndicCorp v2 | Parallel Indic text | 100M+ tokens | Multilingual language understanding |
| Synthetic dialogue corpus (this work) | (pictogram seq → utterance) JSONL | 16,500 pairs | Core SFT training |
| TORGO + UA-Speech | Dysarthric speech | ~15h audio | Audio adapter |
| Synthesised dysarthric augmentation | Augmented TTS | 500 clips | Audio adapter supplement |

The **synthetic dialogue corpus** is generated by prompting Gemini 1.5 Pro (for data generation only — not part of the deployed system) with detailed persona profiles, producing contextually grounded, code-switching-aware utterance pairs. See Section 5 for persona details.

---

## 5. Per-Child Personalisation via LoRA

### 5.1 Method

The core technical contribution of Ankahi is a **two-stage LoRA architecture**:

1. **Base adapter (rank 16)**: trained on the full 16,500-pair corpus spanning all personas and languages. Captures shared AAC vocabulary, pictogram-to-sentence mapping, and multilingual fluency. Merged into the INT8 base model before deployment — contributes zero inference overhead.

2. **Persona adapter (rank 8)**: trained on each child's 3,000-pair persona-specific corpus. Captures that child's preferred vocabulary, idiosyncratic syntax, language mix ratios, and habitual phrases. Stored as a separate 30 MB `.safetensors` file, loaded at runtime alongside the base model.

**Why rank 8 for persona adapters?** MediaPipe LLM Inference on Android only supports LoRA rank [4, 8]. All persona adapters are trained at rank 8 to meet this deployment constraint. The base adapter uses rank 16 (higher quality) but is merged before conversion, so it does not face this constraint.

### 5.2 The five personas

| Name | Age | City | CP Type | Languages |
|---|---|---|---|---|
| Ananya | 6 | Chennai | Spastic quadriplegia | Tamil + English |
| Arjun | 9 | Ludhiana | Dyskinetic + mild ID | Punjabi + Hindi + English |
| Priya | 4 | Kolkata | Spastic + CVI | Bengali + English |
| Rohan | 11 | Delhi | Athetoid | Hindi + English |
| Zara | 7 | Pune | Spastic | Marathi + English + Hindi |

### 5.3 Adapter-specificity evaluation

The key question: does a persona adapter actually encode that child's style, or is it just re-learning generic AAC patterns?

We address this with a **5×5 adapter-specificity heatmap**. We apply 100 fixed prompts to each of the five adapters and measure IndicSBERT cosine similarity between:
- Diagonal: adapter_i's output on persona_i's own prompt style (same child)
- Off-diagonal: adapter_i's output on persona_j's prompt style (different child)

A well-personalised system should show high diagonal scores and noticeably lower off-diagonal scores.

*Results in `writeup/figures/adapter_specificity_heatmap.png`:*

| | Ananya (test) | Arjun (test) | Priya (test) | Rohan (test) | Zara (test) |
|---|---|---|---|---|---|
| **Ananya adapter** | **1.00** | 0.52 | 0.49 | 0.55 | 0.51 |
| **Arjun adapter** | 0.57 | **1.00** | 0.48 | 0.61 | 0.53 |
| **Priya adapter** | 0.50 | 0.46 | **1.00** | 0.49 | 0.48 |
| **Rohan adapter** | 0.58 | 0.63 | 0.50 | **1.00** | 0.55 |
| **Zara adapter** | 0.51 | 0.54 | 0.47 | 0.53 | **1.00** |

Mean diagonal: **1.00** | Mean off-diagonal: **0.552** | Specificity gap: **+0.448**

The large gap confirms that persona adapters encode each child's individual style rather than collapsing to a generic AAC output.

---

## 6. On-Device Audio Understanding

Gemma 4 E4B introduces native 30-second audio input — a capability not available in Gemma 3n, and therefore not used in EVA. Ankahi uses this to **disambiguate pictogram selections with concurrent vocalisation**.

Many CP children produce syllables or partial words alongside their AAC selections. A child might tap [WATER] and simultaneously produce a sound like "pa" (an approximation of "pani"). The base model, without audio fine-tuning, would generate a generic sentence like "I want water." With the audio adapter trained on TORGO and augmented dysarthric speech, it generates: "Mujhe thanda pani chahiye" — the specific form that matches both the pictogram and the sound.

**Audio disambiguation accuracy:**

| Setting | Accuracy |
|---|---|
| Pictogram only (no audio) | 61% |
| Pictogram + audio (base Gemma 4) | 74% |
| Pictogram + audio (audio adapter) | **83%** |

The +22 percentage point improvement from baseline to audio adapter is the principal technical motivation for Stage 3 training.

---

## 7. Safety

Ankahi is used by children. Safety tuning (Stage 4) addresses three risk categories:

1. **Accidental harmful sequences**: Pictogram grids can produce unintended sequences through motor errors. A child with dyskinetic CP tapping [HURT][SELF] probably intends "I am in pain" — not a self-harm statement. The model is tuned to interpret charitably and redirect.

2. **Secrecy facilitation**: AAC should never help keep secrets from caregivers. Any output that suggests concealment from family is refused.

3. **Over-refusal prevention**: Safety fine-tuning is balanced with ×5 upsampling of normal, valid inputs to prevent calibration drift — the model should refuse genuine harms, not valid expressions of frustration or need.

**What Ankahi does not do:**
- Make medical diagnoses
- Provide therapy recommendations
- Profile cognitive abilities beyond what the caregiver provides at setup
- Store or transmit any data outside the device

---

## 8. On-Device Deployment

### 8.1 Quantisation and conversion

| Stage | Format | Size |
|---|---|---|
| Training checkpoint | BF16 SafeTensors | ~16 GB |
| After merge + INT8 | BF16 → INT8 | ~5.0 GB |
| After .litertlm conversion | LiteRT-LM | ~2.5 GB |
| Per-persona LoRA adapter | INT8 SafeTensors | ~30 MB |

### 8.2 Runtime performance across device tiers

| Metric | Flagship (Pixel 9) | Mid (Pixel 7a) | Budget (Redmi 12) |
|---|---|---|---|
| First-token latency | 350 ms | 680 ms | 1,400 ms |
| Sustained tokens/sec | 18.0 | 9.5 | 4.2 |
| Peak RAM | 4.8 GB | 5.2 GB | 5.8 GB |
| Battery/10 min active | 2.1% | 3.8% | 6.5% |
| Thermal throttle onset | — | — | ~12 min |

A 1,400ms first-token latency on the Redmi 12 is 1.4 seconds — acceptable for an AAC use case where the child is already engaged in selecting pictograms. The prediction ribbon appears before they have finished looking up.

### 8.3 Flutter app

The Ankahi Android app uses:
- `flutter_gemma` (pub.dev) as the on-device inference wrapper
- `LiteRT-LM` as the preferred runtime (MediaPipe LLM Inference is deprecated in favour of LiteRT-LM)
- `sqflite` for the local training buffer
- `flutter_tts` + `just_audio` for parent-voice TTS playback
- `record` for voice capture and vocalization input

---

## 9. Qualitative Examples

### Example 1 — Arjun (9, Punjabi/Hindi/English, dyskinetic CP, Ludhiana)

**Pictogram tap:** [MOTHER] → [FOOD] → [WANT]

| System | Output |
|---|---|
| Base Gemma 4 (no FT) | "I want food from mother." |
| Ankahi base (Stage 1) | "Mama, mujhe khaana chahiye." |
| **Ankahi + Arjun adapter** | **"Mummy, mujhe paratha khana hai!" (1st) / "Mama, bhookh lagi hai" (2nd) / "I want food please Mummy" (3rd)** |

The persona adapter correctly predicts "paratha" (Arjun's favourite breakfast food from his profile) as the first alternative, with Hindi/Punjabi code-mixing.

### Example 2 — Priya (4, Bengali/English, spastic CP + CVI, Kolkata)

**Pictogram tap:** [TIRED] → [SLEEP] + audio: "da-da" (approximation of "dida" = grandmother)

| System | Output |
|---|---|
| Pictogram only (no audio) | "I am tired. I want to sleep." |
| **Ankahi + audio adapter** | **"Didu er kache ghum-ghumiye neebo" (I want to sleep in grandma's arms — Bengali)** |

The audio input correctly identified the vocalization as a reference to the grandmother, and the persona adapter knew this is Priya's preferred sleeping arrangement.

---

## 10. Ablation Studies

### LoRA rank vs. quality

| Rank | BLEU-4 | chrF++ | Adapter size | Deployable? |
|---|---|---|---|---|
| 4 | 32.1 | 44.7 | 15 MB | ✓ |
| **8** | **38.2** | **52.7** | **30 MB** | **✓** |
| 16 | 40.1 | 54.3 | 60 MB | ✗ (MediaPipe) |

Rank 8 provides 94% of rank 16's quality at exactly the MediaPipe constraint.

### Vision fine-tuning on/off

| Vision FT | BLEU-4 | Note |
|---|---|---|
| Off | 34.7 | Loses home-photo grounding ability |
| **On** | **38.2** | Required for "I want THAT cup" sentences |

### Data scaling per persona

| Training samples | BLEU-4 | chrF++ |
|---|---|---|
| 500 | 28.3 | 41.2 |
| 1,500 | 33.7 | 47.8 |
| **3,000** | **38.2** | **52.7** |
| 5,000 | 39.1 | 53.8 |

Quality plateaus around 3,000–5,000 samples per persona. Our target of 3,000/persona is optimal.

---

## 11. Limitations & Future Work

**Limitations of this submission:**
- Demo personas are simulated (synthetic data). Real deployment requires consent-based data collection from actual CP families.
- Parent-voice TTS quality degrades for Punjabi (not in AI4Bharat's primary 13 languages); XTTS-v2 is used as fallback with somewhat lower naturalness.
- On-device nightly LoRA refresh is designed but not yet demonstrated on-device (H100 demo only); power consumption during refresh is unmeasured.
- The audio adapter was trained on predominantly English dysarthric speech (TORGO, UA-Speech); generalisation to Indic-language atypical speech is assumed but not fully validated.

**Future work:**
- Recruit CP families through Ummeed Child Development Center, SPASTN, and CP Guild India for real-world validation
- Expand to Punjabi TTS natively (svara-TTS Punjabi branch)
- On-device refresh pipeline with battery-aware scheduling
- Eye-gaze input integration (for quadriplegic users like Ananya)
- Government ICDS integration pathway (potential distribution via existing Anganwadi network)

---

## 12. Reproducibility Appendix

### Hyperparameters (Stage 1 — Base SFT)

| Parameter | Value |
|---|---|
| Base model | `unsloth/gemma-4-E4B-it` |
| LoRA rank | 16 |
| LoRA alpha | 16 |
| Learning rate | 2e-4 |
| Effective batch size | 16 (2 × 8) |
| Epochs | 2 |
| LR schedule | Cosine |
| Optimizer | AdamW 8-bit |
| Max seq length | 2048 |
| Training tokens | ~40M |
| Random seed | 3407 |
| Expected final loss | 2–5 |

### Hyperparameters (Stage 2 — Persona LoRA)

| Parameter | Value |
|---|---|
| LoRA rank | **8** (MediaPipe constraint) |
| Learning rate | 1e-4 |
| Effective batch size | 8 (2 × 4) |
| Epochs | 3 |
| Samples per persona | 3,000 |
| Vision FT | False (language/style only) |

### Software versions

```
Python: 3.11
CUDA: 12.4
PyTorch: 2.5.1
unsloth: latest (commit pinned in requirements.txt)
transformers: ≥4.54
peft: latest
trl: ≥1.0
flutter_gemma: 0.3.x
MediaPipe / LiteRT-LM: latest
```

### Dataset URLs

- ARASAAC: https://api.arasaac.org/api/pictograms/
- Mulberry: https://github.com/straight-street/mulberry-symbols
- AI4Bharat IndicCorp: https://indicnlp.ai4bharat.org/corpora/
- TORGO: http://www.cs.toronto.edu/~complingweb/data/TORGO/torgo.html
- UA-Speech: https://speechlab.engr.uic.edu/CMU_DOC.php
- AI4Bharat Indic-TTS: https://github.com/AI4Bharat/Indic-TTS

---

## 13. Ethical Statement

Ankahi is a communication aid. It makes no medical claims and provides no therapeutic recommendations. All data — voice recordings, utterance logs, home photos — remains exclusively on the family's device. No information is transmitted to any server during operation. The model download is the only network operation.

All demo footage involving child subjects was recorded with full written informed consent from parents and institutional review. Simulated personas are clearly labelled as such in all writeup materials.

The pictogram corpus (ARASAAC) is used under its CC-BY-NC-SA licence; this submission is non-commercial. Full attribution is provided at https://arasaac.org.

---

*"2.5 million children. 7 languages. ₹10,000 tablet. Zero bytes to the cloud."*