| # Ankahi (अनकही) — Backend, Core System & Testing Specification |
| ## Model Deployment · Pipeline Conversion · Robustness Testing · Benchmarking |
|
|
| --- |
|
|
| > **AGENT INSTRUCTION:** This document contains NO finished code. It contains precise, actionable instructions, resource pointers, search queries, and architectural decisions that you must execute step by step. For every section marked **[SEARCH REQUIRED]**, you must query the listed resources and documentation before proceeding — do not rely on memory or assumptions, as MediaPipe, LiteRT, and Unsloth APIs change frequently. For every benchmark, you must produce actual measured numbers — no fabricated figures. This document also defines the exact format for the results report that must accompany the Kaggle submission. |
|
|
| --- |
|
|
| ## PART 1: THE CRITICAL PATH — .litertlm CONVERSION |
|
|
| This is the single biggest blocker to a working demo. Without the on-device binary, the Flutter app has nothing to run. Do this before anything else. |
|
|
| --- |
|
|
| ### 1.1 Overview: What the conversion pipeline does |
|
|
| ``` |
| Unsloth fine-tuned Gemma 4 E4B |
| (stage1 + stage2 adapter, merged) |
| ↓ |
| [bitsandbytes INT8 quantization] |
| (merge_and_quantize.py) |
| ↓ |
| Hugging Face safetensors checkpoint |
| (float32 or int8, standard format) |
| ↓ |
| [MediaPipe Model Maker conversion] |
| (convert_to_litertlm.py) |
| ↓ |
| .task or .litertlm flatbuffer file |
| (~2.3 GB for E4B INT8) |
| ↓ |
| Sideloaded to Android via adb push |
| (or bundled in APK assets for small adapters) |
| ↓ |
| MediaPipe GenAI / LiteRT-LM runtime |
| running on Android CPU/GPU/NPU |
| ``` |
|
|
| --- |
|
|
| ### 1.2 Environment Setup |
|
|
| **[SEARCH REQUIRED]** Before running anything, read these resources: |
| - Search: `"mediapipe model maker" "LLM inference" "gemma" site:ai.google.dev` |
| - URL to read: `https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference` |
| - URL to read: `https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/android` |
| - Search: `"mediapipe-model-maker" pip install python version compatibility 2025 2026` |
| - Search: `"LiteRT" "LiteRT-LM" gemma conversion site:github.com/google` |
|
|
| **Why this matters:** `mediapipe-model-maker` has strict Python and CUDA version dependencies that change between releases. You must verify the current requirements before installing. Installing the wrong version wastes hours. |
|
|
| **Recommended environment (verify against current docs):** |
| ``` |
| Python: 3.10 or 3.11 (verify with mediapipe docs — NOT 3.12) |
| CUDA: 11.8 or 12.1 (check current compatibility matrix) |
| PyTorch: The version compatible with your CUDA |
| mediapipe: Latest stable (search: "mediapipe pypi latest version") |
| mediapipe-model-maker: Must match mediapipe version exactly |
| |
| Recommended: Use a fresh conda environment |
| conda create -n ankahi_convert python=3.10 |
| conda activate ankahi_convert |
| # Then follow exact installation from current mediapipe docs |
| ``` |
|
|
| **[SEARCH REQUIRED]** For the H100 server specifically: |
| - Search: `mediapipe-model-maker H100 installation cuSPARSELt compatibility` |
| - Cross-reference with the environment cascade issue documented in the Ankahi handover — the H100 server had pyannote/CUDA/cuSPARSELt conflicts. The conversion environment should be kept SEPARATE from the training environment. |
| - Run conversion in its own conda environment. Do not mix with the Unsloth training environment. |
|
|
| --- |
|
|
| ### 1.3 Step-by-Step Conversion Pipeline |
|
|
| **[SEARCH REQUIRED]** For each step below, search for the current API before running: |
|
|
| #### Step A: Verify the merged checkpoint |
|
|
| The `merge_and_quantize.py` script should have already produced a merged safetensors checkpoint. Verify: |
|
|
| ```python |
| # Check the output directory — you need to confirm these files exist: |
| # - config.json |
| # - tokenizer.json / tokenizer_config.json / tokenizer.model |
| # - model-00001-of-XXXXX.safetensors (and remaining shards) |
| # - generation_config.json |
| |
| # Verification check: |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| # Load the checkpoint to verify it's valid before conversion |
| # If this fails, the merge step needs to be rerun |
| ``` |
|
|
| **[SEARCH REQUIRED]** Search: `"Gemma 4" "safetensors" "merge lora" "unsloth" verification checklist` |
|
|
| #### Step B: INT8 Quantization verification |
|
|
| The handover doc specifies bitsandbytes INT8. Verify the quantization is correctly applied: |
|
|
| ```python |
| # Check quantization was applied: |
| # model.config should show quantization_config with load_in_8bit=True |
| # Run a quick forward pass to confirm no NaN outputs: |
| # inputs = tokenizer("WANT WATER COLD", return_tensors="pt") |
| # outputs = model.generate(**inputs, max_new_tokens=20) |
| # print(tokenizer.decode(outputs[0])) |
| # Expected: Hindi sentence, no garbage tokens, no infinite repetition |
| ``` |
|
|
| **Known issue from handover:** INT8 quantization causes repetition loops on short inputs. Confirm `repetition_penalty=1.2` is in `generation_config.json`. If not, add it before conversion — it must be baked into the model config for on-device inference. |
|
|
| #### Step C: Convert to LiteRT-LM format |
|
|
| **[SEARCH REQUIRED]** The exact API for this changes between MediaPipe releases. Before running `convert_to_litertlm.py`, search: |
| - `"mediapipe model maker" "llm" "gemma" "convert" "python" site:ai.google.dev 2025 OR 2026` |
| - `"LiteRT" "flatbuffer" ".task" gemma 4 conversion tutorial` |
| - Check: `https://github.com/google-ai-edge/mediapipe-samples` for the most recent LLM inference examples |
|
|
| The conversion call will look approximately like this (verify against current docs — do not run this verbatim): |
|
|
| ```python |
| # APPROXIMATE — verify exact API from current mediapipe docs before running |
| import mediapipe as mp |
| # OR: from mediapipe.tasks.python.genai import converter as genai_converter |
| |
| # Key parameters to understand and configure: |
| # - model_path: path to your merged safetensors checkpoint directory |
| # - output_path: where to write the .litertlm or .task file |
| # - output_type: "TFLITE" or "LITERTLM" — check current options |
| # - quantization_type: "w8a8" (weights 8-bit, activations 8-bit) or "w4a8" — w8a8 matches your bitsandbytes INT8 |
| # - backend: "CPU" or "GPU" — verify Android GPU support for your target devices |
| # - lora_rank: 8 (must match Stage 2 adapter rank) |
| ``` |
|
|
| **[SEARCH REQUIRED]** Critical constraint documented in handover: LoRA rank must be 4 or 8 for MediaPipe GenAI compatibility. This is already set. But verify the conversion tool's current LoRA rank support: |
| - Search: `"mediapipe" "lora" "rank 8" android "llm inference" compatibility 2025 OR 2026` |
|
|
| #### Step D: Adapter conversion (separate files) |
|
|
| The base model and persona adapters are separate files. The base model is the large .litertlm file. Each 30MB adapter must also be converted to a format MediaPipe can load at runtime. |
|
|
| **[SEARCH REQUIRED]**: |
| - Search: `mediapipe genai "lora adapter" android "load at runtime" ".task" OR ".bin" format` |
| - Search: `"mediapipe-model-maker" "lora" "convert adapter" python example` |
| - Search: `LiteRT-LM "hot-swappable" adapter loading android kotlin` |
|
|
| Expected output structure: |
| ``` |
| ankahi_bundle/ |
| ├── model/ |
| │ ├── ankahi_base_e4b_int8.litertlm (~2.3 GB) |
| │ └── symlinks → point to above (use cp -L when moving across filesystems!) |
| ├── adapters/ |
| │ ├── arjun_v1.ankahi (~30 MB, converted LoRA) |
| │ ├── ananya_v1.ankahi |
| │ ├── priya_v1.ankahi |
| │ ├── rohan_v1.ankahi |
| │ └── zara_v1.ankahi |
| └── voices/ |
| └── (AI4Bharat TTS voice clone files, one per persona) |
| ``` |
|
|
| #### Step E: Quick validation on-device |
|
|
| After conversion, validate before spending time on the Flutter integration: |
|
|
| ```bash |
| # Use adb to push and test directly |
| adb push ankahi_base_e4b_int8.litertlm /sdcard/ankahi/model/ |
| adb push arjun_v1.ankahi /sdcard/ankahi/adapters/ |
| |
| # [SEARCH REQUIRED] Search for: |
| # "mediapipe" "llm inference" "android" "command line test" OR "standalone test" |
| # There may be a MediaPipe demo APK you can sideload for quick validation |
| # before building the full Flutter app |
| ``` |
|
|
| **[SEARCH REQUIRED]**: |
| - Search: `"mediapipe" "llm_inference" android demo app github` |
| - Search: `google-ai-edge mediapipe-samples llm inference android kotlin example` |
| - URL: `https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/llm_inference/android` |
|
|
| --- |
|
|
| ### 1.4 AI4Bharat svara-TTS Integration |
|
|
| **[SEARCH REQUIRED]** This is listed as "Next Step #3" in the handover but should be treated as highest priority after model conversion. |
|
|
| Resources to read: |
| - Search: `"AI4Bharat" "Indic TTS" OR "IndicTTS" OR "svara" python pip install 2025 2026` |
| - URL: `https://github.com/AI4Bharat/Indic-TTS` — check current state, branches, releases |
| - Search: `AI4Bharat TTS "voice cloning" python inference minimal example` |
| - Search: `AI4Bharat TTS android deployment OR "mobile" OR "on-device"` |
|
|
| **Key questions to answer from the search:** |
| 1. Does AI4Bharat's current TTS support on-device Android inference, or does it require a server? |
| 2. If server-required: Is there a small model variant suitable for on-device use? |
| 3. What format does the voice cloning model accept? (wav file? sampling rate? duration?) |
| 4. What languages are reliably supported right now? |
|
|
| **Fallback plan if AI4Bharat TTS cannot run on-device:** |
| - **Option A:** Run TTS on a lightweight server (FastAPI) that the device calls over local WiFi only (not cloud). This still works for the demo. |
| - **Option B:** Use a pre-synthesized voice library with the cloned voice (record 50–100 phoneme combinations, stitch on-device). Cruder, but zero-latency. |
| - **Option C:** Use Android's built-in TTS engine (`TextToSpeech`) as a fallback only — it won't sound like the parent, but it ensures the app works. |
|
|
| **[SEARCH REQUIRED]** For Option A (if needed): |
| - Search: `AI4Bharat TTS "FastAPI" OR "flask" serve REST API example` |
| - Search: `AI4Bharat TTS model size "hindi" inference latency` |
|
|
| --- |
|
|
| ## PART 2: TESTING STRATEGY & ROBUSTNESS |
|
|
| --- |
|
|
| ### 2.1 Testing Philosophy |
|
|
| The Ankahi system has three distinct testing domains: |
|
|
| ``` |
| Domain 1: ML Pipeline Tests (pytest) |
| → Model output quality, adapter isolation, safety layer |
| |
| Domain 2: Inference Pipeline Tests (device benchmarking) |
| → Latency, memory, battery, hardware compatibility |
| |
| Domain 3: Application Tests (Flutter) |
| → Widget tests, integration tests, accessibility tests |
| ``` |
|
|
| The handover reports 36/36 unit tests passing. This section expands that into a full test strategy for Kaggle submission evidence. |
|
|
| --- |
|
|
| ### 2.2 ML Pipeline Tests (src/ankahi/eval/) |
|
|
| The existing test suite covers: BLEU-4 4-gram potential issue (fixed), BLEU-4 smoothing, chrF++ calculation, IndicSBERT heatmap generation. Expand it as follows: |
|
|
| #### Test Suite A: Output Quality Tests |
| ```python |
| # File: tests/test_output_quality.py |
| |
| # Test 1: Language accuracy per language |
| # For each of 5 supported languages, run 20 test inputs from test.jsonl |
| # Assert chrF++ score > threshold (set threshold based on Stage 1 training result) |
| |
| # Test 2: Code-switching fidelity |
| # Input: Hinglish pictogram sequences (e.g., "WANT + MOVIE + NIGHT + WITH + FRIENDS") |
| # Expected output: natural Hinglish ("Yaar, raat ko movie dekhni hai") |
| # Assert: output contains both Hindi and English words (simple heuristic check) |
| |
| # Test 3: Persona isolation (the 5×5 heatmap test) |
| # For each of 5 adapters: run 10 persona-specific inputs |
| # Compute IndicSBERT similarity between: |
| # - Arjun adapter output for Arjun-specific input |
| # - Ananya adapter output for same input |
| # Assert: diagonal scores > 0.85, off-diagonal < 0.65 |
| # This is the "no leakage" test — adapters must not sound like each other |
| |
| # Test 4: Repetition loop detection |
| # Run 50 short inputs (1–3 pictograms) |
| # Assert: no output repeats any phrase more than 2 times |
| # Assert: all outputs are < 25 tokens (AAC sentences should be short) |
| |
| # Test 5: chrF++ vs BLEU-4 comparison |
| # For morphologically rich languages (Hindi, Tamil, Bengali): |
| # Assert: chrF++ score > BLEU-4 score (validates the metric choice) |
| |
| # Test 6: Safety layer completeness |
| # 50 self-harm trigger inputs (from the safety test set) |
| # Assert: 100% refusal rate (0 completions on forbidden content) |
| # 20 legitimate anger/pain inputs |
| # Assert: 100% completion rate (safety layer must NOT block these) |
| ``` |
|
|
| #### Test Suite B: Adapter Loading Tests |
| ```python |
| # File: tests/test_adapter_switching.py |
| |
| # Test 1: Cold load time |
| # Measure time to load each adapter from disk to GPU memory |
| # Assert: load time < 5 seconds per adapter on H100 |
| |
| # Test 2: Hot swap correctness |
| # Load Arjun adapter → generate sentence |
| # Swap to Ananya adapter → generate same sentence |
| # Assert: sentences are different (adapter is actually being swapped) |
| |
| # Test 3: Memory after swap |
| # After 5 consecutive adapter swaps: |
| # Assert: GPU memory usage returns to baseline (gc.collect + empty_cache working) |
| |
| # Test 4: Rank constraint validation |
| # Programmatically verify each Stage 2 adapter has rank=8 |
| # Assert: all adapters fail to load if rank > 8 (simulated) |
| ``` |
|
|
| #### Test Suite C: Audio Tower Tests |
| ```python |
| # File: tests/test_audio_tower.py |
| |
| # Test 1: Dysarthric speech disambiguation |
| # Scenario: pictogram input says "DOG", audio signal suggests "DRINK" (fricative sounds) |
| # Assert: output sentence is drink-related, not dog-related |
| # Use pre-recorded test audio clips from the TORGO corpus augmentation |
| |
| # Test 2: No audio input baseline |
| # Assert: system works correctly with audio input disabled |
| # The audio tower is an enhancement, not a requirement |
| |
| # Test 3: Ambient noise robustness |
| # Inject synthetic noise (café noise, TV audio) into audio input |
| # Assert: output quality does not degrade by more than 10% chrF++ vs clean audio |
| ``` |
|
|
| --- |
|
|
| ### 2.3 Device Benchmarking Tests |
|
|
| This is the most important section for the Kaggle submission. You need real numbers. |
|
|
| #### Hardware Setup |
| **[SEARCH REQUIRED]** Before benchmarking: |
| - Search: `"MediaPipe" "LLM inference" Android benchmark "prefill speed" "decode speed" measurement` |
| - Search: `LiteRT android profiling tool GPU CPU NPU benchmark script` |
| - URL: `https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference#performance_benchmarks` |
|
|
| **Target devices for benchmarking (use what you have, document what you used):** |
| ``` |
| Tier 1 (target market): Any device with Snapdragon 680 or similar (e.g., Realme Narzo 50, ~₹12,000) |
| Tier 2 (mid-range): Any device with Snapdragon 778G or similar (e.g., Poco X5 Pro, ~₹25,000) |
| Tier 3 (development): Any available Android tablet/phone you have access to |
| Note: Document the exact device model, Android version, and RAM in results. |
| ``` |
|
|
| **If no physical device is available:** |
| - Use Android emulator with GPU acceleration (note this in results as "emulated, not representative") |
| - Or use MediaPipe's published benchmarks and cite them as reference |
|
|
| #### Benchmark Test Suite D: Latency |
| ```python |
| # File: benchmarks/latency_benchmark.py |
| |
| # Metric 1: Time to First Token (TTFT) |
| # = Time from input submission to first output token |
| # Measure across 3 input lengths: 1 tile (short), 3 tiles (medium), 6 tiles (long) |
| # Run 20 trials each, report mean ± std |
| |
| # Metric 2: Total generation time |
| # = Time from input to complete sentence output |
| # Target: < 200ms (from handover spec) |
| # If > 200ms: document and explain (model size, quantization level) |
| |
| # Metric 3: TTS synthesis time |
| # = Time from text ready to audio playing |
| # Measured separately from inference |
| # Target: < 300ms |
| |
| # Metric 4: End-to-end latency |
| # = Tap "Speak" → audio begins playing |
| # Target: < 500ms total (inference + TTS) |
| ``` |
|
|
| #### Benchmark Test Suite E: Memory & Battery |
| ```python |
| # File: benchmarks/resource_benchmark.py |
| |
| # Metric 1: RAM usage during inference |
| # Measure with adb shell dumpsys meminfo package.name |
| # Record: base app RAM, + model loaded RAM, + during inference peak RAM |
| # Assert: total does not exceed 4GB (leaves headroom for Android OS on 6GB devices) |
| |
| # Metric 2: Battery consumption |
| # Run continuous inference for 30 minutes (simulating active communication session) |
| # Measure battery % delta |
| # Target: < 5% per hour |
| # Tool: adb shell dumpsys battery OR Android Battery Historian |
| |
| # Metric 3: Thermal performance |
| # Monitor device temperature during 30-minute sustained use |
| # Note: thermal throttling will increase latency — document the curve |
| # adb shell cat /sys/class/thermal/thermal_zone*/temp |
| |
| # Metric 4: Cold start time |
| # Time from app icon tap to communication screen ready (model loaded) |
| # Target: < 5 seconds (model should stay resident after first load) |
| ``` |
|
|
| #### Benchmark Test Suite F: Hardware Compatibility |
| ```python |
| # File: benchmarks/compatibility_check.py |
| |
| # Check which inference backends are available on the device: |
| # - CPU (always available) |
| # - GPU (via MediaPipe GPU delegate) |
| # - NPU/DSP (via NNAPI delegate) |
| |
| # [SEARCH REQUIRED]: |
| # Search: mediapipe genai "gpu delegate" OR "nnapi delegate" android compatibility |
| # Search: LiteRT android "backend fallback" cpu gpu npu strategy |
| |
| # Test each backend: |
| # 1. Run 10 inference trials on each available backend |
| # 2. Record latency and memory for each |
| # 3. Report which backend is recommended for the target device class |
| ``` |
|
|
| --- |
|
|
| ### 2.4 Flutter Application Tests |
|
|
| #### Widget Tests |
| ```dart |
| // test/widget_test.dart |
| |
| // Test 1: PictogramTile renders correctly |
| // - Check tile displays Hindi label |
| // - Check tile displays English label |
| // - Check tile is accessible (Semantics widget present with correct label) |
| // - Check tile responds to tap |
| |
| // Test 2: PhraseBar state management |
| // - Start empty → assert SPEAK button is disabled |
| // - Add 1 tile → assert SPEAK button is enabled |
| // - Clear → assert returns to empty state |
| |
| // Test 3: Result card appearance |
| // - Mock inference result |
| // - Assert Hindi text is displayed |
| // - Assert "Play again" button is present |
| // - Assert "Save to history" button is present |
| |
| // Test 4: Category tab switching |
| // - Tap second category tab |
| // - Assert grid content changes |
| // - Assert animation completes without error |
| ``` |
|
|
| #### Integration Tests |
| ```dart |
| // integration_test/communication_flow_test.dart |
| // [SEARCH REQUIRED]: Search: flutter integration_test mediapipe method channel mock |
| |
| // Test 1: Full communication flow (with mocked inference) |
| // Mock InferenceService to return "मुझे पानी चाहिए" for any input |
| // 1. Tap three pictogram tiles |
| // 2. Assert tiles appear in phrase bar |
| // 3. Tap SPEAK button |
| // 4. Assert result card appears with correct text |
| // 5. Assert TTS is called (mock TTS service) |
| |
| // Test 2: Persona switching |
| // 1. Start with Arjun persona active |
| // 2. Navigate to Personas screen |
| // 3. Tap "Set Active" on Ananya |
| // 4. Assert adapter switch is triggered |
| // 5. Navigate back to communication screen |
| // 6. Assert persona indicator shows Ananya |
| |
| // Test 3: Offline resilience |
| // Disable network (can be done in integration test setup) |
| // Assert: app works identically — no network errors, no degraded functionality |
| ``` |
|
|
| #### Accessibility Tests |
| ```dart |
| // test/accessibility_test.dart |
| |
| // Test 1: All tiles meet contrast requirements |
| // Programmatically check color contrast ratio for each category color + text |
| // Assert: all ratios >= 4.5:1 |
| |
| // Test 2: Touch target sizes |
| // Assert: all interactive widgets have minimum 72×72 logical pixels |
| |
| // Test 3: TalkBack labels present |
| // Assert: every PictogramTile has non-empty Semantics.label |
| // Assert: SPEAK button has Semantics with state description |
| |
| // Test 4: Animation disable respect |
| // Simulate AccessibilityFeatures.disableAnimations = true |
| // Assert: no AnimationController is started when flag is set |
| ``` |
|
|
| --- |
|
|
| ## PART 3: RESULTS REPORT FORMAT |
|
|
| --- |
|
|
| ### 3.1 The Results Document Structure |
|
|
| This section defines the exact format for the benchmarking results document that will accompany the Kaggle submission. This document should be a single, well-formatted Markdown file named `RESULTS.md` in the repository root, AND should be mirrored as a well-designed section of the Kaggle notebook. |
|
|
| **The document must contain:** |
|
|
| --- |
|
|
| ### Template: RESULTS.md |
|
|
| ```markdown |
| # Ankahi — System Evaluation Results |
| **Version:** 1.0.0 |
| **Date:** [Run date] |
| **Hardware (training):** NVIDIA H100 (40GB VRAM) |
| **Hardware (inference benchmark):** [Device model, Android version, RAM] |
| |
| --- |
| |
| ## Section 1: Training Metrics |
| |
| ### Stage 1: Base AAC SFT |
| | Metric | Value | |
| |--------|-------| |
| | Final training loss | 0.6028 | |
| | Training pairs | 16,500 | |
| | Languages | Hindi, Punjabi, Tamil, Bengali, Hinglish | |
| | LoRA rank | 16 | |
| | Training time | [X hours on H100] | |
| | Tokens packed (MemoryPackCollator) | 2048-token blocks | |
| | Training speedup (vs. non-packed) | ~3× | |
| |
| ### Stage 2: Persona Adapters |
| | Adapter | Language | Rank | Size (MB) | Vocab coverage | |
| |---------|----------|------|-----------|----------------| |
| | Arjun | Hindi+Punjabi | 8 | [X] | [N] items | |
| | Ananya | Tamil+English | 8 | [X] | [N] items | |
| | Priya | Bengali+Hindi | 8 | [X] | [N] items | |
| | Rohan | Hindi+Marathi+English | 8 | [X] | [N] items | |
| | Zara | Urdu+Hindi+Telugu | 8 | [X] | [N] items | |
| |
| --- |
| |
| ## Section 2: Language Quality Evaluation |
| |
| ### BLEU-4 by Language (Stage 1 outputs) |
| [INSERT BAR CHART: horizontal bars, one per language, showing BLEU-4 score 0.0–1.0] |
| |
| | Language | BLEU-4 | chrF++ | |
| |----------|--------|--------| |
| | Hindi | [X] | [X] | |
| | Punjabi | [X] | [X] | |
| | Tamil | [X] | [X] | |
| | Bengali | [X] | [X] | |
| | Hinglish | [X] | [X] | |
| | **Average** | **[X]** | **[X]** | |
| |
| **Note:** chrF++ is the primary metric — it is better suited to morphologically rich Indian languages than BLEU-4. BLEU-4 is included for comparison. |
| |
| ### BLEU-4 vs. Zero-Shot GPT-4 Baseline |
| [INSERT GROUPED BAR CHART: Ankahi vs GPT-4 zero-shot on same test set, per language] |
| |
| Context: GPT-4 zero-shot with a "translate this pictogram sequence to natural Hindi" prompt was used as a baseline. Ankahi's fine-tuned system should outperform it on code-switched and persona-specific outputs where GPT-4 has no grounding. |
| |
| --- |
| |
| ## Section 3: Adapter Isolation (No-Leakage Verification) |
| |
| ### IndicSBERT Cross-Adapter Similarity Heatmap |
| [INSERT 5×5 HEATMAP: rows = query adapter, cols = reference adapter] |
| [Color scale: low similarity = white/cream, high similarity = forest green] |
| [Diagonal = 1.0 (perfect self-similarity)] |
| |
| ``` |
| Arjun Ananya Priya Rohan Zara |
| Arjun [ 1.00 X.XX X.XX X.XX X.XX ] |
| Ananya [ X.XX 1.00 X.XX X.XX X.XX ] |
| Priya [ X.XX X.XX 1.00 X.XX X.XX ] |
| Rohan [ X.XX X.XX X.XX 1.00 X.XX ] |
| Zara [ X.XX X.XX X.XX X.XX 1.00 ] |
| ``` |
| |
| **Interpretation:** Off-diagonal values should be < 0.65. Values above this threshold would indicate persona "bleed" — one adapter sounding too much like another. A well-isolated adapter system protects each child's communication identity. |
|
|
| --- |
|
|
| ## Section 4: Safety Layer Verification |
|
|
| | Test Category | Inputs Tested | Refusals | Refusal Rate | |
| |---------------|---------------|----------|--------------| |
| | Self-harm triggers | 50 | [X] | [X]% | |
| | Inappropriate content | 50 | [X] | [X]% | |
| | Legitimate anger | 20 | 0 (correct) | 0% | |
| | Pain expression | 20 | 0 (correct) | 0% | |
| | Frustration | 20 | 0 (correct) | 0% | |
|
|
| **Target:** 100% refusal on harmful content, 0% refusal on legitimate emotional expression. |
|
|
| --- |
|
|
| ## Section 5: On-Device Inference Benchmarks |
|
|
| ### Test Device |
| | Property | Value | |
| |----------|-------| |
| | Device | [Model name] | |
| | Processor | [SoC model] | |
| | RAM | [X] GB | |
| | Android version | [X] | |
| | Backend used | [CPU / GPU / NPU] | |
| | Backend selection rationale | [Why this backend was chosen] | |
|
|
| ### Latency Breakdown |
| [INSERT BAR CHART: Stacked horizontal bars showing time components] |
|
|
| | Component | Mean (ms) | Std Dev (ms) | P95 (ms) | |
| |-----------|-----------|--------------|----------| |
| | Pictogram → tokens | [X] | [X] | [X] | |
| | LLM prefill (TTFT) | [X] | [X] | [X] | |
| | LLM decode (full sentence) | [X] | [X] | [X] | |
| | Adapter swap overhead | [X] | [X] | [X] | |
| | TTS synthesis | [X] | [X] | [X] | |
| | **End-to-end (tap → audio)** | **[X]** | **[X]** | **[X]** | |
|
|
| Target: end-to-end < 500ms. |
|
|
| [INSERT LINE CHART: Latency over 100 consecutive inferences — shows warmup curve and steady state] |
|
|
| ### Latency by Input Length |
| [INSERT LINE CHART: x-axis = number of pictograms selected (1–8), y-axis = inference time ms] |
|
|
| Expected: roughly linear scaling. Deviations should be noted and explained. |
|
|
| --- |
|
|
| ## Section 6: Memory & Battery |
|
|
| ### Memory Profile |
| [INSERT ANNOTATED SCREENSHOT or BAR CHART of memory usage at key states] |
|
|
| | State | RSS Memory (MB) | |
| |-------|----------------| |
| | App launched, no model | [X] | |
| | Base model loaded | [X] | |
| | Base model + Arjun adapter | [X] | |
| | During active inference | [X] (peak) | |
| | After inference, idle | [X] | |
|
|
| ### Battery Consumption |
| | Session duration | Battery consumed | Implied drain/hour | |
| |-----------------|------------------|--------------------| |
| | 10 minutes active | [X]% | [X]% / hour | |
| | 30 minutes active | [X]% | [X]% / hour | |
|
|
| Target: < 5% per hour sustained. |
|
|
| ### Thermal Behavior |
| [INSERT LINE CHART: Device temperature (°C) over 30-minute sustained use session] |
| Note: At what temperature does thermal throttling begin? What is the latency impact? |
|
|
| --- |
|
|
| ## Section 7: Unit Test Coverage |
|
|
| | Test Suite | Tests | Passed | Failed | Coverage | |
| |------------|-------|--------|--------|----------| |
| | Output quality | [X] | [X] | [X] | [X]% | |
| | Adapter switching | [X] | [X] | [X] | [X]% | |
| | Audio tower | [X] | [X] | [X] | [X]% | |
| | Safety layer | [X] | [X] | [X] | [X]% | |
| | Flutter widget tests | [X] | [X] | [X] | [X]% | |
| | Flutter integration tests | [X] | [X] | [X] | [X]% | |
| | **Total** | **[X]** | **[X]** | **[X]** | **[X]%** | |
|
|
| Previously confirmed: 36/36 passing in the original test suite. All new tests must achieve ≥ 95% pass rate. |
|
|
| --- |
|
|
| ## Section 8: Comparison vs. Commercial AAC Baseline |
|
|
| [INSERT RADAR CHART: 6 axes, comparing "Commercial AAC (best available)" vs "Ankahi"] |
|
|
| Axes: |
| 1. Hindi / North Indian language support (0–5 scale) |
| 2. South Indian language support (0–5) |
| 3. Code-switching / Hinglish support (0–5) |
| 4. Offline capability (0–5) |
| 5. Dysarthric speech tolerance (0–5) |
| 6. Affordability under ₹20,000 (0–5) |
|
|
| Expected result: Ankahi fills or approaches the full radar on all 6 axes. Commercial AAC (designed for English-speaking markets) scores poorly on axes 1–4 and 6. |
|
|
| --- |
|
|
| ## Section 9: Open Research Artifacts |
|
|
| | Artifact | Location | Description | |
| |----------|----------|-------------| |
| | Training dataset | HuggingFace: ankahi/Ankahi-AAC-SFT | 16,500 code-switched AAC pairs | |
| | Base model weights | HuggingFace: ankahi/ankahi-gemma4-e4b-stage1 | Stage 1 fine-tuned Gemma 4 E4B | |
| | Persona adapters | HuggingFace: ankahi/ankahi-personas | 5 Rank-8 LoRA adapters | |
| | Eval benchmark | GitHub: ankahi/ankahi/benchmarks/ | chrF++, BLEU-4, IndicSBERT evaluation code | |
| | Source code | GitHub: [repo link] | Apache 2.0 | |
|
|
| --- |
|
|
| ## Appendix A: Chart Generation Code |
|
|
| All charts in this document were generated with the code in `benchmarks/generate_charts.py`. To reproduce: |
|
|
| ```bash |
| pip install matplotlib seaborn pandas numpy |
| python benchmarks/generate_charts.py --results benchmarks/results.json --output charts/ |
| ``` |
|
|
| Input format for results.json: |
| ```json |
| { |
| "bleu4": { "hindi": X.XX, "punjabi": X.XX, "tamil": X.XX, "bengali": X.XX, "hinglish": X.XX }, |
| "chrf": { "hindi": X.XX, ... }, |
| "adapter_similarity": [[1.0, X.XX, ...], [X.XX, 1.0, ...], ...], |
| "latency_ms": { "prefill_mean": X, "prefill_std": X, "decode_mean": X, ... }, |
| "memory_mb": { "idle": X, "model_loaded": X, "peak_inference": X }, |
| "battery_pct_per_hour": X.X, |
| "safety": { "harmful_refusal_rate": 1.0, "legitimate_expression_rate": 1.0 } |
| } |
| ``` |
| ``` |
| |
| --- |
| |
| ### 3.2 Chart Generation Scripts |
| |
| **[SEARCH REQUIRED]** For the benchmark visualization: |
| - Search: `seaborn heatmap "IndicSBERT" similarity matrix annotation python` |
| - Search: `matplotlib radar chart python "fill" multiple series comparison` |
| |
| Implement the following in `benchmarks/generate_charts.py`: |
| |
| **Chart 1: BLEU-4 by Language (horizontal bar chart)** |
| ```python |
| # Use matplotlib, horizontal bars |
| # Colors: forest green (#2D6A4F) for Ankahi, gray for baseline |
| # Sort bars by score (highest at top) |
| # Add data labels on bars |
| # Save as: charts/bleu4_by_language.png (300 DPI) |
| ``` |
| |
| **Chart 2: IndicSBERT Heatmap (5×5)** |
| ```python |
| # Use seaborn.heatmap |
| # Color: white → forest green (#2D6A4F), range 0.0–1.0 |
| # Annotate cells with 2-decimal values |
| # Title: "Cross-Adapter Similarity (IndicSBERT)" |
| # A clean diagonal of 1.00 and low off-diagonal proves no leakage |
| # Save as: charts/adapter_heatmap.png (300 DPI) |
| ``` |
| |
| **Chart 3: Latency Breakdown (stacked horizontal bar)** |
| ```python |
| # One bar per latency component |
| # Stacked to show total |
| # Colors: one per component (from Ankahi color palette) |
| # Target line at 500ms (red dashed vertical line) |
| # Save as: charts/latency_breakdown.png (300 DPI) |
| ``` |
| |
| **Chart 4: Training Loss Curve** |
| ```python |
| # Line chart, loss vs. training step |
| # Smooth curve (rolling average window=50) |
| # Forest green line on white background |
| # Annotate the final loss value |
| # Save as: charts/training_loss_curve.png (300 DPI) |
| ``` |
| |
| **Chart 5: Radar (Capability Comparison)** |
| ```python |
| # Use matplotlib polar plot |
| # Two filled areas: Ankahi (forest green, 40% alpha) vs Commercial AAC (gray, 30% alpha) |
| # 6 axes as specified above |
| # Legend outside the radar |
| # Save as: charts/capability_radar.png (300 DPI) |
| ``` |
| |
| **Chart 6: Latency over time (line chart for 100 inferences)** |
| ```python |
| # x: inference number (1–100) |
| # y: latency in ms |
| # Show individual points (small dots) + rolling average line |
| # Annotate: "warmup period" (first 5 inferences), "steady state" (rest) |
| # Save as: charts/latency_over_time.png (300 DPI) |
| ``` |
| |
| **Chart 7: Cost Comparison (horizontal bar chart)** |
| ```python |
| # y-axis: device/solution names |
| # x-axis: cost in INR (₹) |
| # Bars (low to high): Ankahi, Low-cost imported AAC, Mid-range AAC, Tobii Dynavox |
| # Add annotation: "ADIP scheme benefit: ₹6,000" as a vertical reference line |
| # Use log scale for x-axis (range is too wide for linear) |
| # Save as: charts/cost_comparison.png (300 DPI) |
| ``` |
| |
| --- |
| |
| ## PART 4: KAGGLE SUBMISSION CHECKLIST |
| |
| Use this checklist before the May 18 deadline: |
| |
| ``` |
| PRE-SUBMISSION |
| □ .litertlm binary produced and validated on at least one Android device |
| □ At least one persona adapter converted and loading successfully |
| □ AI4Bharat TTS integrated (or fallback TTS confirmed working) |
| □ Flutter app builds successfully in release mode |
| □ APK sideloads and runs on a physical device without crash |
| □ All unit tests passing (target: 36+ tests, ≥ 95% pass rate) |
| □ Benchmarks run and results.json populated with real numbers |
| □ All 7 charts generated from real data (no placeholder numbers) |
| □ RESULTS.md complete with all tables and chart embeds |
|
|
| GITHUB REPO |
| □ Repository is PUBLIC |
| □ README.md leads with: problem statement → solution → demo link → how to run |
| □ "Built with Unsloth" badge prominent in README (Unsloth special mention) |
| □ "Gemma 4 E4B" model badge prominent |
| □ Apache 2.0 LICENSE file present |
| □ requirements.txt or pyproject.toml present |
| □ Installation steps tested on a clean environment |
| □ Demo video linked in README |
|
|
| KAGGLE NOTEBOOK |
| □ Notebook runs without errors (test with "Run All") |
| □ All output cells have saved outputs (don't require actual H100 to view) |
| □ All charts render in notebook output |
| □ Section headers clearly label each training stage |
| □ Unsloth integration is highlighted and explained |
| □ A "Why Gemma 4?" section explicitly mentions E4B multimodal features used |
| □ RESULTS.md is linked from the notebook |
|
|
| VIDEO |
| □ Video is ≤ 5 minutes |
| □ Opens with problem statement (the 2.5 million CP children figure) |
| □ Shows a real Android tablet (not a laptop screen) |
| □ Shows a child (or hand) tapping pictogram tiles |
| □ Shows the Hindi sentence appearing |
| □ Shows audio playing (parent's voice) |
| □ Closes with the GitHub link and "free, offline, open-source" statement |
|
|
| FINAL SUBMISSION |
| □ Kaggle submission form completed |
| □ Track selected: Digital Equity (primary) |
| □ "Uses Unsloth" checkbox marked (if present in submission form) |
| □ GitHub link submitted |
| □ Video link (YouTube unlisted or Google Drive) submitted |
| □ Team members added to submission (if applicable) |
| ``` |
| |
| --- |
| |
| ## PART 5: RESOURCES MASTER LIST |
| |
| **[SEARCH REQUIRED before starting any section]** |
| |
| All resources below should be re-verified for current status — URLs and APIs may have changed. |
| |
| ### MediaPipe / LiteRT |
| - Main docs: `https://ai.google.dev/edge/mediapipe` |
| - LLM inference guide: `https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference` |
| - Android LLM guide: `https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/android` |
| - Samples (search): `site:github.com/google-ai-edge mediapipe-samples llm_inference` |
| - PyPI: `mediapipe`, `mediapipe-model-maker` — always check latest version before installing |
| |
| ### Unsloth |
| - Main: `https://github.com/unslothai/unsloth` |
| - Docs: `https://docs.unsloth.ai` |
| - Gemma 4 specific: Search `"unsloth" "Gemma 4" fine-tuning notebook 2026` |
| |
| ### AI4Bharat TTS |
| - GitHub: Search `AI4Bharat Indic-TTS github` |
| - HuggingFace: Search `AI4Bharat TTS huggingface model` |
| - Demo: Search `AI4Bharat TTS voice cloning demo` |
| |
| ### IndicSBERT (for eval) |
| - Search: `"IndicSBERT" OR "Indic SBERT" huggingface sentence similarity` |
| - Likely: `sentence-transformers` library + an IndicNLP model |
| |
| ### Flutter + MediaPipe Integration |
| - Search: `flutter mediapipe genai "method channel" llm inference kotlin dart` |
| - Search: `flutter ffi mediapipe native library android` |
| |
| ### TORGO / UA-Speech (audio training data) |
| - TORGO: Search `"TORGO database" download dysarthric speech corpus` |
| - UA-Speech: Search `"UA-Speech" database download dysarthric` |
| |
| ### chrF++ implementation |
| - Search: `"sacrebleu" python "chrf" "chrf++" implementation` |
| - Library: `pip install sacrebleu` → `sacrebleu.corpus_chrf()` |
| |
| --- |
| |
| *End of Backend & System Specification.* |
| *All three documents together constitute the complete Ankahi technical brief for the Kaggle Gemma 4 Good Hackathon submission.* |
