---
title: SignLingo ASL Coach
emoji: 🤟
colorFrom: blue
colorTo: indigo
sdk: gradio
app_file: app.py
pinned: true
license: mit
hardware: zero-gpu
tags:
  - asl
  - sign-language
  - mediapipe
  - minicpm
  - education
  - backyard-ai
  - build-small
  - off-brand
  - tiny-titan
  - best-demo
  - codex
  - track:backyard
  - sponsor:openbmb
  - sponsor:openai
  - achievement:offgrid
  - achievement:offbrand
---

# SignLingo ASL Coach

Learn American Sign Language with real-time AI coaching. Built for the **Hugging Face × Gradio Build Small** hackathon.

## What it does

- Watch the reference ASL sign on the left
- Sign in front of your webcam on the right
- **MediaPipe** detects your hands automatically — recording starts when you raise your hands, stops when you drop them
- **DTW (Dynamic Time Warping)** scores your attempt against the reference in real time
- On a failed attempt, frames from your recording are sent to **MiniCPM-V** which returns one specific coaching tip

## How it works

- Hand + pose landmarks extracted in-browser via MediaPipe WASM (no server round-trip for detection)
- Normalized landmark sequences compared with DTW for a 0–100 similarity score
- Failed attempts: 1-2fps frames → MiniCPM-V 4.6 → natural language coaching tip
- No cloud APIs — scoring runs entirely in the browser; coaching runs on HF Zero GPU

## Tech Stack

| Component | Choice |
|-----------|--------|
| Frontend | React + Vite served via `gr.Server` |
| Hand detection | MediaPipe Tasks Vision (WASM, in-browser) |
| Scoring | DTW on normalized landmark sequences |
| Coaching VLM | MiniCPM-V 4.6 on HF Zero GPU |
| Reference Signs | Lifeprint ASL Dictionary |

## Vocabulary (23 signs)

Hello, Yes, No, Thank You, Help, Pain, Head, Back, Sick, Fever, Medicine, Water, Blood, Doctor, Nurse, Emergency, Bank, Money, Pay, Credit Card, Account, Deposit, Withdraw

## Demo Video

https://youtu.be/qynGluC2ZLc

## Social Post

https://x.com/JaspalS44414885/status/2066665273739092445

## Built for

**Hugging Face × Gradio "Build Small" Hackathon** — Track: Backyard AI (Practical)

Made with care for the deaf and hard-of-hearing community.