README.md

metadata

title: SignLingo ASL Coach
emoji: 🤟
colorFrom: blue
colorTo: indigo
sdk: gradio
app_file: app.py
pinned: true
license: mit
hardware: zero-gpu
tags:
  - asl
  - sign-language
  - mediapipe
  - minicpm
  - education
  - backyard-ai
  - build-small
  - off-brand
  - tiny-titan
  - best-demo
  - codex
  - track:backyard
  - sponsor:openbmb
  - sponsor:openai
  - achievement:offgrid
  - achievement:offbrand

SignLingo ASL Coach

Learn American Sign Language with real-time AI coaching. Built for the Hugging Face × Gradio Build Small hackathon.

What it does

• Watch the reference ASL sign on the left
• Sign in front of your webcam on the right
• MediaPipe detects your hands automatically — recording starts when you raise your hands, stops when you drop them
• DTW (Dynamic Time Warping) scores your attempt against the reference in real time
• On a failed attempt, frames from your recording are sent to MiniCPM-V which returns one specific coaching tip

How it works

• Hand + pose landmarks extracted in-browser via MediaPipe WASM (no server round-trip for detection)
• Normalized landmark sequences compared with DTW for a 0–100 similarity score
• Failed attempts: 1-2fps frames → MiniCPM-V 4.6 → natural language coaching tip
• No cloud APIs — scoring runs entirely in the browser; coaching runs on HF Zero GPU

Tech Stack

Component	Choice
Frontend	React + Vite served via gr.Server
Hand detection	MediaPipe Tasks Vision (WASM, in-browser)
Scoring	DTW on normalized landmark sequences
Coaching VLM	MiniCPM-V 4.6 on HF Zero GPU
Reference Signs	Lifeprint ASL Dictionary

Vocabulary (23 signs)

Hello, Yes, No, Thank You, Help, Pain, Head, Back, Sick, Fever, Medicine, Water, Blood, Doctor, Nurse, Emergency, Bank, Money, Pay, Credit Card, Account, Deposit, Withdraw

Demo Video

https://youtu.be/qynGluC2ZLc

Social Post

https://x.com/JaspalS44414885/status/2066665273739092445

Built for

Hugging Face × Gradio "Build Small" Hackathon — Track: Backyard AI (Practical)

Made with care for the deaf and hard-of-hearing community.