README.md

metadata

title: ASL Hand Gesture Recognizer
emoji: 🤟
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
python_version: '3.12'
app_file: app.py
pinned: false
license: cc-by-nc-4.0

ASL Hand Gesture Recognizer

What it does

Real-time American Sign Language recognition from your webcam. Each frame runs MediaPipe hand tracking, extracts 21 normalized hand landmarks, and classifies the gesture with a lightweight MLP exported to ONNX. Predictions are smoothed over the last 10 frames so the label stays stable while you hold a sign.

Classes

36 total: letters A–Z and digits 0–9.

How to use

1. Allow camera access.
2. Show your hand to the webcam.
3. Hold the gesture steady — the predicted letter/digit appears top-left with a confidence bar. Landmark dots are drawn on your hand. Low-confidence frames show ... until the prediction settles.

Model

MLP (63 → 256 → 128 → 36) trained on the ASL-HG dataset. Input is wrist-centered, scale-normalized hand landmarks (x,y,z × 21). Held-out validation accuracy is 100% on 6,193 landmark samples — the landmark representation makes the 36 classes near-linearly separable, and it avoids the live-webcam distribution shift that hurt the earlier image-CNN baseline.

Results & Analysis

Evaluated on the stratified 20% validation split (6,193 samples).

Metric	Value
Validation accuracy	100.00%
Macro per-class accuracy	100.00%
Classes	36 (A–Z, 0–9)
Landmark samples	30,962 (hand detected)

The confusion matrix is fully diagonal. class_distribution.png shows how many images yielded a detected hand per class (closed-fist signs like O, C, A, T retain fewer samples because MediaPipe detects them less often).

Tech stack

• Gradio — streaming webcam UI
• MediaPipe Hands — 21-point hand landmark detection
• ONNX Runtime — CPU inference
• OpenCV / NumPy — frame annotation and preprocessing

Dataset & Credits

Dataset: ASL-HG — American Sign Language Hand Gesture Image Dataset
Pranto et al. (2026), Data in Brief
Article: https://www.sciencedirect.com/science/article/pii/S2352340926000454
Mendeley Data: https://data.mendeley.com/datasets/j4y5w2c8w9/1
DOI: https://doi.org/10.17632/j4y5w2c8w9.1

Model & App: Doruk Doğular (nocontextdoruk)
Landmark-based MLP trained on ASL-HG processed split. Model repo: https://huggingface.co/nocontextdoruk/asl-landmark-mlp

License

CC BY-NC 4.0 — free for research, education, and personal/open projects with attribution; no commercial or enterprise resale. See LICENSE. If you use it, please cite via CITATION.cff. The ASL-HG dataset is owned by its original authors (cite separately).