Skunk0/openglasses-fingerspelling-ctc

Fingerspelling CTC (Core ML)

On-device American Sign Language fingerspelling recognizer used by the OpenGlasses accessibility tier. A CTC sequence model over MediaPipe Holistic landmarks, converted to a fixed-window Core ML program:

• Input: features (1, 768, 1629) float32 — up to 768 frames × 543 landmarks × xyz (canonical order: face 0–467, left hand 468–488, pose 489–521, right hand 522–542), per-window standardised, NaN→0, zero-padded — plus mask (1, 768) float32 validity.
• Output: (1, 384, 62) CTC logits (stride 2; read the first ⌈T/2⌉ rows; class 0 is the blank, classes 1–59 map through vocab.txt, the last two are auxiliary tokens).
• Greedy decode scores 20.8% mean CER (median 10.8%) on a 300-sequence held-out set of the training corpus's rerun benchmark.

Files

File	Purpose
Fingerspelling2P.mlpackage	the Core ML model (fp16, iOS 17+)
vocab.txt	CTC charset sidecar (<blank> + 59 symbols, one per line)
holistic_landmarker.task	MediaPipe holistic landmark extractor consumed alongside the model

License & attribution

• Model weights and conversion: Apache-2.0 (see LICENSE).
• Trained on the Google ASL Fingerspelling corpus (Google / Deaf Professional Arts Network), CC-BY 4.0.
• holistic_landmarker.task is a MediaPipe model asset (Google), Apache-2.0.