Spaces:
Sleeping
Sleeping
| title: LSF Interpreter | |
| emoji: 🤟 | |
| colorFrom: green | |
| colorTo: blue | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| DISCLAIMER: Used claude for django, UI, Gunicorn and some comments (and maybe git) | |
| # LSF_transcripter — French Sign Language interpreter | |
| Real-time **Langue des Signes Française (LSF)** interpreter. It reads the signer | |
| from a camera, tracks the body parts that carry meaning in sign language — | |
| **fingers / hand shape, eyes & gaze, mouth, chest and arms** — draws the | |
| skeleton over the live feed, and transcribes recognised signs underneath. | |
| A **Django** web UI shows the live camera feed with the skeleton at the top and | |
| the running transcription at the bottom (HTML / CSS / JS in separate files). | |
| **Architecture:** the **browser** captures the camera and posts frames; the | |
| **server** runs MediaPipe + the recognizer and returns landmarks + glosses that | |
| the browser draws. The same app therefore runs locally *and* on Hugging Face | |
| (where the server has no webcam), and dual-pixel detection reads the real client | |
| device. See [Deploy](#deploy--docker--hugging-face). | |
| --- | |
| ## How it works | |
| ``` | |
| camera frame | |
| │ | |
| ├─▶ MediaPipe Holistic ──▶ pose (33) · face mesh (468) · 2×hands (21) | |
| │ │ fingers ▸ eyes ▸ mouth ▸ chest ▸ arms | |
| │ ▼ | |
| │ depth refinement ◀── dual-pixel / depth stream (or MiDaS, or none) | |
| │ ▼ | |
| │ normalised body-relative features (features.py) | |
| │ ▼ | |
| │ sign recogniser ──▶ DTW templates · trained model · heuristics | |
| │ ▼ | |
| └─▶ skeleton overlay + transcribed gloss | |
| ``` | |
| | Body part the brief asked for | Where it comes from | | |
| |-------------------------------|----------------------------------| | |
| | Fingers / hand shape | `left_hand` / `right_hand` (21 pts each) | | |
| | Eyes / gaze | face-mesh eye + iris landmarks | | |
| | Mouth (mouthing, non-manuals) | face-mesh lip landmarks | | |
| | Chest | pose shoulders + hips | | |
| | Arms | pose shoulders / elbows / wrists | | |
| | Hand | hand wrist + pose wrist | | |
| ### Dual-pixel sensor detection (e.g. Pixel devices) | |
| "Dual-pixel" depth is produced by the *camera stack* on certain devices (e.g. | |
| Pixel phones); it lives on the **client device**, not the server. Detection | |
| therefore runs in the browser ([app.js](web/interpreter/static/interpreter/app.js), | |
| `detectDualPixel`): it probes the `MediaStreamTrack` capabilities | |
| (`getCapabilities`, `getSupportedConstraints`), enumerates devices for a depth | |
| camera, and matches the camera label (`pixel`, `depth`, `tof`, `truedepth`). The | |
| result is POSTed to `/depth`; the status pill then shows `dual_pixel` vs | |
| `mediapipe`, with the reason in its tooltip. | |
| Honest limit: browsers don't generally expose the *dual-pixel depth map* itself, | |
| so when detected we flag the source and keep MediaPipe's relative `z`. If a | |
| platform does hand you a depth map, feed it to | |
| `DepthEstimator.depth_for_frame(rgb, external_depth=...)` and every landmark's | |
| `z` is resampled from it. A server-side MiDaS monocular fallback is also | |
| available (`enable_monocular_depth`, needs `torch`). | |
| ### Recognition — be realistic | |
| There is no off-the-shelf production LSF translation model. This project gives | |
| you a working, extensible pipeline rather than a magic black box: | |
| 0. **Vocabulary catalogue.** It ships with ~80 common LSF words | |
| (`signs/lexicon.json`) grouped by theme — greetings, family, verbs, | |
| questions, emotions, time, food, places, colours. These are the *targets* | |
| shown in the web UI's vocabulary browser; a word becomes *recognisable* once | |
| you train it (step 1). You can add your own words from the UI too. | |
| 1. **Learn-by-example (default).** Click **Entraîner** next to a word, perform | |
| the sign, click **Sauvegarder**. Samples are stored as normalised landmark | |
| sequences and matched with **Dynamic Time Warping**. Runs on a laptop CPU, no | |
| training needed, good for a small/medium vocabulary. | |
| 2. **Trained model (optional).** Drop a Keras `models/model.h5` (+ | |
| `models/model.labels.json`) and it's used automatically. Train your own | |
| sequence model on a corpus for real coverage. | |
| 3. **Heuristics.** A couple of geometry-only signs so the demo isn't silent | |
| before you've recorded anything. | |
| --- | |
| ## Setup | |
| ```bash | |
| ./setup.sh # creates .venv and installs deps | |
| source .venv/bin/activate | |
| ``` | |
| Requires Python 3.9–3.11 (MediaPipe constraint). On macOS, grant camera access | |
| to your terminal in *System Settings ▸ Privacy & Security ▸ Camera*. | |
| ## Run | |
| **Quick start — one command (creates the venv on first run):** | |
| ```bash | |
| ./run.sh # → http://127.0.0.1:8000 (ships pre-trained, 72 LSF signs) | |
| ``` | |
| **Web UI (Django) — everything is in the browser, no terminal needed:** | |
| ```bash | |
| python web/manage.py runserver | |
| # open http://127.0.0.1:8000 (allow the camera when prompted) | |
| ``` | |
| The browser captures the camera; `getUserMedia` needs a secure context, which | |
| `localhost` satisfies (on a LAN IP it won't — use localhost or HTTPS). | |
| * **Top-left:** live feed with the skeleton (drawn in the browser). | |
| * **Bottom-left:** transcription. | |
| * **Right:** vocabulary browser — search/filter the ~80-word catalogue, see which | |
| signs are trained, **Entraîner** a word (record → *Sauvegarder*), delete | |
| trained samples, **Ajouter** a custom word, or click a word for its tip + Elix | |
| sign video. | |
| * Status pills show the depth source (incl. `dual_pixel` when detected), FPS, and | |
| how many signs are trained. | |
| **Standalone window (local OpenCV, uses the server's own webcam):** | |
| ```bash | |
| python main.py # keys: r=record, s=save, c=clear, q=quit | |
| ``` | |
| ## Deploy — Docker / Hugging Face | |
| **Docker (works locally too — the camera is the browser's, no device passthrough):** | |
| ```bash | |
| docker build -t lsf . | |
| docker run -p 7860:7860 lsf | |
| # open http://localhost:7860 | |
| ``` | |
| **Hugging Face Spaces:** push the repo to a **Docker** Space (the front-matter at | |
| the top of this README sets `sdk: docker` and `app_port: 7860`). HF builds the | |
| [Dockerfile](Dockerfile) and serves it over HTTPS, so the browser camera and | |
| dual-pixel detection work. Tuning is via env vars: `LSF_MODEL_COMPLEXITY` | |
| (0–2), `LSF_REFINE_FACE` (0/1), `LSF_DEBUG`, `LSF_SECRET_KEY`. | |
| > Note: a hosted Space has an **ephemeral, often read-only** filesystem — trained | |
| > templates/added words persist in memory for the session but may not survive a | |
| > restart. For durable training, run locally or import clips (below) into a | |
| > mounted volume. | |
| ### Pre-training words from video clips (batch) | |
| Rather than recording each sign live, you can batch-import reference clips so the | |
| words are recognised immediately. Name clips by their gloss: | |
| ``` | |
| clips/BONJOUR.mp4 # one sample for BONJOUR | |
| clips/AU REVOIR/01.mp4 # several samples for AU REVOIR | |
| clips/AU REVOIR/02.mp4 | |
| ``` | |
| ```bash | |
| python tools/import_videos.py clips/ | |
| # or a single file: | |
| python tools/import_videos.py merci.mov --gloss MERCI | |
| ``` | |
| This runs the clips through the same pipeline as the live recorder and writes | |
| templates into `signs/lsf_signs.json`. Record 2–3 clips per sign (slightly | |
| different speed/framing) for better accuracy. Note: there is no public drop-in | |
| *pre-trained LSF model* — reference clips (filmed by you or any LSF video you're | |
| allowed to use) are how the vocabulary gets "pre-trained". | |
| **Auto-train from the Elix dictionary.** [tools/elix_train.py](tools/elix_train.py) | |
| fetches each catalogue word's reference sign video from | |
| [dico.elix-lsf.fr](https://dico.elix-lsf.fr/) and trains a template from it — | |
| **72 of the 81 words ship pre-trained** this way: | |
| ```bash | |
| python tools/elix_train.py # all words (or --words BONJOUR MERCI …) | |
| ``` | |
| It rate-limits and identifies itself; robots.txt permits crawling. The Elix | |
| videos are © Signes de sens — this derives abstract landmark templates for | |
| personal/on-device use and does **not** redistribute the videos. The 9 unmatched | |
| words are multi-word phrases (AU REVOIR, ÇA VA, …) — record those by hand. | |
| --- | |
| ## Layout | |
| ``` | |
| main.py standalone CLI runner (OpenCV window, local webcam) | |
| tools/import_videos.py batch clips → templates ("pre-train" words) | |
| Dockerfile / .dockerignore container build (HF Spaces / local) | |
| requirements.txt deps (mediapipe, opencv, django, gunicorn, whitenoise) | |
| setup.sh venv bootstrap | |
| signs/lexicon.json ~80-word LSF vocabulary catalogue (themes + tips) | |
| signs/lsf_signs.json learned-sign templates (shared by CLI + web) | |
| models/ optional model.h5 + model.labels.json | |
| lsf/ | |
| landmarks.py MediaPipe Holistic wrapper + compact points payload | |
| skeleton.py server-side skeleton draw (CLI only) | |
| depth.py depth refinement + MiDaS fallback | |
| features.py landmarks ▸ normalised body-relative feature vector | |
| recognizer.py motion segmentation + DTW / model / heuristics | |
| lexicon.py loads/extends the vocabulary catalogue (+ Elix links) | |
| camera.py InterpreterPipeline + local-webcam singleton (CLI) | |
| session.py FrameSession: browser-pushed frames → landmarks+glosses | |
| web/ | |
| manage.py | |
| web/ Django project (settings, urls, wsgi, asgi) | |
| interpreter/ Django app | |
| views.py page · /process · /depth · vocab + training APIs | |
| urls.py | |
| templates/interpreter/index.html | |
| static/interpreter/style.css | |
| static/interpreter/app.js getUserMedia · dual-pixel · /process · skeleton draw | |
| ``` | |
| ## Limitations & next steps | |
| * Out of the box it recognises only what you record (plus 2 rough heuristics). | |
| For broad LSF coverage, collect a dataset and train the model hook. | |
| * LSF grammar (spatial reference, classifiers, facial grammar) is **not** parsed | |
| — output is a stream of glosses, not fluent French. A gloss→French language | |
| step would sit after the recogniser. | |
| * Dual-pixel depth needs hardware that exposes it (see above). | |