| # Hand Tracking Drawing |
|
|
| Draw in the air with your hand, then view the drawing as a rotatable 3D object. |
|
|
|  |
|
|
|
|
| **[DOCKER IMAGE HERE](https://hub.docker.com/r/flugmaschine/hand-tracking-drawing)** |
|
|
| ## Model |
|
|
| Uses the **MediaPipe Hand Landmarker** (`models/hand_landmarker.task`), which returns |
| 21 keypoints per hand. Gestures are classified from ratios and joint angles between |
| those keypoints, so recognition is independent of hand size, distance, or handedness. |
|
|
| ## Logic |
|
|
| - **One index finger** - draw |
| - **Index + middle** - move cursor without drawing |
| - **Fist (hold)** - grab the drawing; move the fist to rotate it in 3D |
| - **Two fists** - scale (canvas in 2D, object in 3D) |
| - **Open palm** - release, back to drawing |
| - **OK (hold)** - clear the canvas |
| - **Pinky (hold)** - undo the last stroke |
| - **Middle finger** - shows `:((` for a second |
| - **Both palms waving** - say goodbye, closes the app |
|
|
| Keys: `S` save, `C` clear, `U` undo, `1..6` color, `[` / `]` thickness, |
| `SPACE` 2D/3D, `R` reset view, `K` skeleton, `D` debug, `H` help, `Q` quit. |
|
|
| ## 3D Drawing |
|
|
| The **3D Drawing** mode uses the YOLO depth model (`models/yolo26s-depth.pt`), which |
| returns a per pixel depth map in meters. Every drawn point takes its Z from the metric |
| depth under the fingertip, so moving the hand closer or further really draws into the |
| scene: the stroke is a 3D curve in space, not a flat picture placed on a board. The |
| first drawn point sets the zero plane, everything after that is relative to it. |
| Depth runs in a background thread, so tracking and drawing stay at camera speed. |
|
|
| Raw monocular depth flickers, so the pen does not follow it directly: the global scale |
| drift of the map is cancelled against the static background, the fingertip depth goes |
| through a one euro filter with a noise deadband and a speed limit, and the newest points |
| of a stroke are low passed along the curve. A circle drawn without moving the hand comes |
| out flat, while a deliberate push of 25 cm is kept in full. |
|
|
| By default strokes are colored by a depth gradient - warm near the camera, cool further |
| away. `1..6` switch to a solid color, `G` goes back to the gradient. |
|
|
| Gestures and keys are the same as in 2D, with two additions: |
|
|
| - **Two ring fingers (hold)** - toggle depth colors for the whole frame, drawing keeps working |
| - **Open palm (hold)** - reset the view |
|
|
| In 3D the fist rotates the drawing itself and two fists scale it, `SPACE` toggles |
| depth colors, `R` resets the view. |
|
|
| ## Run |
|
|
| ```bash |
| pip install -r requirements.txt |
| python src/main.py |
| ``` |
|
|
| Pick the mode in the menu, or skip it: |
|
|
| ```bash |
| python src/main.py --no-menu --mode 3d |
| python src/main.py --no-menu --mode 3d --depth-size 448 --depth-every 1 |
| ``` |
|
|
| by Salimli Ayzek (Салимли Айзек): https://mathematiclove.github.io |
|
|
