| --- |
| library_name: lyrebird |
| tags: |
| - gesture-recognition |
| - pose |
| - mediapipe |
| - skeleton |
| pipeline_tag: video-classification |
| --- |
| |
| # Lyrebird motion encoder |
|
|
| This repository holds the frozen encoder that |
| [Lyrebird](https://github.com/BruhLemma-Yadecha/lyrebird) uses to identify gestures in |
| webcam video. |
|
|
| The encoder reads MediaPipe landmark data and gives one token for each body area |
| in each frame. Lyrebird trains a single linear layer on top of it from about ten |
| examples of each gesture. The encoder itself does not change. |
|
|
| ## Files |
|
|
| | File | Contents | |
| | --- | --- | |
| | `encoder.pt` | The encoder parameters and its configuration. | |
| | `mediapipe/holistic_landmarker.task` | The MediaPipe model that finds the landmarks. | |
|
|
| The MediaPipe file is a copy of Google's holistic landmarker. Lyrebird keeps a |
| copy here so that it needs only one repository. Google gives its terms and its |
| model card at https://ai.google.dev/edge/mediapipe/solutions/vision/holistic_landmarker |
| |
| ## Use |
| |
| Lyrebird downloads this file for you. To get it directly: |
| |
| ```python |
| from huggingface_hub import hf_hub_download |
|
|
| from lyrebird.classifier import load_encoder |
| |
| encoder = load_encoder(hf_hub_download("byadecha/lyrebird", "encoder.pt")) |
| ``` |
| |
| ## The model |
| |
| | Property | Value | |
| | --- | --- | |
| | Parameters | 1.70 million | |
| | Input | 33 body joints, and 21 joints for each hand | |
| | Output | 4 tokens for each frame, of 128 values each | |
| | Frame rate | 30 frames each second | |
| |
| Four encoders operate in parallel across the body, each hand, and the movement |
| of the body through the frame. Each one reduces its joints to one token for each |
| frame. It uses graph convolution across the skeleton, and then attention across |
| time. A fusion stage mixes the four tokens in one frame, and then across time. |
| The fusion stage keeps all four tokens. It does not make an average of them. |
| |
| The encoder is causal. A later frame does not change an earlier frame. Thus the |
| same parameters give a score to a recorded clip and to live video in the same |
| way. |
| |
| ## Training data |
| |
| The encoder is trained on the BABEL annotations of the AMASS motion capture |
| data. The training task is to identify which of 43 motion categories occurs in |
| each frame. |
| |
| BABEL and AMASS have their own terms of use. This repository makes no statement |
| about what those terms permit for these parameters. Read the terms of both |
| datasets before you use this file: |
| |
| - AMASS: https://amass.is.tue.mpg.de/ |
| - BABEL: https://babel.is.tue.mpg.de/ |
| |
| ## Limits |
| |
| The classifier that Lyrebird trains on this encoder always gives the name of the |
| most probable gesture that it knows. It cannot answer "none of these". Movement |
| that is not like any recorded gesture still gets a name. Record an idle gesture, |
| and record more than one form of it, to reduce this effect. |
| |
| The gesture vector is a mean across a window of frames. Thus a gesture that |
| fills only a small part of a long window becomes too weak to find. Give a score |
| to windows of about the same length as the recorded clips. |
| |
