| --- |
| license: apache-2.0 |
| tags: |
| - computer-vision |
| - gesture-recognition |
| - vision-transformer |
| - lstm |
| language: en |
| library_name: pytorch |
| datasets: |
| - jester |
| model_index: |
| - hf://hslay13/vit_lstm_jester |
| --- |
| |
| # Hand Gesture Recognition – Vision Transformer + LSTM |
|
|
| ## Model Description |
|
|
| This repository contains a Bidirectional LSTM model for dynamic hand gesture recognition, trained on features extracted from the Jester dataset. The model is designed to classify temporal sequences of frame-level embeddings. |
|
|
| The overall architecture is a two-stage process: |
|
|
| 1. **Feature Extraction**: A pre-trained Vision Transformer (ViT-B/16), fine-tuned on the HaGRID dataset, is used as a frozen feature extractor. It processes each video frame (224x224) and generates a 768-dimensional embedding. No training was performed on the ViT in this project; it is used as-is. |
| 2. **Temporal Classification**: The sequence of frame embeddings is then fed into the Bidirectional LSTM model, which learns to recognize the temporal patterns of different hand gestures. |
|
|
| Model Version: clean-baseline-v2.0-hf |
| Architecture: Bidirectional LSTM with ViT-B/16 feature extractor |
| Parameters (LSTM): ~514k |
| Framework: TensorFlow + Keras (for LSTM), PyTorch + timm (for ViT feature extraction) |
|
|
| ## Performance Metrics (LSTM Classifier) |
|
|
| | Metric | Value | |
| | ------------------------- | --------------- | |
| | Validation Accuracy | 0.8100 (81.00%) | |
| | Validation Loss | 0.7010 | |
| | Weighted Average F1 Score | 0.81 | |
|
|
| ## Training Details |
|
|
| ### Dataset |
|
|
| Name: 20bn-Jester Dataset |
| Source: local |
| Train samples: 50,420 |
| Validation samples: 7,047 |
|
|
| ### Training Configuration (LSTM) |
|
|
| Batch Size: 8 |
| Optimizer: Adam |
| Loss: Categorical Cross-Entropy |
| Learning Rate: 0.001 (reduced by ReduceLROnPlateau) |
| Epochs Trained: Up to 30 (best weights restored from epoch 29 based on validation accuracy) |
| Regularization: Dropout (0.4, 0.3), L2 (1e-4) on Dense layer |
| Callbacks: ModelCheckpoint, ReduceLROnPlateau, EarlyStopping (patience=7) |
|
|
| ### Data Processing |
|
|
| Preprocessing: Single deterministic pipeline (no augmentation) |
| Resize: 224×224 |
| Normalization: ImageNet statistics (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) |
| Temporal Sampling: Fixed sequence length of 37 frames |
| Padding: Zero-vector with masking |
| Augmentation: none (clean baseline principle) |
| Test-Time Augmentation: False (disabled) |
|
|
| ## Architecture Details |
|
|
| Type: standard ViT-B/16 (no modifications, frozen during LSTM training) |
| ViT Output: 768-D feature embeddings per frame |
| Classification Head (ViT): Removed for feature extraction |
| LSTM Model: - Input: Sequences of ViT embeddings (37 frames, 768 dimensions) - Layers: Masking, Bidirectional LSTM (256 units, return_sequences=True), Dropout (0.4), Bidirectional LSTM (128 units), Dropout (0.4), Dense (128 units, ReLU, L2 regularization), Dropout (0.3), Dense (27 classes, Softmax) |
| Total Trainable Parameters (LSTM): ~514k |
| Dropout: 0.4, 0.3 |
| |
| ## Gesture Classes (27 total) |
| |
| 'Doing other things', 'Drumming Fingers', 'No gesture', 'Pulling Hand In', 'Pulling Two Fingers In', 'Pushing Hand Away', 'Pushing Two Fingers Away', 'Rolling Hand Backward', 'Rolling Hand Forward', 'Shaking Hand', 'Sliding Two Fingers Down', 'Sliding Two Fingers Left', 'Sliding Two Fingers Right', 'Sliding Two Fingers Up', 'Stop Sign', 'Swiping Down', 'Swiping Left', 'Swiping Right', 'Swiping Up', 'Thumb Down', 'Thumb Up', 'Turning Hand Clockwise', 'Turning Hand Counterclockwise', 'Zooming In With Full Hand', 'Zooming In With Two Fingers', 'Zooming Out With Full Hand', 'Zooming Out With Two Fingers' |
| |
| ## Usage |
| |
| Refer to `how_to_use.py` for a complete end-to-end example. |
| |
| ## Limitations |
| |
| • Requires consistent frame rate and sampling |
| • Sensitive to heavy occlusion and motion blur |
| • Assumes a single dominant gesture per clip |
| • Performance depends on ViT embedding quality |
| |
| ## License |
| |
| MIT License |
| |
| ## Dataset Acknowledgment |
| |
| This model was trained on the **Jester Dataset**, a large-scale video dataset for hand gesture recognition. We would like to thank **Twenty Billion Neurons (TwentyBN)** for creating and sharing this dataset. |
| |
| - **Dataset Homepage:** [Jester Dataset](https://developer.qualcomm.com/software/jester-dataset) |
| - **License:** The Jester dataset is available under a Creative Commons license. Please refer to the dataset's official website for more information on its license. |
| |
| ## How to Use |
| |
| This model consists of two parts: a Vision Transformer (ViT) feature extractor and an LSTM-based temporal classifier. |
| |
| ### 1. Installation |
| |
| First, make sure you have the required libraries installed: |
| |
| ```bash |
| pip install tensorflow torch torchvision timm transformers |
| ``` |
| |
| ### 2. Loading the Models |
| |
| The ViT backbone can be loaded from the Hugging Face Hub [here](https://huggingface.co/neilrigaud/hagrid-vit-gesture), while the trained LSTM model can be loaded from the `best_lstm_model.keras` file in this repository. |
| |
