README.md

---
license: apache-2.0
tags:
    - computer-vision
    - gesture-recognition
    - vision-transformer
    - lstm
language: en
library_name: pytorch
datasets:
    - jester
model_index:
    - hf://hslay13/vit_lstm_jester
---

# Hand Gesture Recognition – Vision Transformer + LSTM

## Model Description

This repository contains a Bidirectional LSTM model for dynamic hand gesture recognition, trained on features extracted from the Jester dataset. The model is designed to classify temporal sequences of frame-level embeddings.

The overall architecture is a two-stage process:

1.  **Feature Extraction**: A pre-trained Vision Transformer (ViT-B/16), fine-tuned on the HaGRID dataset, is used as a frozen feature extractor. It processes each video frame (224x224) and generates a 768-dimensional embedding. No training was performed on the ViT in this project; it is used as-is.
2.  **Temporal Classification**: The sequence of frame embeddings is then fed into the Bidirectional LSTM model, which learns to recognize the temporal patterns of different hand gestures.

Model Version: clean-baseline-v2.0-hf
Architecture: Bidirectional LSTM with ViT-B/16 feature extractor
Parameters (LSTM): ~514k
Framework: TensorFlow + Keras (for LSTM), PyTorch + timm (for ViT feature extraction)

## Performance Metrics (LSTM Classifier)

| Metric                    | Value           |
| ------------------------- | --------------- |
| Validation Accuracy       | 0.8100 (81.00%) |
| Validation Loss           | 0.7010          |
| Weighted Average F1 Score | 0.81            |

## Training Details

### Dataset

Name: 20bn-Jester Dataset
Source: local
Train samples: 50,420
Validation samples: 7,047

### Training Configuration (LSTM)

Batch Size: 8
Optimizer: Adam
Loss: Categorical Cross-Entropy
Learning Rate: 0.001 (reduced by ReduceLROnPlateau)
Epochs Trained: Up to 30 (best weights restored from epoch 29 based on validation accuracy)
Regularization: Dropout (0.4, 0.3), L2 (1e-4) on Dense layer
Callbacks: ModelCheckpoint, ReduceLROnPlateau, EarlyStopping (patience=7)

### Data Processing

Preprocessing: Single deterministic pipeline (no augmentation)
Resize: 224×224
Normalization: ImageNet statistics (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
Temporal Sampling: Fixed sequence length of 37 frames
Padding: Zero-vector with masking
Augmentation: none (clean baseline principle)
Test-Time Augmentation: False (disabled)

## Architecture Details

Type: standard ViT-B/16 (no modifications, frozen during LSTM training)
ViT Output: 768-D feature embeddings per frame
Classification Head (ViT): Removed for feature extraction
LSTM Model: - Input: Sequences of ViT embeddings (37 frames, 768 dimensions) - Layers: Masking, Bidirectional LSTM (256 units, return_sequences=True), Dropout (0.4), Bidirectional LSTM (128 units), Dropout (0.4), Dense (128 units, ReLU, L2 regularization), Dropout (0.3), Dense (27 classes, Softmax)
Total Trainable Parameters (LSTM): ~514k
Dropout: 0.4, 0.3

## Gesture Classes (27 total)

'Doing other things', 'Drumming Fingers', 'No gesture', 'Pulling Hand In', 'Pulling Two Fingers In', 'Pushing Hand Away', 'Pushing Two Fingers Away', 'Rolling Hand Backward', 'Rolling Hand Forward', 'Shaking Hand', 'Sliding Two Fingers Down', 'Sliding Two Fingers Left', 'Sliding Two Fingers Right', 'Sliding Two Fingers Up', 'Stop Sign', 'Swiping Down', 'Swiping Left', 'Swiping Right', 'Swiping Up', 'Thumb Down', 'Thumb Up', 'Turning Hand Clockwise', 'Turning Hand Counterclockwise', 'Zooming In With Full Hand', 'Zooming In With Two Fingers', 'Zooming Out With Full Hand', 'Zooming Out With Two Fingers'

## Usage

Refer to `how_to_use.py` for a complete end-to-end example.

## Limitations

• Requires consistent frame rate and sampling
• Sensitive to heavy occlusion and motion blur
• Assumes a single dominant gesture per clip
• Performance depends on ViT embedding quality

## License

MIT License

## Dataset Acknowledgment

This model was trained on the **Jester Dataset**, a large-scale video dataset for hand gesture recognition. We would like to thank **Twenty Billion Neurons (TwentyBN)** for creating and sharing this dataset.

- **Dataset Homepage:** [Jester Dataset](https://developer.qualcomm.com/software/jester-dataset)
- **License:** The Jester dataset is available under a Creative Commons license. Please refer to the dataset's official website for more information on its license.

## How to Use

This model consists of two parts: a Vision Transformer (ViT) feature extractor and an LSTM-based temporal classifier.

### 1. Installation

First, make sure you have the required libraries installed:

```bash
pip install tensorflow torch torchvision timm transformers
```

### 2. Loading the Models

The ViT backbone can be loaded from the Hugging Face Hub [here](https://huggingface.co/neilrigaud/hagrid-vit-gesture), while the trained LSTM model can be loaded from the `best_lstm_model.keras` file in this repository.