VSRS — V1 Machine Learning Models
Visual Speech Recognition System
Real-Time Indian Sign Language Recognition
This repository contains the trained machine-learning models developed for Version 1 (V1/MVP) of the Visual Speech Recognition System (VSRS).
VSRS is a real-time Indian Sign Language recognition system capable of identifying a predefined vocabulary of isolated signs and converting recognized signs into text.
Main GitHub Repository:
https://github.com/YojitArora/VSRS
Development Version: V1 / MVP
Status: V1 Feature Complete
1. Project Overview
VSRS aims to reduce communication barriers between Indian Sign Language (ISL) users and individuals who do not understand sign language.
The system uses computer vision and machine learning to recognize predefined ISL signs through ordinary webcam input.
The V1 implementation supports 11 isolated signs and converts recognized signs into text.
Recognition Pipeline
Webcam Input
|
v
OpenCV Frame Capture
|
v
MediaPipe HandLandmarker
|
v
Hand Landmark Extraction
|
v
Feature Normalization
|
v
126-Dimensional Feature Vector
|
v
Machine Learning Classifier
|
v
5-Frame Probability Aggregation
|
v
Recognized ISL Sign
|
v
Text Output
2. Supported Vocabulary
V1 supports 11 Indian Sign Language classes:
| Class | Sign |
|---|---|
| 1 | cellphone |
| 2 | good |
| 3 | hello |
| 4 | house |
| 5 | i |
| 6 | loud |
| 7 | quiet |
| 8 | red |
| 9 | shoes |
| 10 | teacher |
| 11 | time |
The models perform classification within this predefined vocabulary.
They do not support unrestricted or continuous Indian Sign Language translation.
3. Dataset
The models were trained using an 11-class subset of the INCLUDE-50 Indian Sign Language dataset.
Dataset Distribution
| Partition | Videos | Feature Samples |
|---|---|---|
| Training | 152 | 760 |
| Validation | 18 | 90 |
| Testing | 39 | 195 |
| Total | 209 | 1,045 |
Each video contributes five sampled frames.
Dataset Source
Dataset: INCLUDE / INCLUDE-50
Paper:
Sridhar et al., "INCLUDE: A Large Scale Dataset for Indian Sign Language Recognition", ACM Multimedia 2020.
DOI:
https://doi.org/10.1145/3394171.3413528
Official Dataset:
https://zenodo.org/records/4010759
Dataset License: CC BY 4.0
The training data originates from the INCLUDE dataset. The original dataset and its attribution requirements remain separate from the licensing of the trained model artifacts.
4. Feature Extraction
Hand landmarks are extracted using MediaPipe HandLandmarker.
Each detected hand contains:
- 21 landmarks
- 3 coordinates per landmark (x, y, z)
- 63 numerical features per hand
The system supports two hands.
Total feature dimensions:
21 × 3 × 2 = 126 features
Feature Representation
| Feature Range | Description |
|---|---|
| 0–62 | Right-hand landmarks |
| 63–125 | Left-hand landmarks |
Missing-hand slots are zero-padded during feature extraction.
Normalization
The preprocessing pipeline performs:
- Wrist-centered coordinate translation.
- Maximum-radius scale normalization.
SVM and MLP additionally use a StandardScaler fitted exclusively on training data.
Random Forest uses the normalized features without StandardScaler.
All models require the same 126-dimensional feature ordering used during training.
5. Available Models
Three machine-learning models are available.
Model 1 — Multi-Layer Perceptron (MLP)
Status: Official V1 Validation-Selected Model
Architecture:
Input Layer: 126 features
|
v
Hidden Layer 1: 256 neurons
|
v
Hidden Layer 2: 128 neurons
|
v
Output Layer: 11 classes
Configuration:
- Algorithm: MLPClassifier
- Hidden layers: (256, 128)
- Activation: ReLU
- Optimizer: Adam
- Alpha: 0.01
- Maximum iterations: 500
- Actual training iterations: 136
- Preprocessing: StandardScaler
Artifact: mlp/model.joblib
This is the original trained model selected during V1 development.
Model 2 — Random Forest
Status: Reproduced V1 Experimental Baseline
Configuration:
- Algorithm: RandomForestClassifier
- Decision trees: 100
- Maximum depth: None
- Random seed: 42
- Feature scaling: Not required
Artifact: random_forest/model.joblib
This model was reproduced using the original V1 training configuration and dataset.
Model 3 — Linear Support Vector Machine
Status: Reproduced V1 Experimental Baseline
Configuration:
- Algorithm: SVC
- Kernel: Linear
- C: 1.0
- Probability estimates: Enabled
- Random seed: 42
- Preprocessing: StandardScaler
Artifact: linear_svm/model.joblib
This model was reproduced using the original V1 training configuration and dataset.
6. Model Evaluation
The original V1 experiment compared seven candidate configurations across Random Forest, SVM, and MLP models.
Model selection was based on validation performance.
Validation Results
| Model | Video Accuracy | Frame Accuracy |
|---|---|---|
| MLP (256,128) | 100% | 92.22% |
| Random Forest (100 trees) | 100% | 88.89% |
| Linear SVM (C=1) | 100% | 88.89% |
Historical V1 Test Results
| Model | Video Accuracy | Correct Videos |
|---|---|---|
| MLP (256,128) | 89.74% | 35/39 |
| Random Forest (100 trees) | 89.74% | 35/39 |
| Linear SVM (C=1) | 92.31% | 36/39 |
Important:
The MLP was selected using the predefined validation-based selection procedure.
Although Linear SVM achieved a higher observed test accuracy, the selected MLP was not retroactively replaced using test-set results.
The Random Forest and Linear SVM files in this repository are reproduced artifacts. Their validation results were verified during reproduction; their historical test results are reported from the original V1 experiment.
These results were obtained on the INCLUDE-50 V1 subset and should not be interpreted as guaranteed accuracy for arbitrary webcam users.
7. Temporal Prediction Aggregation
VSRS uses mean-probability aggregation.
Offline Dataset Evaluation
Five frames are sampled uniformly from approximately 42%–58% of each prerecorded video.
This window is an engineering sampling heuristic based on representative video observations.
Real-Time Inference
Live webcam recognition uses a rolling buffer of five consecutive valid prediction frames.
The class probability estimates are averaged to obtain a more stable final prediction.
When no hands are detected, the live inference system clears its prediction buffer.
8. Repository Structure
VSRS-V1-Models/
|
|-- mlp/
| |-- model.joblib
|
|-- random_forest/
| |-- model.joblib
|
|-- linear_svm/
| |-- model.joblib
|
|-- README.md
Each artifact contains:
- Trained classifier
- Fitted scaler where required
- Label encoder
- Class-label mapping
- Model configuration
- Prediction aggregation strategy
- Relevant evaluation metadata
9. Downloading a Model
Install the required packages:
pip install huggingface_hub joblib scikit-learn numpy
Download the official V1 MLP:
from huggingface_hub import hf_hub_download
import joblib
model_path = hf_hub_download(
repo_id="YojitArora/VSRS-V1-Models",
filename="mlp/model.joblib"
)
artifact = joblib.load(model_path)
model = artifact["model"]
scaler = artifact["scaler"]
classes = artifact["classes"]
print("Model:", type(model).__name__)
print("Supported classes:", classes)
The downloaded model requires correctly preprocessed 126-dimensional MediaPipe hand-landmark features.
Raw images cannot be passed directly to these classifiers.
For the complete real-time recognition pipeline, refer to the main VSRS GitHub repository.
Security Note
These models use Python joblib serialization.
Only load artifacts from trusted sources because loading pickle-based model files can execute arbitrary code.
Compatible Python and scikit-learn environments are required.
For reproducible deployment, pin compatible dependency versions and a specific Hugging Face repository revision.
10. Model Artifact Integrity
SHA-256 fingerprints of the verified local model files:
| Model | SHA-256 |
|---|---|
| MLP | 4f13ec96dee5999d2fb6ff8cefbd10d23f80b762ef2ddbe1c7656a5a00b23098 |
| Random Forest | 2339d1e4381cf6db9e9eba386ab39575c5e1cfc11fa4d3b5b3a9683b836ae444 |
| Linear SVM | c3143f5fa9d7e2561bf255746b140d459a7f4a5dfc71745be6377a29d33d3d55 |
These fingerprints can be used to verify downloaded artifacts.
11. Current Limitations
VSRS V1 has the following limitations:
- Recognition limited to 11 predefined signs.
- Primarily designed for static/hold-dominant isolated signs.
- No continuous sign-language recognition.
- No explicit dynamic trajectory modelling.
- No sentence-level translation.
- No speech generation.
- Hand detection can be affected by occlusion and camera conditions.
- Performance on unseen signers and environments requires further validation.
- Predictions are restricted to the known class vocabulary.
The system is a functional research prototype, not a replacement for professional sign-language interpreters.
12. Future Development
The VSRS development roadmap includes:
V2 — Vocabulary Expansion
Extend the supported Indian Sign Language vocabulary.
V3 — Dynamic Sign Recognition
Introduce recognition of signs involving meaningful temporal motion.
V4 — Sentence Recognition
Progress toward continuous sign recognition and sentence construction.
V5 — Text-to-Speech
Convert generated text into audible speech.
Future models will be developed and evaluated separately.
13. Project Repository
GitHub:
https://github.com/YojitArora/VSRS
This Hugging Face repository serves as the model artifact collection for VSRS V1.
The GitHub repository contains the implementation, preprocessing, training, inference, testing, and technical documentation.
14. References
Sridhar et al. (2020). INCLUDE: A Large Scale Dataset for Indian Sign Language Recognition. ACM Multimedia. https://doi.org/10.1145/3394171.3413528
INCLUDE Dataset. Zenodo. https://zenodo.org/records/4010759
AI4Bharat INCLUDE Repository. https://github.com/AI4Bharat/INCLUDE
MediaPipe Hand Landmarker Documentation. https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker
scikit-learn Documentation. https://scikit-learn.org/
15. Model Repository Status
Version: VSRS V1 / MVP
ML Models: 3
Supported Signs: 11
Original Production Model: MLPClassifier (256,128)
Model Collection: Uploaded to Hugging Face
Next Development Phase: V2 — Vocabulary Expansion