README.md

metadata

title: Hand2Voice
emoji: 🤟
colorFrom: gray
colorTo: indigo
sdk: streamlit
pinned: false
short_description: Converting Hand Gestures into Speech using Computer Vision

---

🤟 Hand2Voice: AI Sign Language Assistant

Hand2Voice is an accessibility tool designed to bridge the communication gap for the speech-impaired. It uses computer vision to translate hand gestures into spoken audio in real-time.

Unlike basic classifiers that rely on static screen coordinates, this project utilizes Euclidean geometry to calculate relative finger positions, ensuring accurate detection even if the hand is rotated or tilted.

🚀 Key Features

• 📷 Dual Input Modes: Supports real-time camera capture and image uploads.
• 🧠 Robust Recognition Logic: Uses 3D Euclidean distance calculations relative to the wrist, making detection rotation-invariant.
• 🦴 Skeletal Visualization: Real-time feedback overlay showing exactly what the computer vision model "sees."
• 🗣️ Text-to-Speech (TTS): Instantly vocalizes the detected gesture using Google TTS.
• 📂 JSON-Based Rule Engine: Gestures are defined in an external gesture_rules.json file, making it easy to add new signs without changing code.

🛠️ Tech Stack

• Frontend: Streamlit (Web Interface)
• Computer Vision: MediaPipe Hands (Google)
• Image Processing: OpenCV & NumPy
• Audio: gTTS (Google Text-to-Speech)

📂 Project Structure

Hand2Voice/
├── app.py                 # Main Streamlit application (UI & Logic)
├── gesture_classifier.py  # Advanced logic using Euclidean distance
├── gesture_rules.json     # Database of supported gestures
├── tts.py                 # Text-to-Speech helper function
├── requirements.txt       # List of python dependencies
├── NIELIT-LOGO.png        # Institution Logo
└── README.md              # Project Documentation

💿 Installation & Setup

1. Clone the Repository

git clone [https://github.com/imarshbir/Hand2Voice.git](https://github.com/imarshbir/Hand2Voice.git)
cd Hand2Voice

2. Create a Virtual Environment (Optional but Recommended)

python -m venv venv
# Windows
venv\Scripts\activate
# Mac/Linux
source venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

4. Run the Application

streamlit run app.py

🤟 Supported Gestures

The current version supports the following gestures (defined in gesture_rules.json):

Gesture	Description
HELLO	Open palm (All fingers extended)
YES / POINT	Index finger raised
NO	Closed fist
PEACE	Index & Middle fingers raised (V-sign)
OK	Thumb & Index touching, others extended
ROCK ON	Index & Pinky extended
THUMBS UP	Thumb extended only

🔬 How It Works (Technical Deep Dive)

Most basic hand gesture tutorials use simple if y_tip < y_knuckle logic. This fails if the hand is tilted sideways.

Hand2Voice solves this by using Vector Math:

1. Landmark Extraction: MediaPipe extracts 21 3D landmarks () for the hand.
2. Distance Calculation: We calculate the Euclidean distance between each fingertip and the Wrist (Landmark 0). $$d=\sqrt{(x_2-x_1{)}^2+(y_2-y_1{)}^2+(z_2-z_1{)}^2}d\; =\; \backslash sqrt\{(x\_2\; -\; x\_1)^2\; +\; (y\_2\; -\; y\_1)^2\; +\; (z\_2\; -\; z\_1)^2\}$$
3. State Determination:

• If , the finger is considered OPEN.
• Otherwise, it is CLOSED.

4. Pattern Matching: The resulting binary array (e.g., [0, 1, 1, 0, 0]) is compared against the definitions in gesture_rules.json.

🔮 Future Scope

• Real-time Video Stream: Integration with streamlit-webrtc for continuous streaming without clicking "capture."
• Dynamic Gestures: Support for moving gestures (like waving) using LSTM networks.
• Multi-Language Support: Adding Hindi/Punjabi TTS output.

👨‍💻 Author

Arshbir Singh

• @imarshbir

• Expertise: AI, Computer Vision, IoT

• B.Tech (CSE) Researcher