--- title: LumiSign emoji: 🚀 colorFrom: blue colorTo: red sdk: docker sdk_version: 4.19.2 python_version: 3.9 app_file: app.py pinned: false short_description: Indian Sign Language Recognition Web UI --- # LumiSign Indian Sign Language recognition with FastAPI + React UI, using a Transformer model trained on `isl-split-dataset`. This README is written for developers who are new to the project. ## What You Need 1. Python `3.9` 2. Node.js `18+` and npm 3. Git 4. A Linux/macOS shell (or WSL on Windows) ## Project Layout ```text LumiSign/ app.py inference.py runner.py prepare_custom_dataset.py check_split_leakage.py transformer_large.pth label_maps/ label_map_isl_split_dataset.json isl-split-dataset/ train/ eval/ test/ ui/ ``` ## Dataset Layout Expected `prepare_custom_dataset.py` expects this format: ```text isl-split-dataset/ train/ bank/ court/ store or shop/ eval/ bank/ court/ store or shop/ test/ bank/ court/ store or shop/ ``` Labels are normalized internally: - `store or shop` becomes `storeorshop` ## Quick Start (Use Existing `transformer_large.pth`) Run these from the project root. ### 1) Create and install environment ```bash python3.9 -m venv venv source venv/bin/activate venv/bin/pip install --upgrade pip venv/bin/pip install -r requirements.txt ``` ### 2) Build frontend once ```bash cd ui npm install npm run build cd .. ``` ### 3) Start backend with ISL model ```bash MODEL_CHECKPOINT=./transformer_large.pth \ MODEL_LABEL_MAP_PATH=./label_maps/label_map_isl_split_dataset.json \ MODEL_DATASET=isl_split_dataset \ MODEL_TYPE=transformer \ MODEL_TRANSFORMER_SIZE=large \ MODEL_MAX_FRAME_LEN=169 \ venv/bin/python -m uvicorn app:app --host 0.0.0.0 --port 8070 --reload ``` Open: - UI: `http://localhost:8070` - API docs: `http://localhost:8070/docs` ## Full Pipeline (Prepare Data -> Train -> Evaluate) Use this if you want to regenerate keypoints and retrain. ### 1) Set paths ```bash PROJECT_ROOT="$(pwd)" ISL_SPLIT_ROOT="$PROJECT_ROOT/isl-split-dataset" ISL_DATASET_NAME="isl_split_dataset" ISL_PROCESSED_DIR="$PROJECT_ROOT/processed_data_islsplit" ISL_CKPT_DIR="$PROJECT_ROOT/checkpoints_islsplit" mkdir -p "$ISL_PROCESSED_DIR" "$ISL_CKPT_DIR" ``` ### 2) Convert split folders to keypoint JSON files ```bash venv/bin/python prepare_custom_dataset.py \ --data_dir "$ISL_SPLIT_ROOT" \ --save_dir "$ISL_PROCESSED_DIR" \ --dataset_name "$ISL_DATASET_NAME" \ --jobs 4 \ --use_holistic \ --face_mode full \ --write_placeholders ``` This creates: - `processed_data_islsplit/isl_split_dataset_train_keypoints` - `processed_data_islsplit/isl_split_dataset_val_keypoints` - `processed_data_islsplit/isl_split_dataset_test_keypoints` - `label_maps/label_map_isl_split_dataset.json` ### 3) Check split leakage ```bash venv/bin/python check_split_leakage.py \ --data_dir "$ISL_PROCESSED_DIR" \ --dataset "$ISL_DATASET_NAME" \ --dark_suffix __dark ``` ### 4) Train Transformer (large) ```bash venv/bin/python runner.py \ --dataset "$ISL_DATASET_NAME" \ --model transformer \ --transformer_size large \ --max_frame_len 169 \ --data_dir "$ISL_PROCESSED_DIR" \ --save_path "$ISL_CKPT_DIR" \ --batch_size 8 \ --early_stop_metric val_loss \ --early_stop_patience 5 ``` ### 5) Evaluate on test split ```bash venv/bin/python runner.py \ --dataset "$ISL_DATASET_NAME" \ --model transformer \ --transformer_size large \ --max_frame_len 169 \ --data_dir "$ISL_PROCESSED_DIR" \ --save_path "$ISL_CKPT_DIR" \ --batch_size 1 \ --epochs 0 \ --eval_split test ``` ## Single Video CLI Inference ```bash venv/bin/python inference.py \ --video /absolute/path/to/video.mp4 \ --dataset isl_split_dataset \ --model transformer \ --transformer_size large \ --checkpoint ./transformer_large.pth \ --label_map_path ./label_maps/label_map_isl_split_dataset.json \ --max_frame_len 169 ``` ## Run UI + API in Development Terminal 1: ```bash MODEL_CHECKPOINT=./transformer_large.pth \ MODEL_LABEL_MAP_PATH=./label_maps/label_map_isl_split_dataset.json \ MODEL_DATASET=isl_split_dataset \ MODEL_TYPE=transformer \ MODEL_TRANSFORMER_SIZE=large \ MODEL_MAX_FRAME_LEN=169 \ venv/bin/python -m uvicorn app:app --host 0.0.0.0 --port 8070 --reload ``` Terminal 2: ```bash cd ui npm install npm run dev ``` If Vite starts on `5173`, open `http://localhost:5173`. ## Common Errors and Fixes ### 1) `size mismatch for l2.weight` when loading checkpoint Cause: - Label map class count does not match checkpoint output classes. Fix: ```bash MODEL_CHECKPOINT=./transformer_large.pth \ MODEL_LABEL_MAP_PATH=./label_maps/label_map_isl_split_dataset.json \ MODEL_DATASET=isl_split_dataset ``` Your ISL checkpoint expects 3 classes: `bank`, `court`, `storeorshop`. ### 2) CUDA warning like `Error 804: forward compatibility was attempted` Cause: - GPU driver/CUDA compatibility mismatch. Fix: - Run on CPU, or align NVIDIA driver + CUDA + PyTorch versions. - This warning is not the same as the class mismatch error. ### 3) `No label map found for dataset ...` Fix: - Ensure `label_maps/label_map_isl_split_dataset.json` exists. - Pass `MODEL_LABEL_MAP_PATH` explicitly. ## Useful Entrypoints - `app.py`: FastAPI server startup and model loading - `inference.py`: model load and single-video prediction - `prepare_custom_dataset.py`: convert split video folders to keypoint JSON - `check_split_leakage.py`: verify train/val/test split leakage - `runner.py`: train and evaluate models ## Reference Commands - Full command history used in this project is in `commands.txt`. ## Citation If you use this work, cite INCLUDE : ```bibtex @inproceedings{10.1145/3394171.3413528, author = {Sridhar, Advaith and Ganesan, Rohith Gandhi and Kumar, Pratyush and Khapra, Mitesh}, title = {INCLUDE: A Large Scale Dataset for Indian Sign Language Recognition}, year = {2020}, isbn = {9781450379885}, publisher = {Association for Computing Machinery}, doi = {10.1145/3394171.3413528}, series = {MM '20} } ```