| --- |
| language: |
| - en |
| license: cc-by-nc-sa-4.0 |
| tags: |
| - yoga |
| - posture-correction |
| - pose-estimation |
| - pytorch |
| - resmlp |
| - gru |
| - attention |
| - computer-vision |
| - mediapipe |
| datasets: |
| - custom |
| metrics: |
| - accuracy |
| - f1 |
| pipeline_tag: tabular-classification |
| model-index: |
| - name: YogaResMLP (Single-Head Classifier) |
| results: |
| - task: |
| type: tabular-classification |
| dataset: |
| name: yoga-pose-features-dataset |
| type: custom |
| metrics: |
| - name: Validation Pose Accuracy |
| type: accuracy |
| value: 92.84 |
| - name: Yoga3HeadMLP (Multi-Output Model) |
| results: |
| - task: |
| type: tabular-classification |
| dataset: |
| name: yoga-pose-features-dataset |
| type: custom |
| metrics: |
| - name: Base Pose Identification Accuracy |
| type: accuracy |
| value: 93.38 |
| - name: Pose Correctness Accuracy |
| type: accuracy |
| value: 96.81 |
| - name: YogaSequenceLSTM (ST-GCN/Sequence Model) |
| results: |
| - task: |
| type: sequence-classification |
| dataset: |
| name: yoga-pose-features-dataset |
| type: custom |
| metrics: |
| - name: Flow Sequence Validation Accuracy |
| type: accuracy |
| value: 75.25 |
| --- |
| |
| # Smart Yoga Posture Correction System (Project P05) |
| This repository hosts the model weights and label encoders for the **Smart Yoga Posture Correction System** (Final Year Project P05, RCC IIT Kolkata). |
|
|
| The system leverages a multi-model cooperative framework to classify and correct yoga poses: |
| 1. **Single-Head ResMLP Model (`mlp_model.pth`)**: A frame-level static posture classifier trained on 15 biomechanical joint angles, achieving **92.84%** validation accuracy across 29 classes. |
| 2. **3-Head MLP Model (`mlp_3head_model.pth`)**: A multi-output static posture model predicting **Pose ID** (across 23 base classes, achieving **93.38%** pose accuracy), **Pose Correctness** (achieving **96.81%** accuracy), and **Joint Angle Deviations** (regression output) simultaneously. |
| 3. **Sequence Flow Model (`stgcn_sequence_model.pth`)**: A hybrid 1D Temporal Convolution + Stacked Residual GRU + Self-Attention model trained on 60-frame skeleton coordinate sequences, achieving **75.25%** validation accuracy across 27 classes. |
| |
| All models incorporate class-weight smoothing and normalization techniques to resolve pose imbalance and coordinate noise. |
| |
| --- |
| |
| ## Model Architectures & Training Logs |
| |
| ### 1. Static Pose Classifier (Single-Head ResMLP) |
| |
| #### Architecture |
| The **ResMLP** classifier processes 15 frame-level joint angles (computed from MediaPipe Pose landmarks): |
| * **Input Layer**: `Linear(15 -> 256)` followed by Batch Normalization and `GELU` activation. |
| * **Residual blocks**: 2 stacked residual blocks. Each block consists of: |
| * `Linear(256 -> 256)` -> `BatchNorm1d` -> `GELU` -> `Dropout(0.3)` |
| * `Linear(256 -> 256)` -> `BatchNorm1d` -> `GELU` -> `Dropout(0.3)` |
| * Residual skip connection: `x_out = x + block(x)` |
| * **Classification Head**: `Linear(256 -> 128)` -> `BatchNorm1d` -> `GELU` -> `Dropout(0.2)` -> `Linear(128 -> 29)`. |
| |
| #### Dataset & Preprocessing |
| * **Dataset size**: 654,488 frames in total. |
| * **Train size**: 523,590 frames |
| * **Validation size**: 130,898 frames |
| * **Class Weights**: Smoothed using the square-root count inverse function `1.0 / sqrt(count)` to prevent minor classes (such as `transition/unknown` and `lunge_pose`) from dominating the gradients. |
| |
| #### Training Performance & Curves |
| * **Best Validation Loss**: **0.1644** at Epoch 39. |
| * **Final Epoch (40/40)**: |
| * **Train Loss**: 0.2238 | **Train Acc**: 90.78% |
| * **Val Loss**: 0.1651 | **Val Acc**: 92.84% |
| |
| Below is the training progress for selected epochs: |
| | Epoch | Train Loss | Train Acc | Val Loss | Val Acc | |
| |---|---|---|---|---| |
| | Epoch 01 | 0.6523 | 77.57% | 0.3930 | 83.94% | |
| | Epoch 02 | 0.4576 | 82.71% | 0.3231 | 86.79% | |
| | Epoch 03 | 0.4080 | 84.19% | 0.3005 | 87.17% | |
| | Epoch 04 | 0.3811 | 85.05% | 0.2700 | 88.29% | |
| | Epoch 05 | 0.3620 | 85.71% | 0.2756 | 87.24% | |
| | Epoch 10 | 0.3102 | 87.56% | 0.2421 | 89.24% | |
| | Epoch 20 | 0.2732 | 89.00% | 0.2091 | 90.78% | |
| | Epoch 30 | 0.2420 | 90.18% | 0.1872 | 91.57% | |
| | Epoch 39 | 0.2259 | 90.73% | **0.1644** | 92.66% | |
| | Epoch 40 | 0.2238 | 90.78% | 0.1651 | **92.84%** | |
| |
| #### Static Pose Classification Report |
| ``` |
| precision recall f1-score support |
| |
| chair_pose 0.56 0.94 0.70 366 |
| chaturanga 0.45 1.00 0.62 5 |
| child 0.06 0.57 0.10 7 |
| child_pose 0.91 0.99 0.95 3260 |
| cobra_pose 0.90 0.96 0.93 5116 |
| corpse 0.36 0.85 0.51 20 |
| downward_dog 0.90 0.95 0.92 4398 |
| halfway_lift 0.55 0.94 0.70 479 |
| imperfect_corpse 0.66 0.97 0.78 290 |
| imperfect_plank 0.86 0.96 0.91 1825 |
| imperfect_seated_forward 0.87 0.99 0.92 938 |
| imperfect_triangle 0.87 0.96 0.91 2607 |
| imperfect_upward_dog 0.91 0.97 0.94 2556 |
| lunge_pose 0.97 0.93 0.95 19496 |
| mountain_pose 0.77 0.97 0.86 1233 |
| plank 0.58 0.63 0.61 174 |
| seated_easy_pose 0.94 0.97 0.95 17465 |
| seated_forward 0.91 0.96 0.94 75 |
| seated_staff 0.80 0.94 0.86 1600 |
| standing_forward_fold 0.95 0.96 0.96 7907 |
| standing_pose 0.85 0.92 0.89 1405 |
| table_top 0.51 0.94 0.66 501 |
| transition/unknown 0.98 0.88 0.93 44781 |
| tree_pose 0.73 0.97 0.83 1474 |
| triangle 0.58 0.75 0.66 485 |
| upward_dog 0.42 0.60 0.49 67 |
| upward_salute 0.76 0.99 0.86 528 |
| warrior_1 0.94 0.98 0.96 4736 |
| warrior_2 0.88 0.95 0.91 7104 |
| |
| weighted avg 0.94 0.93 0.93 130898 |
| accuracy 0.93 130898 |
| ``` |
| |
| --- |
| |
| ### 2. Multi-Output Posture Correction Model (3-Head MLP) |
| |
| #### Architecture |
| The **3-Head MLP** classifier processes 15 frame-level joint angles (computed from MediaPipe Pose landmarks): |
| * **Shared Feature Trunk**: |
| * Input layer `Linear(15 -> 256)` -> `BatchNorm1d` -> `GELU` activation. |
| * 2 stacked residual blocks (`ResBlock` of size 256). Each block contains: |
| * `Linear(256 -> 256)` -> `BatchNorm1d` -> `GELU` -> `Dropout(0.3)` |
| * `Linear(256 -> 256)` -> `BatchNorm1d` -> `GELU` -> `Dropout(0.3)` |
| * Skip connection: `x_out = x + block(x)` |
| * **Head 1: Pose ID (Classification)**: |
| * `Linear(256 -> 128)` -> `BatchNorm1d` -> `GELU` -> `Dropout(0.2)` -> `Linear(128 -> 23)` (Softmax over 23 base posture classes). |
| * **Head 2: Correctness (Binary Classification)**: |
| * `Linear(256 -> 64)` -> `BatchNorm1d` -> `GELU` -> `Dropout(0.2)` -> `Linear(64 -> 1)` (Binary Logit output: correct vs. imperfect/transition). |
| * **Head 3: Joint Deviation (Regression)**: |
| * `Linear(256 -> 128)` -> `BatchNorm1d` -> `GELU` -> `Dropout(0.2)` -> `Linear(128 -> 15)` (Predicts normalized deviation values in $[0, 1]$ where 1.0 represents 180° deviation). |
|
|
| #### Dataset & Preprocessing |
| * **Dataset size**: 654,488 frames in total. |
| * **Train size**: 523,590 frames |
| * **Validation size**: 130,898 frames |
| * **Class Weights**: Smoothed using the square-root count inverse function `1.0 / sqrt(count)` to prevent major classes (such as `transition/unknown` and `lunge_pose`) from dominating the Pose ID loss gradients. |
| * **Loss Function**: $\mathcal{L}_{total} = \mathcal{L}_{pose} + \mathcal{L}_{correctness} + \mathcal{L}_{deviation}$ (combining Cross-Entropy, Binary Cross-Entropy with logits, and Huber SmoothL1 loss). |
|
|
| #### Training Performance & Curves |
| * **Best Validation Loss**: **0.2263** at Epoch 39/40. |
| * **Validation Pose Accuracy**: **93.38%** |
| * **Validation Correctness Accuracy**: **96.81%** |
|
|
| Below is the training progress for selected epochs: |
| | Epoch | Train Loss | Train Pose Acc | Val Loss | Val Pose Acc | Val Correctness Acc | |
| |---|---|---|---|---|---| |
| | Epoch 01 | 0.8631 | 79.04% | 0.5059 | 86.08% | 92.83% | |
| | Epoch 02 | 0.6321 | 83.78% | 0.4506 | 87.46% | 93.72% | |
| | Epoch 03 | 0.5702 | 85.20% | 0.3939 | 89.38% | 94.36% | |
| | Epoch 04 | 0.5321 | 86.14% | 0.3811 | 88.70% | 94.51% | |
| | Epoch 05 | 0.5055 | 86.71% | 0.3550 | 90.19% | 94.75% | |
| | Epoch 10 | 0.4389 | 88.33% | 0.3079 | 91.43% | 95.30% | |
| | Epoch 20 | 0.3864 | 89.72% | 0.2873 | 91.54% | 95.60% | |
| | Epoch 30 | 0.3597 | 90.42% | 0.2545 | 92.22% | 96.42% | |
| | Epoch 39 | 0.3224 | 91.36% | **0.2263** | **93.38%** | **96.81%** | |
| | Epoch 40 | 0.3215 | 91.37% | 0.2380 | 92.62% | 96.65% | |
|
|
| --- |
|
|
| ### 3. Sequence Flow Classifier (ST-GCN/GRU-Attention) |
|
|
| #### Architecture |
| The sequence classifier processes **60-frame coordinate sequences** (shape `[batch_size, 60, 99]`, representing 33 joints in 3D): |
| * **Coordinate Normalization**: Translates coordinate sequences to be **pelvis-centered** (using the midpoint between the left and right hip joints) and divides by hip-width. This guarantees absolute **translation and scale invariance**. |
| * **1D Temporal Convolution**: `Conv1d(in_channels=99, out_channels=128, kernel_size=5, padding=2)` -> `BatchNorm1d` -> `GELU` -> `Dropout(0.2)` to smooth coordinate sequence noise. |
| * **Stacked Residual GRU blocks**: Two bidirectional GRU blocks with hidden dimension 128. Output is projected back from 256 to 128, normalized with LayerNorm, dropped out with 30% rate, and summed with input (residual connection). |
| * **Self-Attention Pooling**: Learns step importance weights dynamically and returns a weighted summary vector across the 60-frame window. |
| * **Classification Head**: `Linear(128 -> 64)` -> `GELU` -> `Dropout(0.3)` -> `Linear(64 -> 27)`. |
|
|
| #### Dataset & Preprocessing |
| * **Total sequences**: 18,165 (60-frame windows). |
| * **Train size**: 14,532 sequences. |
| * **Validation size**: 3,633 sequences. |
| * **Training Hyperparameters**: |
| * Batch Size: 64 |
| * Optimizer: `AdamW(lr=2e-3, weight_decay=1e-3)` |
| * Target Metric: Best Validation Accuracy. |
|
|
| #### Training Performance & Curves |
| * **Best Validation Accuracy**: **75.25%** at Epoch 90. |
| * **Early Stopping**: Triggered at Epoch 110. |
|
|
| Selected epochs during training: |
| | Epoch | Train Loss | Train Acc | Val Loss | Val Acc | |
| |---|---|---|---|---| |
| | Epoch 01 | 3.7886 | 37.03% | 3.4506 | 45.09% | |
| | Epoch 02 | 3.4884 | 42.83% | 3.2596 | 49.55% | |
| | Epoch 10 | 2.9655 | 60.80% | 2.8399 | 64.35% | |
| | Epoch 20 | 2.7688 | 67.86% | 2.7106 | 68.98% | |
| | Epoch 30 | 2.6449 | 71.72% | 2.6624 | 69.83% | |
| | Epoch 50 | 2.5094 | 76.72% | 2.6160 | 72.20% | |
| | Epoch 90 | 2.3185 | 83.82% | 2.5877 | **75.25%** | |
| | Epoch 110 | 2.2777 | 85.25% | 2.6001 | 74.40% (Early Stopping) | |
|
|
| --- |
|
|
| ## Inference and Usage Guide |
|
|
| All model state dicts and label encoder maps can be downloaded and loaded in Python as follows: |
|
|
| ```python |
| import numpy as np |
| import torch |
| import torch.nn as nn |
| |
| # Load label encoders |
| mlp_classes = np.load("mlp_label_encoder.npy", allow_pickle=True) |
| mlp_3head_classes = np.load("mlp_3head_pose_encoder.npy", allow_pickle=True) |
| stgcn_classes = np.load("stgcn_label_encoder.npy", allow_pickle=True) |
| |
| # 1. Instantiate the Single-Head ResMLP Model |
| mlp_model = YogaMLP(input_dim=15, num_classes=len(mlp_classes)) |
| mlp_model.load_state_dict(torch.load("mlp_model.pth", map_location="cpu")) |
| mlp_model.eval() |
| |
| # 2. Instantiate the 3-Head MLP Model |
| mlp_3head_model = Yoga3HeadMLP(input_dim=15, num_poses=len(mlp_3head_classes)) |
| mlp_3head_model.load_state_dict(torch.load("mlp_3head_model.pth", map_location="cpu")) |
| mlp_3head_model.eval() |
| |
| # 3. Instantiate the Sequence Model |
| sequence_model = YogaSequenceLSTM(input_dim=99, hidden_dim=128, num_layers=2, num_classes=len(stgcn_classes)) |
| sequence_model.load_state_dict(torch.load("stgcn_sequence_model.pth", map_location="cpu")) |
| sequence_model.eval() |
| ``` |
|
|
| ### Cooperative Prediction Protocol |
| For production deployment (e.g. FastAPI backend): |
| 1. Extract frame joint coordinate sequences (shape `[N, 60, 99]`) using MediaPipe. |
| 2. If the sequence is classified by `stgcn_sequence_model.pth` as `transition/unknown`, the backend falls back to using either the static single-head `mlp_model.pth` or the multi-output `mlp_3head_model.pth` classifier on individual frames. |
| 3. This cooperative approach minimizes false positives, provides real-time latency optimization, and ensures smooth transition tracking while practicing. |
|
|
| --- |
| **RCC Institute of Information Technology, Kolkata** |
| *Department of Computer Science & Engineering* |
| *Final Year Project 2026* |
|
|
