manavdhamecha77/iSign-t5-pose-to-text

iSign T5 Pose-to-Text Models

This repository contains trained models for Indian Sign Language Translation (SLT) using pose-based representations and T5 encoder-decoder models.

The models translate sequences of sign language pose keypoints into natural language text.

Models

The repository contains six trained checkpoints:

Model	Description
t5-small	T5-Small with spatial pose features
t5-small-motion	T5-Small with spatial and motion features
t5-base	T5-Base with spatial pose features
t5-base-motion-v2	T5-Base with spatial and motion features
t5-large	T5-Large with spatial pose features
t5-large-motion	T5-Large with spatial and motion features

Repository Structure

.
├── inference/
│   ├── inference.py
│   ├── model.py
│   ├── pose_utils.py
│   └── requirements.txt
│
├── t5-small/
│   ├── pose_encoder.pt
│   └── t5/
│       ├── config.json
│       ├── generation_config.json
│       └── model.safetensors
│
├── t5-small-motion/
│   ├── pose_encoder.pt
│   └── t5/
│       ├── config.json
│       ├── generation_config.json
│       └── model.safetensors
│
├── t5-base/
│   ├── pose_encoder.pt
│   └── t5/
│       ├── config.json
│       ├── generation_config.json
│       └── model.safetensors
│
├── t5-base-motion-v2/
│   ├── pose_encoder.pt
│   └── t5/
│       ├── config.json
│       ├── generation_config.json
│       └── model.safetensors
│
├── t5-large/
│   ├── pose_encoder.pt
│   └── t5/
│       ├── config.json
│       ├── generation_config.json
│       └── model.safetensors
│
└── t5-large-motion/
    ├── pose_encoder.pt
    └── t5/
        ├── config.json
        ├── generation_config.json
        └── model.safetensors

Model Architecture

The system uses a T5-based encoder-decoder architecture adapted for pose-to-text translation.

The pipeline consists of:

1. Pose input represented as sequences of sign language keypoints.
2. Pose encoder that projects pose features into the T5 hidden representation space.
3. T5 encoder-decoder that generates the corresponding natural language sentence.

The pose encoder is a lightweight multilayer perceptron that maps pose features into the hidden dimension of the corresponding T5 model.

The pose data is processed using the pose-format Python library.

Spatial Models

The spatial models use pose keypoints as the input representation:

• t5-small
• t5-base
• t5-large

Motion Models

The motion-enhanced models concatenate frame-wise velocity features with the original pose representation:

• t5-small-motion
• t5-base-motion-v2
• t5-large-motion

For a pose sequence (x_t), the motion feature is calculated as:

$$v_t=x_t-x_{t-1}v\_t\; =\; x\_t\; -\; x\_\{t-1\}$$

The first frame uses zero velocity.

The resulting representation contains both spatial pose information and temporal motion information.

Pose Representation

The models use the following pose components:

• POSE_LANDMARKS
• LEFT_HAND_LANDMARKS
• RIGHT_HAND_LANDMARKS

Confidence values are included when available.

Pose sequences are normalized using z-score normalization before being passed to the pose encoder.

The preprocessing implementation is provided in:

inference/pose_utils.py

Dataset

The models were trained using a subset of the iSign dataset for Indian Sign Language processing.

The experimental subset contains:

• Total samples: 18,867
• Training samples: 16,979
• Held-out test samples: 1,887

The input consists of pose files, while the target consists of corresponding textual sentence annotations.

Installation

Clone the repository:

git clone https://huggingface.co/manavdhamecha77/iSign-t5-pose-to-text
cd iSign-t5-pose-to-text

Install the inference dependencies:

pip install -r inference/requirements.txt

Alternatively:

pip install torch transformers huggingface_hub pose-format numpy

Inference

An inference implementation is provided in:

inference/inference.py

The script automatically downloads the selected checkpoint from this repository using the Hugging Face Hub.

Single Pose File

For example, using T5-Small:

python inference/inference.py \
    --pose_file path/to/sample.pose \
    --model t5-small

Using the motion-enhanced T5-Small:

python inference/inference.py \
    --pose_file path/to/sample.pose \
    --model t5-small-motion

Using the new T5-Base motion model:

python inference/inference.py \
    --pose_file path/to/sample.pose \
    --model t5-base-motion-v2

Available Model Arguments

t5-small
t5-small-motion
t5-base
t5-base-motion-v2
t5-large
t5-large-motion

GPU Inference

CUDA is automatically selected when available.

To explicitly use CUDA:

python inference/inference.py \
    --pose_file path/to/sample.pose \
    --model t5-base-motion-v2 \
    --device cuda

For CPU:

python inference/inference.py \
    --pose_file path/to/sample.pose \
    --model t5-small \
    --device cpu

Generation Parameters

The default generation settings are:

Maximum pose length: 500 frames
Maximum output length: 128 tokens
Beam size: 4
Length penalty: 2.0

The beam size and maximum generation length can be changed:

python inference/inference.py \
    --pose_file path/to/sample.pose \
    --model t5-small \
    --max_pose_length 500 \
    --max_length 128 \
    --num_beams 4

Loading the T5 Component

The T5 component and custom pose encoder are stored separately.

For example:

from transformers import T5ForConditionalGeneration

model = T5ForConditionalGeneration.from_pretrained(
    "manavdhamecha77/iSign-t5-pose-to-text",
    subfolder="t5-small/t5"
)

The corresponding custom pose encoder is:

t5-small/pose_encoder.pt

The complete inference pipeline should be used rather than loading the T5 component alone because the model expects pose embeddings produced by the custom pose encoder.

Model Details

Model	T5 Size	Pose Features	Motion
t5-small	Small	Spatial	No
t5-small-motion	Small	Spatial + Motion	Yes
t5-base	Base	Spatial	No
t5-base-motion-v2	Base	Spatial + Motion	Yes
t5-large	Large	Spatial	No
t5-large-motion	Large	Spatial + Motion	Yes

Files in Each Checkpoint

Each trained model contains:

pose_encoder.pt
t5/
├── config.json
├── generation_config.json
└── model.safetensors

model.safetensors contains the trained T5 parameters.

pose_encoder.pt contains the trained custom pose encoder parameters.

config.json and generation_config.json contain the T5 model configuration and generation configuration.

Reproducibility

For reproducible inference, use the preprocessing implementation included in:

inference/pose_utils.py

This ensures that pose loading, component selection, confidence handling, normalization, padding/truncation, and motion feature generation follow the same implementation used by the models.

Citation

If you use these models in your research, please cite the associated work:

@misc{dhamecha2026isign,
  title={Pose-Based Indian Sign Language Translation using T5 and Motion Features},
  author={Manav Dhamecha},
  year={2026}
}

License

Please refer to the original iSign dataset license and terms of use when using the dataset or derived models.

Acknowledgements

This work uses:

• The iSign dataset
• Hugging Face Transformers
• PyTorch
• The pose-format Python library
• T5 models