---
library_name: pytorch
license: other
tags:
  - sign-language-recognition
  - african-sign-language
  - multimodal
  - pose-estimation
  - rgb-video
  - metric-learning
  - ksl
  - casl
  - nsl
datasets:
  - mwakalucky/casl-w60
  - awsaf49/kslc-dataset
metrics:
  - accuracy
  - f1
---

# AfriSign Encoder Best Models

This repository contains the best focused AfriSign Encoder checkpoints for the
KSL + CASL + NSL contribution. The models are trained for African sign-language
recognition using shared multilingual representations across pose landmarks,
RGB video frames, and static sign images.

## What Is Included

| Folder | Model | Checkpoint | Result file |
|---|---|---|---|
| `models/e9_5_research_v2_pose` | AfriSign Encoder E9.5 Research-v2 Pose-only | `pytorch_model.pt` | `results.json` |
| `models/e9_6_research_v2_rgb_image` | AfriSign Encoder E9.6 Research-v2 RGB/image-only | `pytorch_model.pt` | `results.json` |
| `models/e9_7_research_v2_multimodal` | AfriSign Encoder E9.7 Research-v2 RGB+pose/image | `pytorch_model.pt` | `results.json` |

## Main Results

| Experiment | Model | Modality | Test macro-F1 | Note |
|---|---|---:|---:|---|
| E9.1 | naive pooled pose-only | pose | 32.00% | baseline |
| E9.2 | proposed pose-only | pose | 79.75% | language-aware model |
| E9.3 | proposed RGB/image-only | RGB/image | 57.18% | visual-only model |
| E9.4 | proposed RGB+pose/image | multimodal | 69.87% | first multimodal model |
| E9.5 | research-v2 pose-only | pose | 80.26% | best overall focused model |
| E9.6 | research-v2 RGB/image-only | RGB/image | 57.58% | best visual-only model |
| E9.7 | research-v2 RGB+pose/image | multimodal | 71.82% | best multimodal model |

## Methodology

The focused benchmark uses KSL, CASL, and NSL streams:

- CASL signer-independent word-level pose landmarks.
- KSL word-level pose landmarks.
- NSL image-level pose landmarks.
- CASL signer-independent RGB video frames.
- KSLC/KSL static RGB images.

The proposed encoder uses shared modality encoders with language/task
conditioning, task-specific heads, balanced sampling, supervised contrastive
learning, and metric-learning heads. The research-v2 models add motion-aware
pose features, multi-scale temporal modeling, ArcFace/CosFace-style heads, and
center-loss regularization.

## Loading A Checkpoint

```python
import torch

ckpt = torch.load("models/e9_5_research_v2_pose/pytorch_model.pt", map_location="cpu")
print(ckpt.keys())
```

The checkpoints store the model state dict, normalization statistics, task
metadata, language mappings, and training arguments. Use the code in
`code/experiments/` or the GitHub repository to rebuild the architecture before
loading the state dict.

## Project

GitHub: https://github.com/Yen-hub/AfriSignEncoder

Model repo: https://huggingface.co/luciayen/afrisign-encoder-best-models