Blame for README.md

---
license: cc-by-nc-sa-4.0
language:
- en
- pl
pipeline_tag: translation
tags:
- sign-language
- polish-sign-language
- pjm
- sign-language-translation
- onnx
datasets:
- CrocoSign/pjm-data
---

# PJM-to-text models (CrocoSign)

Models that translate Polish Sign Language (PJM) video into English text, trained on
[CrocoSign/pjm-data](https://huggingface.co/datasets/CrocoSign/pjm-data) (29,928 video-sentence pairs, 151 Deaf
signers). Three families: Uni-Sign (pose-based, mT5), SpaMo and our modified SpaMo (video features, Flan-T5-XL),
each trained from scratch and from a checkpoint trained on another sign language. They are research prototypes,
not a replacement for a sign language interpreter. Method and results: our paper, *A Polish Sign Language to
Text Dataset and Benchmark* (in preparation); overview at [crocosign.pl](https://crocosign.pl).

## Use

The ONNX exports run with the `signsdk` package from [github.com/crocosign/pjm-models](https://github.com/crocosign/pjm-models), which downloads them from here:

```bash
git clone https://github.com/crocosign/pjm-models
cd pjm-models
uv sync
uv run python -c "import signsdk; print(signsdk.from_pretrained('unisign-pjm-openasl-ms').predict('clip.mp4').text)"
```

Uni-Sign runs on a CPU: in our measurements with the PyTorch models (Ryzen 9 9900X, RTX 3090), a 6.6-second clip took about 11 s on the CPU and 1.6 s on the GPU. SpaMo is much heavier (about 110 s and 13 s) and needs a 24 GB GPU. The Uni-Sign ONNX graphs give the same output as the PyTorch models (fp32, batch 1) on all 160 clips we compared.

## Results

Test scores; every configuration was trained once (a second seed of the two best MS models moved BLEU-4 by 0.14); MS: test signers also appear in training, SI: test signers are unseen. BLEU-4 and BLEURT-20, scaled to 0-100. References are DeepL translations of the Polish transcripts.

| Model | Transfer source | MS BLEU-4 | MS BLEURT | SI BLEU-4 | SI BLEURT |
|---|---|---|---|---|---|
| Uni-Sign | OpenASL | 8.53 | **42.43** | **6.09** | **38.16** |
| Uni-Sign | How2Sign | **8.61** | 41.98 | 5.91 | 37.83 |
| Uni-Sign | CSL-News | 5.16 | 36.36 | 2.41 | 31.31 |
| Uni-Sign | scratch | 1.82 | 30.04 | 0.85 | 30.08 |
| SpaMo | PHOENIX-2014T | 5.54 | 35.12 | 2.09 | 28.84 |
| SpaMo | scratch | 2.03 | 30.25 | 0.93 | 27.97 |
| Mod-SpaMo | PHOENIX-2014T | 5.06 | 34.30 | 2.06 | 27.86 |
| Mod-SpaMo | scratch | 1.56 | 30.42 | 1.17 | 27.67 |

An ensemble of six Uni-Sign models reaches 10.36 BLEU-4 and 44.37 BLEURT on MS. The SpaMo SI rows from scratch were trained and tested on the SI split before the final quality filter, so the SI comparison with them is approximate.

## Contents

| Path | What it holds |
|---|---|
| `unisign/unisign-pjm-{cslnews,how2sign,openasl,scratch}-{ms,si}/` | ONNX export of each Uni-Sign row: `projector.onnx`, `encoder.onnx`, `decoder.onnx`, `prefix.npy`, `meta.json` |
| `spamo/{spamo,mod-spamo}-pjm-{phoenix,scratch}-{ms,si}/` | ONNX export of the SpaMo and Mod-SpaMo rows, sharing `spamo/shared/` (CLIP ViT, VideoMAE, hand ViT); the PHOENIX-2014T SI rows have no ONNX export, only checkpoints |
| `assets/` | Tokenizers (mT5, Flan-T5), YOLO person detector and MediaPipe models used by `signsdk` |
| `checkpoints/unisign/<run>/best_checkpoint.pth` | PyTorch checkpoints: the eight table rows (`pjm_phase3_full_{ms,si}` from CSL-News, `_how2sign`, `_openasl`, `_scratch`), the second seeds (`_s43`), the chained and distilled models (`chain_openasl_how2sign`, `_ms_chain`, `_ms_how2sign_kfold_mix`) and the two models trained on Polish targets (`pjm_pl_{ms,si}_how2sign`), and the MS training-size control (`pjm_ms_sub_openasl`, see `results/README.md`) |
| `checkpoints/spamo/<model>/*.ckpt` | PyTorch Lightning checkpoints of the eight SpaMo and Mod-SpaMo rows; the BLEU-4 in a file name is the validation score at that epoch |
| `results/` | Test outputs, references, metric logs and training logs of every row (see `results/README.md`) |

To train or evaluate the checkpoints, use [Uni-Sign-PJM](https://github.com/crocosign/Uni-Sign-PJM) and [SpaMo-PJM](https://github.com/crocosign/SpaMo-PJM); their READMEs map each row to its config and command.

## Polish output

All models above output English. The two `pjm_pl_*_how2sign` checkpoints are Uni-Sign trained on the corrected Polish transcripts instead: 5.99 BLEU-4 on MS and 3.85 on SI against the Polish references, about the same as translating the English How2Sign outputs to Polish with a language model (5.67 and 3.86), while BLEURT prefers the translated outputs.

## Licence

CC BY-NC-SA 4.0. The weights build on the PJM Corpus (CC BY-SA 4.0) and on checkpoints released for non-commercial use (PHOENIX-2014T for SpaMo, the Uni-Sign checkpoints), so they may not be used commercially. When you use them, cite the corpus and our work; BibTeX for both is in the [pjm-data card](https://huggingface.co/datasets/CrocoSign/pjm-data).