PJM-to-text models (CrocoSign)
Models that translate Polish Sign Language (PJM) video into English text, trained on CrocoSign/pjm-data (29,928 video-sentence pairs, 151 Deaf signers). Three families: Uni-Sign (pose-based, mT5), SpaMo and our modified SpaMo (video features, Flan-T5-XL), each trained from scratch and from a checkpoint trained on another sign language. They are research prototypes, not a replacement for a sign language interpreter. Method and results: our paper, A Polish Sign Language to Text Dataset and Benchmark (in preparation); overview at crocosign.pl.
Use
The ONNX exports run with the signsdk package from github.com/crocosign/pjm-models, which downloads them from here:
git clone https://github.com/crocosign/pjm-models
cd pjm-models
uv sync
uv run python -c "import signsdk; print(signsdk.from_pretrained('unisign-pjm-openasl-ms').predict('clip.mp4').text)"
Uni-Sign runs on a CPU: in our measurements with the PyTorch models (Ryzen 9 9900X, RTX 3090), a 6.6-second clip took about 11 s on the CPU and 1.6 s on the GPU. SpaMo is much heavier (about 110 s and 13 s) and needs a 24 GB GPU. The Uni-Sign ONNX graphs give the same output as the PyTorch models (fp32, batch 1) on all 160 clips we compared.
Results
Test scores; every configuration was trained once (a second seed of the two best MS models moved BLEU-4 by 0.14); MS: test signers also appear in training, SI: test signers are unseen. BLEU-4 and BLEURT-20, scaled to 0-100. References are DeepL translations of the Polish transcripts.
| Model | Transfer source | MS BLEU-4 | MS BLEURT | SI BLEU-4 | SI BLEURT |
|---|---|---|---|---|---|
| Uni-Sign | OpenASL | 8.53 | 42.43 | 6.09 | 38.16 |
| Uni-Sign | How2Sign | 8.61 | 41.98 | 5.91 | 37.83 |
| Uni-Sign | CSL-News | 5.16 | 36.36 | 2.41 | 31.31 |
| Uni-Sign | scratch | 1.82 | 30.04 | 0.85 | 30.08 |
| SpaMo | PHOENIX-2014T | 5.54 | 35.12 | 2.09 | 28.84 |
| SpaMo | scratch | 2.03 | 30.25 | 0.93 | 27.97 |
| Mod-SpaMo | PHOENIX-2014T | 5.06 | 34.30 | 2.06 | 27.86 |
| Mod-SpaMo | scratch | 1.56 | 30.42 | 1.17 | 27.67 |
An ensemble of six Uni-Sign models reaches 10.36 BLEU-4 and 44.37 BLEURT on MS. The SpaMo SI rows from scratch were trained and tested on the SI split before the final quality filter, so the SI comparison with them is approximate.
Contents
| Path | What it holds |
|---|---|
unisign/unisign-pjm-{cslnews,how2sign,openasl,scratch}-{ms,si}/ |
ONNX export of each Uni-Sign row: projector.onnx, encoder.onnx, decoder.onnx, prefix.npy, meta.json |
spamo/{spamo,mod-spamo}-pjm-{phoenix,scratch}-{ms,si}/ |
ONNX export of the SpaMo and Mod-SpaMo rows, sharing spamo/shared/ (CLIP ViT, VideoMAE, hand ViT); the PHOENIX-2014T SI rows have no ONNX export, only checkpoints |
assets/ |
Tokenizers (mT5, Flan-T5), YOLO person detector and MediaPipe models used by signsdk |
checkpoints/unisign/<run>/best_checkpoint.pth |
PyTorch checkpoints: the eight table rows (pjm_phase3_full_{ms,si} from CSL-News, _how2sign, _openasl, _scratch), the second seeds (_s43), the chained and distilled models (chain_openasl_how2sign, _ms_chain, _ms_how2sign_kfold_mix) and the two models trained on Polish targets (pjm_pl_{ms,si}_how2sign), and the MS training-size control (pjm_ms_sub_openasl, see results/README.md) |
checkpoints/spamo/<model>/*.ckpt |
PyTorch Lightning checkpoints of the eight SpaMo and Mod-SpaMo rows; the BLEU-4 in a file name is the validation score at that epoch |
results/ |
Test outputs, references, metric logs and training logs of every row (see results/README.md) |
To train or evaluate the checkpoints, use Uni-Sign-PJM and SpaMo-PJM; their READMEs map each row to its config and command.
Polish output
All models above output English. The two pjm_pl_*_how2sign checkpoints are Uni-Sign trained on the corrected Polish transcripts instead: 5.99 BLEU-4 on MS and 3.85 on SI against the Polish references, about the same as translating the English How2Sign outputs to Polish with a language model (5.67 and 3.86), while BLEURT prefers the translated outputs.
Licence
CC BY-NC-SA 4.0. The weights build on the PJM Corpus (CC BY-SA 4.0) and on checkpoints released for non-commercial use (PHOENIX-2014T for SpaMo, the Uni-Sign checkpoints), so they may not be used commercially. When you use them, cite the corpus and our work; BibTeX for both is in the pjm-data card.