| # PJM Sign-Language → English — ONNX models |
|
|
| Repo-free ONNX exports of two PJM (Polish Sign Language) → English translation model families, |
| served by the `signsdk` runtime (onnxruntime + bundled tokenizers/assets — no model-repo needed). |
| All graphs **fp32**. Each model = `projector.onnx` + `encoder.onnx` + `decoder.onnx` (+ `.onnx.data` |
| sidecar for SpaMo's flan-t5-xl) + `meta.json` + prompt/prefix embedding. |
|
|
| - **Uni-Sign** — pose-based (RTMPose keypoints) → mT5. Self-contained graphs. |
| - **SpaMo / Mod-SpaMo** — CLIP ViT-L/14 + VideoMAE-large (+ hand-CLIP + MediaPipe for `mod-`) → flan-t5-xl. |
| Shares `spamo/shared/` (CLIP+VideoMAE+hand-CLIP) across all SpaMo checkpoints. |
|
|
| ## Layout |
| ``` |
| assets/ mt5_tokenizer/ flan_t5_tokenizer/ yolo/yolo11n.onnx mediapipe/*.task |
| unisign/<name>/ projector.onnx encoder.onnx decoder.onnx prefix.npy meta.json |
| spamo/shared/ clip_vit_s2.onnx(+.data) videomae.onnx(+.data) hand_clip.onnx(+.data) |
| spamo/<name>/ projector.onnx(+.data) encoder.onnx(+.data) decoder.onnx(+.data) prompt_embeds.npy meta.json |
| ``` |
|
|
| ## Models (test-set BLEU-4, ✓ = uploaded) |
|
|
| | model | family | source pretrain | split | BLEU-4 | status | |
| |-------|--------|-----------------|-------|--------|--------| |
| | `unisign-pjm-cslnews-ms` | Uni-Sign | CSL-News | ms | 2.42 | ✓ | |
| | `unisign-pjm-cslnews-si` | Uni-Sign | CSL-News | si | 1.43 | ✓ | |
| | `unisign-pjm-how2sign-ms` | Uni-Sign | How2Sign | ms | 7.61 | ✓ | |
| | `unisign-pjm-how2sign-si` | Uni-Sign | How2Sign | si | 4.77 | — | |
| | `unisign-pjm-openasl-ms` | Uni-Sign | OpenASL | ms | 7.38 | — | |
| | `unisign-pjm-openasl-si` | Uni-Sign | OpenASL | si | 4.79 | — | |
| | `unisign-pjm-scratch-ms` | Uni-Sign | scratch | ms | 1.62 | — | |
| | `unisign-pjm-scratch-si` | Uni-Sign | scratch | si | 0.72 | — | |
| | `spamo-pjm-phoenix-ms` | SpaMo | PHOENIX | ms | 5.54 | ✓ | |
| | `spamo-pjm-phoenix-si` | SpaMo | PHOENIX | si | 0.53 | — | |
| | `spamo-pjm-scratch-ms` | SpaMo | scratch | ms | 2.03 | — | |
| | `spamo-pjm-scratch-si` | SpaMo | scratch | si | 0.93 | — | |
| | `mod-spamo-pjm-phoenix-ms` | Mod-SpaMo | PHOENIX | ms | 5.06 | ✓ | |
| | `mod-spamo-pjm-phoenix-si` | Mod-SpaMo | PHOENIX | si | 0.47 | — | |
| | `mod-spamo-pjm-scratch-ms` | Mod-SpaMo | scratch | ms | 1.56 | — | |
| | `mod-spamo-pjm-scratch-si` | Mod-SpaMo | scratch | si | 1.17 | — | |
|
|
| BLEU-4 = reported PJM test score. Decoding faithful to the original eval |
| (Uni-Sign beam=4; SpaMo beam-sample num_beams=5, top_p=0.9). fp32 chosen for deploy |
| (fp16 overflows the bf16-trained flan-t5-xl; int8 is size-only). |
|
|
