VLTK model weights

Model checkpoints and assets for the Sign-Language-Toolkit (SLTK). SLTK downloads what it needs from this repo on first use (sltk.weights.hub, HF repo id cogvis-cvssp/vltk).

Layout

Task checkpoints live in per-model directories. All parametric human models (SMPL-X, MANO, FLAME) and the assets shared between them live under body_models/.

body_models/
  smplx/    SMPL-X body model            (license-gated — user-provided)
  mano/     MANO hand models + extras    (models license-gated)
  flame/    FLAME face model + assets    (model + masks license-gated)
  shared/   cross-model correspondence   (one copy, downloadable)
nlf/        NLF pose model + YOLO detector
wilor/      WiLoR hand model + detector + config
teaser/     TEASER FLAME face encoder
pear/       PEAR expressive-mesh regression checkpoint
segmenter/  sign-segmentation model
signrep/    SignRep video-embedding model
ipn/        Index Proposal Network: index vs lexical sign classifier (ipn-b BSL, ipn-m DGS)
signseek/   SignSeek pose encoder: RTMPose 133-keypoint sign clip → 128-d embedding, for dictionary retrieval
fingerspelling/bsl/  BSL fingerspelling detector + recognizer (over WiLoR hand features)
fingerspelling/asl/  ASL fingerspelling recognizer (+ experimental detector)
signmatch/  SignMatch dictionary-to-video matching weights (non-commercial licence)
fingerspelling/rgb-dinov2-128f/    fingerspelling recogniser from RGB frames (DINOv2 + CTC)
fingerspelling/flan-t5-corrector/  optional spelling corrector for the RGB recogniser's letters (Flan-T5)

body_models/shared/ holds the cross-model correspondence files (SMPL-X__FLAME_vertex_ids.npy, MANO_SMPLX_vertex_ids.pkl, SMPLX_to_J14.pkl, mediapipe_landmark_embedding.npz, smplx_vert_segmentation.json). EHM's SMPL-X, FLAME and TEASER model code all read them from here (previously each directory kept its own duplicate copy).

License-gated files (not included)

The parametric models require registration on the authors' sites and cannot be redistributed. Place them at the paths below (or point SLTK at them with the listed environment variables):

Model	Files	Location	Register at	Env var
SMPL-X	SMPLX_NEUTRAL.npz	body_models/smplx/	smpl-x.is.tue.mpg.de	SLTK_SMPLX_MODEL
MANO	MANO_LEFT.pkl, MANO_RIGHT.pkl	body_models/mano/	mano.is.tue.mpg.de	SLTK_MANO_DIR
FLAME	generic_model.pkl, FLAME_masks.pkl	body_models/flame/	flame.is.tue.mpg.de	SLTK_FLAME_ASSETS_DIR

These paths are listed in .gitignore so they are never committed.

SignSeek

signseek/signseek-agcn-rtm/ (config.json + model.safetensors) is the released SignSeek encoder by Sobhan Asasi, converted for SLTK. It embeds a sign clip of 2D RTMPose COCO-WholeBody (133-keypoint) poses into a 128-d vector for dictionary retrieval.

• Source: Hugging Face elsobhano/test_model, file signseek_agcn_rtm.pth, revision 5615ec7e41b6e751635a989af8e317c5039b1169 (sha256 1ca206664c0aec3e128082396f4972f6dda70a9acbff33372510aaff5f29be4b, also stored as source_sha256 in config.json).
• Conversion: SLTK scripts/convert_signseek_checkpoint.py. The tensors are unchanged and still bf16 (SLTK casts them to fp32 on load). Only the training-only art_mask_tokens.* and _smplx_* entries are dropped, and the training config is not included.
• Trained on MeinDGS, SemLex, AUTSL, SLR500 and MSASL.
• Override with SLTK_SIGNSEEK_DIR=/path/to/signseek-agcn-rtm.

Fingerspelling (RGB)

fingerspelling/rgb-dinov2-128f/ (config.json + model.safetensors) is the RGB fingerspelling recogniser, converted for SLTK. It reads the letters fingerspelled in a signer-centred clip of up to 128 frames (DINOv2-B features per frame, a temporal transformer and a CTC head).

• Source: the original author's finger_spelling.pth (the 128-frame model) (sha256 c2da2c530b111d1876c19685fddc5a185763668d1abe1c2feddc9069bed842ff, also stored as source_sha256 in config.json).
• Conversion: SLTK scripts/convert_fingerspelling_checkpoint.py. fp32. The duplicate shared_backbone.* tensors are dropped, and the 224-pixel DINOv2 position table is added (newer transformers interpolate it differently).
• Trained on ChicagoFSWild+ and BOBSL.

fingerspelling/flan-t5-corrector/ is the author's fine-tuned google/flan-t5-base, unchanged, plus SLTK's sltk_corrector.json (prompt, generation settings and file checksums). It maps the recogniser's letters to a word (sltk fingerspell --correct); it is not reliable on BSL.

• Override with SLTK_FINGERSPELLING_DIR=/path/to/folder, holding rgb-dinov2-128f/, flan-t5-corrector/, asl/ and bsl/. When it is set, SLTK looks only there.

SignMatch

signmatch/signmatch_rgb_v1.pt is the released SignMatch RGB checkpoint, unchanged.

• Source: https://github.com/ryanwongsa/signmatch/releases/tag/v1.0.0 (250,605,149 bytes, sha256 1f7566411c20f655dc8d8b84b9634c134b57bb790613dac10e4dcd9073759ab3).
• Licence: the SignMatch Non-Commercial License, copied to signmatch/LICENSE. Non-commercial use only; keep that notice with any copy.
• Override with SLTK_SIGNMATCH_CHECKPOINT=/path/to/signmatch_rgb_v1.pt.

License

Everything in this repository is released under CC-BY-NC-4.0, except the license-gated files above and signmatch/ (SignMatch Non-Commercial License), which are governed by their respective authors' licenses.