Experiment 9: KSL + CASL + NSL Focused Unified Encoder
Purpose
Experiment 9 is the focused paper track after the broad unified run. It keeps the most reliable African sign-language streams for a clean AAAI-style result:
- KSL word-level sign recognition
- CASL word-level sign recognition
- NSL image-level sign recognition
The goal is to answer:
Can one shared multilingual encoder outperform naive pooled training across African sign-language datasets?
The current E9 results should be treated as the focused baseline. The new E9-v2 jobs add stronger metric-learning components to try to improve it.
Dataset Streams
| Stream | Language | Input | Level | Classes | Split note |
|---|---|---|---|---|---|
| CASL-W60 signer-independent | CASL | pose landmarks | word | 60 | signer-independent split |
| KSL word video landmarks | KSL | pose landmarks | word | 4 | existing KSL split |
| NSL image landmarks | NSL | pose landmarks | image | 139 | existing NSL split |
| CASL-W60 frames | CASL | RGB frames | word | 60 | signer-independent frame manifest |
| KSLC fixed image split | KSL | RGB images | image | 9 | labeled train.csv split into train/val/test |
Why These Three Languages
The broader Exp8 run included GSL/GhSL and other streams, but some of those streams are train-only, sentence-level with thousands of labels, or not yet paired cleanly with RGB and pose. For a fair core table, E9 focuses on streams with usable evaluation splits and stable task definitions.
Models Compared
| ID | Model | Meaning |
|---|---|---|
| E9.1 | Pooled pose-only | One shared pose encoder trained naively on KSL + CASL + NSL |
| E9.2 | Proposed pose-only | Shared pose encoder with language/task conditioning, adapters, and supervised contrastive loss |
| E9.3 | Proposed RGB/image-only | Same proposed encoder, but only RGB/image streams |
| E9.4 | Proposed RGB+pose/image | Unified multimodal model using both pose streams and RGB/image streams |
Main Test Results
| Experiment | Model Setting | Test Top-1 | Test Macro-F1 | Runs | Main Meaning |
|---|---|---|---|---|---|
| E9.1 | Pooled pose-only | 37.15% | 32.00% | 3 | Naive shared encoder baseline |
| E9.2 | Proposed pose-only | 80.95% | 79.75% | 3 | Best focused multilingual result so far |
| E9.3 | Proposed RGB/image-only | 58.01% | 57.18% | 3 | RGB/image streams alone |
| E9.4 | Proposed RGB+pose/image | 70.84% | 69.87% | 3 | Unified multimodal model |
Per-Task Test Results
| Experiment | Task | Modality | Classes | Test Top-1 | Test Macro-F1 | Runs |
|---|---|---|---|---|---|---|
| E9.1 pooled pose-only | CASL word | pose | 60 | 63.59% | 62.88% | 3 |
| E9.1 pooled pose-only | KSL word | pose | 4 | 40.32% | 27.80% | 3 |
| E9.1 pooled pose-only | NSL image | pose | 139 | 7.55% | 5.33% | 3 |
| E9.2 proposed pose-only | CASL word | pose | 60 | 88.88% | 88.69% | 3 |
| E9.2 proposed pose-only | KSL word | pose | 4 | 100.00% | 100.00% | 3 |
| E9.2 proposed pose-only | NSL image | pose | 139 | 53.97% | 50.55% | 3 |
| E9.3 proposed RGB/image-only | CASL word | RGB frames | 60 | 34.50% | 32.73% | 3 |
| E9.3 proposed RGB/image-only | KSLC image | RGB image | 9 | 81.52% | 81.64% | 3 |
| E9.4 proposed RGB+pose/image | CASL word | pose | 60 | 88.44% | 88.18% | 3 |
| E9.4 proposed RGB+pose/image | KSL word | pose | 4 | 100.00% | 100.00% | 3 |
| E9.4 proposed RGB+pose/image | NSL image | pose | 139 | 52.80% | 49.36% | 3 |
| E9.4 proposed RGB+pose/image | CASL word | RGB frames | 60 | 28.83% | 27.59% | 3 |
| E9.4 proposed RGB+pose/image | KSLC image | RGB image | 9 | 84.13% | 84.21% | 3 |
Interpretation
The most important result is that E9.2 strongly beats E9.1:
- Naive pooled pose Macro-F1: 32.00%
- Proposed pose-only Macro-F1: 79.75%
- Improvement: +47.75 Macro-F1 points
This means simple pooling is not enough. The model needs task/language conditioning to avoid negative transfer between datasets.
The RGB/image result is mixed:
- KSLC image recognition is strong.
- CASL RGB frame recognition is weak compared with CASL pose.
- The full multimodal model improves KSLC image performance, but it does not beat pose-only overall because the CASL RGB stream pulls down the macro score.
Therefore, the current strongest claim is:
A language-aware shared pose encoder is much stronger than naive pooled multilingual pose training for KSL, CASL, and NSL.
The current cautious claim is:
RGB/image data contributes useful information for KSLC, but the CASL RGB frame stream still needs stronger video modeling before RGB+pose becomes the best overall model.
Research-Informed Next Step: E9-v2
The new E9-v2 code treats E9.1-E9.4 as baselines and tests stronger metric-learning training:
| ID | Model | New additions |
|---|---|---|
| E9.5 | Research-v2 pose-only | motion-aware pose stem + ArcFace heads + center loss + higher SupCon |
| E9.6 | Research-v2 RGB/image-only | ArcFace heads + center loss on RGB/image streams |
| E9.7 | Research-v2 RGB+pose/image | motion-aware pose stem plus the same metric heads/losses in the unified multimodal model |
The motivation comes from:
- supervised contrastive learning for class clustering
- ArcFace/CosFace-style angular margins for low-sample/many-class recognition
- masked/pose-transformer sign recognition work such as SignBERT
- multimodal SLR work using RGB, skeleton/pose, and temporal modeling
New Code
| File | Purpose |
|---|---|
experiments/exp9b_kcn_research_unified_encoder.py |
Research-v2 model wrapper with motion-aware pose stem, ArcFace/CosFace heads, and center loss |
psc_jobs/92_kcn_v2_research_array.sbatch |
Slurm sweep for E9.5-E9.7 across 3 seeds |
scripts/aggregate_kcn_focus_results.py |
Updated to include E9.5-E9.7 in the same summary |
Run Commands
Check the scaffold:
.venv/bin/python scripts/check_experiment_scaffold.py --bash
Dry-run one v2 model:
.venv/bin/python experiments/exp9b_kcn_research_unified_encoder.py \
--focus-mode pose \
--run-name kcn_v2_pose_arcface_center \
--metric-head arcface \
--center-weight 0.015 \
--pose-languages casl_si ksl nsi \
--no-gsl-sentence-landmarks \
--dry-run
Submit the full E9-v2 sweep:
sbatch --parsable psc_jobs/92_kcn_v2_research_array.sbatch
Aggregate after completion:
.venv/bin/python scripts/aggregate_kcn_focus_results.py \
--results-root results \
--out-dir results/exp9_kcn_focus_summary
Expected Improvement Target
The realistic target is not just higher overall macro-F1. We specifically want:
- E9.5 pose-only to beat E9.2 pose-only.
- E9.6 to improve CASL RGB frame Macro-F1.
- E9.7 to beat E9.4 and get closer to E9.2 while preserving the KSLC RGB gain.
- NSL Macro-F1 to improve beyond 50.55% without relying on KSL's saturated four-class score.
References For The Next Iteration
- Supervised Contrastive Learning, NeurIPS 2020
- ArcFace: Additive Angular Margin Loss for Deep Face Recognition
- CosFace: Large Margin Cosine Loss for Deep Face Recognition
- SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition
- Multiscale Vision Transformers
- Multiscale Vision Transformers V2
- Skeleton-Based Action Recognition with Spatial Temporal Graph Convolutional Networks