File size: 7,764 Bytes
3d02762 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 | # Experiment 9: KSL + CASL + NSL Focused Unified Encoder
## Purpose
Experiment 9 is the focused paper track after the broad unified run. It keeps
the most reliable African sign-language streams for a clean AAAI-style result:
- KSL word-level sign recognition
- CASL word-level sign recognition
- NSL image-level sign recognition
The goal is to answer:
> Can one shared multilingual encoder outperform naive pooled training across
> African sign-language datasets?
The current E9 results should be treated as the focused baseline. The new E9-v2
jobs add stronger metric-learning components to try to improve it.
## Dataset Streams
| Stream | Language | Input | Level | Classes | Split note |
|---|---|---|---|---:|---|
| CASL-W60 signer-independent | CASL | pose landmarks | word | 60 | signer-independent split |
| KSL word video landmarks | KSL | pose landmarks | word | 4 | existing KSL split |
| NSL image landmarks | NSL | pose landmarks | image | 139 | existing NSL split |
| CASL-W60 frames | CASL | RGB frames | word | 60 | signer-independent frame manifest |
| KSLC fixed image split | KSL | RGB images | image | 9 | labeled train.csv split into train/val/test |
## Why These Three Languages
The broader Exp8 run included GSL/GhSL and other streams, but some of those
streams are train-only, sentence-level with thousands of labels, or not yet
paired cleanly with RGB and pose. For a fair core table, E9 focuses on streams
with usable evaluation splits and stable task definitions.
## Models Compared
| ID | Model | Meaning |
|---|---|---|
| E9.1 | Pooled pose-only | One shared pose encoder trained naively on KSL + CASL + NSL |
| E9.2 | Proposed pose-only | Shared pose encoder with language/task conditioning, adapters, and supervised contrastive loss |
| E9.3 | Proposed RGB/image-only | Same proposed encoder, but only RGB/image streams |
| E9.4 | Proposed RGB+pose/image | Unified multimodal model using both pose streams and RGB/image streams |
## Main Test Results
| Experiment | Model Setting | Test Top-1 | Test Macro-F1 | Runs | Main Meaning |
|---|---|---:|---:|---:|---|
| E9.1 | Pooled pose-only | 37.15% | 32.00% | 3 | Naive shared encoder baseline |
| E9.2 | Proposed pose-only | **80.95%** | **79.75%** | 3 | Best focused multilingual result so far |
| E9.3 | Proposed RGB/image-only | 58.01% | 57.18% | 3 | RGB/image streams alone |
| E9.4 | Proposed RGB+pose/image | 70.84% | 69.87% | 3 | Unified multimodal model |
## Per-Task Test Results
| Experiment | Task | Modality | Classes | Test Top-1 | Test Macro-F1 | Runs |
|---|---|---|---:|---:|---:|---:|
| E9.1 pooled pose-only | CASL word | pose | 60 | 63.59% | 62.88% | 3 |
| E9.1 pooled pose-only | KSL word | pose | 4 | 40.32% | 27.80% | 3 |
| E9.1 pooled pose-only | NSL image | pose | 139 | 7.55% | 5.33% | 3 |
| E9.2 proposed pose-only | CASL word | pose | 60 | 88.88% | 88.69% | 3 |
| E9.2 proposed pose-only | KSL word | pose | 4 | **100.00%** | **100.00%** | 3 |
| E9.2 proposed pose-only | NSL image | pose | 139 | 53.97% | 50.55% | 3 |
| E9.3 proposed RGB/image-only | CASL word | RGB frames | 60 | 34.50% | 32.73% | 3 |
| E9.3 proposed RGB/image-only | KSLC image | RGB image | 9 | 81.52% | 81.64% | 3 |
| E9.4 proposed RGB+pose/image | CASL word | pose | 60 | 88.44% | 88.18% | 3 |
| E9.4 proposed RGB+pose/image | KSL word | pose | 4 | **100.00%** | **100.00%** | 3 |
| E9.4 proposed RGB+pose/image | NSL image | pose | 139 | 52.80% | 49.36% | 3 |
| E9.4 proposed RGB+pose/image | CASL word | RGB frames | 60 | 28.83% | 27.59% | 3 |
| E9.4 proposed RGB+pose/image | KSLC image | RGB image | 9 | **84.13%** | **84.21%** | 3 |
## Interpretation
The most important result is that E9.2 strongly beats E9.1:
- Naive pooled pose Macro-F1: 32.00%
- Proposed pose-only Macro-F1: 79.75%
- Improvement: +47.75 Macro-F1 points
This means simple pooling is not enough. The model needs task/language
conditioning to avoid negative transfer between datasets.
The RGB/image result is mixed:
- KSLC image recognition is strong.
- CASL RGB frame recognition is weak compared with CASL pose.
- The full multimodal model improves KSLC image performance, but it does not
beat pose-only overall because the CASL RGB stream pulls down the macro score.
Therefore, the current strongest claim is:
> A language-aware shared pose encoder is much stronger than naive pooled
> multilingual pose training for KSL, CASL, and NSL.
The current cautious claim is:
> RGB/image data contributes useful information for KSLC, but the CASL RGB
> frame stream still needs stronger video modeling before RGB+pose becomes the
> best overall model.
## Research-Informed Next Step: E9-v2
The new E9-v2 code treats E9.1-E9.4 as baselines and tests stronger
metric-learning training:
| ID | Model | New additions |
|---|---|---|
| E9.5 | Research-v2 pose-only | motion-aware pose stem + ArcFace heads + center loss + higher SupCon |
| E9.6 | Research-v2 RGB/image-only | ArcFace heads + center loss on RGB/image streams |
| E9.7 | Research-v2 RGB+pose/image | motion-aware pose stem plus the same metric heads/losses in the unified multimodal model |
The motivation comes from:
- supervised contrastive learning for class clustering
- ArcFace/CosFace-style angular margins for low-sample/many-class recognition
- masked/pose-transformer sign recognition work such as SignBERT
- multimodal SLR work using RGB, skeleton/pose, and temporal modeling
## New Code
| File | Purpose |
|---|---|
| `experiments/exp9b_kcn_research_unified_encoder.py` | Research-v2 model wrapper with motion-aware pose stem, ArcFace/CosFace heads, and center loss |
| `psc_jobs/92_kcn_v2_research_array.sbatch` | Slurm sweep for E9.5-E9.7 across 3 seeds |
| `scripts/aggregate_kcn_focus_results.py` | Updated to include E9.5-E9.7 in the same summary |
## Run Commands
Check the scaffold:
```bash
.venv/bin/python scripts/check_experiment_scaffold.py --bash
```
Dry-run one v2 model:
```bash
.venv/bin/python experiments/exp9b_kcn_research_unified_encoder.py \
--focus-mode pose \
--run-name kcn_v2_pose_arcface_center \
--metric-head arcface \
--center-weight 0.015 \
--pose-languages casl_si ksl nsi \
--no-gsl-sentence-landmarks \
--dry-run
```
Submit the full E9-v2 sweep:
```bash
sbatch --parsable psc_jobs/92_kcn_v2_research_array.sbatch
```
Aggregate after completion:
```bash
.venv/bin/python scripts/aggregate_kcn_focus_results.py \
--results-root results \
--out-dir results/exp9_kcn_focus_summary
```
## Expected Improvement Target
The realistic target is not just higher overall macro-F1. We specifically want:
1. E9.5 pose-only to beat E9.2 pose-only.
2. E9.6 to improve CASL RGB frame Macro-F1.
3. E9.7 to beat E9.4 and get closer to E9.2 while preserving the KSLC RGB gain.
4. NSL Macro-F1 to improve beyond 50.55% without relying on KSL's saturated
four-class score.
## References For The Next Iteration
- [Supervised Contrastive Learning, NeurIPS 2020](https://arxiv.org/abs/2004.11362)
- [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](https://arxiv.org/abs/1801.07698)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://arxiv.org/abs/1801.09414)
- [SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition](https://arxiv.org/abs/2111.03607)
- [Multiscale Vision Transformers](https://arxiv.org/abs/2104.11227)
- [Multiscale Vision Transformers V2](https://arxiv.org/abs/2112.01526)
- [Skeleton-Based Action Recognition with Spatial Temporal Graph Convolutional Networks](https://arxiv.org/abs/1801.07455)
|