code/docs/EXP9_KCN_FOCUSED_RESULTS_README.md

# Experiment 9: KSL + CASL + NSL Focused Unified Encoder

## Purpose

Experiment 9 is the focused paper track after the broad unified run. It keeps
the most reliable African sign-language streams for a clean AAAI-style result:

- KSL word-level sign recognition
- CASL word-level sign recognition
- NSL image-level sign recognition

The goal is to answer:

> Can one shared multilingual encoder outperform naive pooled training across
> African sign-language datasets?

The current E9 results should be treated as the focused baseline. The new E9-v2
jobs add stronger metric-learning components to try to improve it.

## Dataset Streams

| Stream | Language | Input | Level | Classes | Split note |
|---|---|---|---|---:|---|
| CASL-W60 signer-independent | CASL | pose landmarks | word | 60 | signer-independent split |
| KSL word video landmarks | KSL | pose landmarks | word | 4 | existing KSL split |
| NSL image landmarks | NSL | pose landmarks | image | 139 | existing NSL split |
| CASL-W60 frames | CASL | RGB frames | word | 60 | signer-independent frame manifest |
| KSLC fixed image split | KSL | RGB images | image | 9 | labeled train.csv split into train/val/test |

## Why These Three Languages

The broader Exp8 run included GSL/GhSL and other streams, but some of those
streams are train-only, sentence-level with thousands of labels, or not yet
paired cleanly with RGB and pose. For a fair core table, E9 focuses on streams
with usable evaluation splits and stable task definitions.

## Models Compared

| ID | Model | Meaning |
|---|---|---|
| E9.1 | Pooled pose-only | One shared pose encoder trained naively on KSL + CASL + NSL |
| E9.2 | Proposed pose-only | Shared pose encoder with language/task conditioning, adapters, and supervised contrastive loss |
| E9.3 | Proposed RGB/image-only | Same proposed encoder, but only RGB/image streams |
| E9.4 | Proposed RGB+pose/image | Unified multimodal model using both pose streams and RGB/image streams |

## Main Test Results

| Experiment | Model Setting | Test Top-1 | Test Macro-F1 | Runs | Main Meaning |
|---|---|---:|---:|---:|---|
| E9.1 | Pooled pose-only | 37.15% | 32.00% | 3 | Naive shared encoder baseline |
| E9.2 | Proposed pose-only | **80.95%** | **79.75%** | 3 | Best focused multilingual result so far |
| E9.3 | Proposed RGB/image-only | 58.01% | 57.18% | 3 | RGB/image streams alone |
| E9.4 | Proposed RGB+pose/image | 70.84% | 69.87% | 3 | Unified multimodal model |

## Per-Task Test Results

| Experiment | Task | Modality | Classes | Test Top-1 | Test Macro-F1 | Runs |
|---|---|---|---:|---:|---:|---:|
| E9.1 pooled pose-only | CASL word | pose | 60 | 63.59% | 62.88% | 3 |
| E9.1 pooled pose-only | KSL word | pose | 4 | 40.32% | 27.80% | 3 |
| E9.1 pooled pose-only | NSL image | pose | 139 | 7.55% | 5.33% | 3 |
| E9.2 proposed pose-only | CASL word | pose | 60 | 88.88% | 88.69% | 3 |
| E9.2 proposed pose-only | KSL word | pose | 4 | **100.00%** | **100.00%** | 3 |
| E9.2 proposed pose-only | NSL image | pose | 139 | 53.97% | 50.55% | 3 |
| E9.3 proposed RGB/image-only | CASL word | RGB frames | 60 | 34.50% | 32.73% | 3 |
| E9.3 proposed RGB/image-only | KSLC image | RGB image | 9 | 81.52% | 81.64% | 3 |
| E9.4 proposed RGB+pose/image | CASL word | pose | 60 | 88.44% | 88.18% | 3 |
| E9.4 proposed RGB+pose/image | KSL word | pose | 4 | **100.00%** | **100.00%** | 3 |
| E9.4 proposed RGB+pose/image | NSL image | pose | 139 | 52.80% | 49.36% | 3 |
| E9.4 proposed RGB+pose/image | CASL word | RGB frames | 60 | 28.83% | 27.59% | 3 |
| E9.4 proposed RGB+pose/image | KSLC image | RGB image | 9 | **84.13%** | **84.21%** | 3 |

## Interpretation

The most important result is that E9.2 strongly beats E9.1:

- Naive pooled pose Macro-F1: 32.00%
- Proposed pose-only Macro-F1: 79.75%
- Improvement: +47.75 Macro-F1 points

This means simple pooling is not enough. The model needs task/language
conditioning to avoid negative transfer between datasets.

The RGB/image result is mixed:

- KSLC image recognition is strong.
- CASL RGB frame recognition is weak compared with CASL pose.
- The full multimodal model improves KSLC image performance, but it does not
  beat pose-only overall because the CASL RGB stream pulls down the macro score.

Therefore, the current strongest claim is:

> A language-aware shared pose encoder is much stronger than naive pooled
> multilingual pose training for KSL, CASL, and NSL.

The current cautious claim is:

> RGB/image data contributes useful information for KSLC, but the CASL RGB
> frame stream still needs stronger video modeling before RGB+pose becomes the
> best overall model.

## Research-Informed Next Step: E9-v2

The new E9-v2 code treats E9.1-E9.4 as baselines and tests stronger
metric-learning training:

| ID | Model | New additions |
|---|---|---|
| E9.5 | Research-v2 pose-only | motion-aware pose stem + ArcFace heads + center loss + higher SupCon |
| E9.6 | Research-v2 RGB/image-only | ArcFace heads + center loss on RGB/image streams |
| E9.7 | Research-v2 RGB+pose/image | motion-aware pose stem plus the same metric heads/losses in the unified multimodal model |

The motivation comes from:

- supervised contrastive learning for class clustering
- ArcFace/CosFace-style angular margins for low-sample/many-class recognition
- masked/pose-transformer sign recognition work such as SignBERT
- multimodal SLR work using RGB, skeleton/pose, and temporal modeling

## New Code

| File | Purpose |
|---|---|
| `experiments/exp9b_kcn_research_unified_encoder.py` | Research-v2 model wrapper with motion-aware pose stem, ArcFace/CosFace heads, and center loss |
| `psc_jobs/92_kcn_v2_research_array.sbatch` | Slurm sweep for E9.5-E9.7 across 3 seeds |
| `scripts/aggregate_kcn_focus_results.py` | Updated to include E9.5-E9.7 in the same summary |

## Run Commands

Check the scaffold:

```bash
.venv/bin/python scripts/check_experiment_scaffold.py --bash
```

Dry-run one v2 model:

```bash
.venv/bin/python experiments/exp9b_kcn_research_unified_encoder.py \
  --focus-mode pose \
  --run-name kcn_v2_pose_arcface_center \
  --metric-head arcface \
  --center-weight 0.015 \
  --pose-languages casl_si ksl nsi \
  --no-gsl-sentence-landmarks \
  --dry-run
```

Submit the full E9-v2 sweep:

```bash
sbatch --parsable psc_jobs/92_kcn_v2_research_array.sbatch
```

Aggregate after completion:

```bash
.venv/bin/python scripts/aggregate_kcn_focus_results.py \
  --results-root results \
  --out-dir results/exp9_kcn_focus_summary
```

## Expected Improvement Target

The realistic target is not just higher overall macro-F1. We specifically want:

1. E9.5 pose-only to beat E9.2 pose-only.
2. E9.6 to improve CASL RGB frame Macro-F1.
3. E9.7 to beat E9.4 and get closer to E9.2 while preserving the KSLC RGB gain.
4. NSL Macro-F1 to improve beyond 50.55% without relying on KSL's saturated
   four-class score.

## References For The Next Iteration

- [Supervised Contrastive Learning, NeurIPS 2020](https://arxiv.org/abs/2004.11362)
- [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](https://arxiv.org/abs/1801.07698)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://arxiv.org/abs/1801.09414)
- [SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition](https://arxiv.org/abs/2111.03607)
- [Multiscale Vision Transformers](https://arxiv.org/abs/2104.11227)
- [Multiscale Vision Transformers V2](https://arxiv.org/abs/2112.01526)
- [Skeleton-Based Action Recognition with Spatial Temporal Graph Convolutional Networks](https://arxiv.org/abs/1801.07455)