code/docs/EXPERIMENT_CODE_MAP.md

# AfriSignEncoder Experiment Code Map

This document maps the research experiments to the code that prepares data,
trains models, aggregates results, and checks reproducibility. It is meant to be
the first place to look before submitting PSC jobs or explaining the codebase to
a collaborator.

## Quick Scaffold Check

Run this from the repository root before pushing or submitting jobs:

```bash
python scripts/check_experiment_scaffold.py
```

On PSC, also check Slurm script syntax:

```bash
.venv/bin/python scripts/check_experiment_scaffold.py --bash
```

## Data Preparation Code

| Purpose | Main code | Notes |
|---|---|---|
| Global image/video/landmark manifests | `scripts/build_global_manifests.py`, `scripts/build_remote_manifests.py` | Remote-first catalogs; raw data stays outside git |
| CASL RGB + pose paired manifest | `scripts/build_casl_rgb_pose_pair_manifest.py` | Random split; useful baseline only |
| CASL signer-independent manifest | `scripts/build_casl_signer_independent_rgb_pose_manifest.py` | Preferred protocol for CASL |
| CASL official split audit | `scripts/build_casl_official_rgb_pose_manifest.py`, `scripts/filter_manifest_unseen_test_signers.py` | Used to compare against reported CASL split counts and signer leakage |
| CASL/KSL video frames | `scripts/extract_paired_video_frames.py`, `scripts/extract_word_video_frames.py` | Converts videos to fixed frame folders for cheaper RGB training |
| KSL word RGB + pose manifest | `scripts/build_ksl_word_rgb_pose_manifest.py` | Uses local/remote KSL word video sources when available |
| KSLC fixed image split | `scripts/build_kslc_fixed_image_manifest.py` | Uses labeled Kaggle train.csv; hidden Kaggle test stays unlabeled |
| NSL RGB image manifest | `scripts/build_nsl_local_image_manifest.py` | Used when local NSL images are available |
| GSL/GhSL health sentence landmarks | `scripts/download_gsl_sentence_landmarks.py`, `scripts/build_gsl_sentence_landmark_manifest.py` | Uses Hugging Face landmark dataset plus metadata |
| Leakage audit | `scripts/audit_video_pose_manifest_leakage.py` | Checks clip/path/stem/signer overlap |

## Single-Language And Modality Experiments

| Experiment | Dataset/language | Modality | Main training code | PSC job | Aggregator/output |
|---|---|---|---|---|---|
| KSL notebook baselines | KSL word-level | RGB, pose, fusion | `ksl_word_video_multimodal_experiments.ipynb` | Colab/manual | Notebook result tables |
| CASL RGB/pose architecture sweep | CASL-W60 | RGB, pose, RGB+pose | `experiments/exp5_video_rgb_pose_architectures.py` | `psc_jobs/60_casl_rgb_pose_arch_array.sbatch` | `scripts/aggregate_video_rgb_pose_results.py` |
| CASL signer-independent modality | CASL-W60 | RGB, pose, fusion | `experiments/exp5_video_rgb_pose_architectures.py` | `psc_jobs/64_casl_signer_independent_modality_array.sbatch` | `results/rgb_pose_summary_signer_independent/` |
| CASL frame RGB baseline | CASL-W60 | RGB frames | `experiments/exp7_frame_baselines.py` | `psc_jobs/67_e2_rgb_casl_si_frame_array.sbatch` | `scripts/aggregate_frame_baseline_results.py` |
| KSL frame/pose ablation | KSL word videos | RGB frames, pose, fusion | `experiments/exp4_frame_pose_modality_ablation.py` | `psc_jobs/77_e4_ksl_frame_pose_modality_array.sbatch` | `scripts/aggregate_video_rgb_pose_results.py` |
| GhSL/GSL health sentence pose | GSL Health Sentences | pose sentence | `experiments/exp6_ghsl_health_sentence_landmark.py` | `psc_jobs/70_ghsl_health_sentence_landmark_array.sbatch` | `scripts/aggregate_ghsl_health_sentence_results.py` |

## Multilingual Pose Experiments

| Experiment | Meaning | Main training code | PSC job | Aggregator |
|---|---|---|---|---|
| E1 monolingual pose | Separate model per language | `experiments/exp1_single_baseline.py` | `psc_jobs/10_e1_pose_mono_array.sbatch` | `scripts/aggregate_multilingual_landmark_results.py` |
| E2 naive pooled pose | One shared encoder trained on pooled pose data | `experiments/exp2_pooled_landmark_baseline.py` | `psc_jobs/20_e2_pooled_pose_array.sbatch` | `scripts/aggregate_multilingual_landmark_results.py` |
| E2b pooled architecture sweep | Pooled variants such as BiLSTM/TCN/Transformer/TransSLR | `experiments/exp2b_pooled_landmark_architectures.py` | `psc_jobs/25_e2b_pooled_architectures_array.sbatch` | `scripts/aggregate_pooled_architecture_results.py` |
| E2c pooled TransSLR CASL-SI | Stronger pooled pose baseline using signer-independent CASL | `experiments/exp2c_pooled_transslr_casl_si.py` | `psc_jobs/27_e2c_pooled_transslr_casl_si_array.sbatch` | `scripts/aggregate_pooled_architecture_results.py` |
| E3 language-aware pose | Shared encoder with language conditioning | `experiments/exp3_language_aware_landmark.py` | `psc_jobs/30_e3_language_aware_pose_array.sbatch` | `scripts/aggregate_multilingual_landmark_results.py` |
| E3 proposed pose encoder | Shared encoder + language embeddings/adapters/metric loss | `experiments/exp3b_proposed_language_aware_encoder.py` | `psc_jobs/37_e3b_metric_adapter_array.sbatch` | `scripts/aggregate_metric_adapter_results.py` |
| E4 balanced adapter pose | Balanced multilingual adapter model | `experiments/exp4_balanced_adapter_landmark.py` | `psc_jobs/35_e4_balanced_adapter_pose_array.sbatch` | `scripts/aggregate_multilingual_landmark_results.py` |

## Unified Mixed-Modality Experiments

| Experiment | Main idea | Main training code | PSC job | Aggregator |
|---|---|---|---|---|
| Exp8 unified mixed encoder | One model across pose word, pose image, pose sentence, RGB image, RGB word tasks | `experiments/exp8_unified_mixed_encoder.py` | `psc_jobs/80_exp8_unified_mixed_encoder_array.sbatch` | `scripts/aggregate_unified_encoder_results.py` |
| Exp8 full multimodal | Adds prepared RGB frame streams where available | `experiments/exp8_unified_mixed_encoder.py` | `psc_jobs/82_exp8_unified_full_multimodal_array.sbatch` | `scripts/aggregate_unified_encoder_results.py` |
| Exp8-v2 strong unified | Stronger unified model with better training defaults | `experiments/exp8_v2_strong_unified_encoder.py` | `psc_jobs/84_exp8_v2_strong_unified_full_array.sbatch` | `scripts/aggregate_unified_encoder_results.py` |
| Exp9 KSL+CASL+NSL focused | Main paper track focused on the reliable languages/modalities | `experiments/exp9_kcn_focused_unified_encoder.py` | `psc_jobs/90_kcn_joint_experiments_array.sbatch` | `scripts/aggregate_kcn_focus_results.py` |
| Exp9-v2 research upgrade | E9.5-E9.7 ArcFace/CosFace + center-loss upgrade over the focused baseline | `experiments/exp9b_kcn_research_unified_encoder.py` | `psc_jobs/92_kcn_v2_research_array.sbatch` | `scripts/aggregate_kcn_focus_results.py` |

## Exp9 KSL+CASL+NSL Joint Experiments

These are the focused joint experiments for the paper after removing the
noisiest train-only or huge-class streams.

| ID | Name | What it tests |
|---|---|---|
| E9.1 | KSL+CASL+NSL pooled pose-only | Strong pooled TransSLR-style pose baseline |
| E9.2 | KSL+CASL+NSL proposed pose-only | Same languages, but language/task-aware unified encoder |
| E9.3 | KSL+CASL+NSL proposed RGB/image-only | Whether RGB/image appearance streams can learn useful shared representations |
| E9.4 | KSL+CASL+NSL proposed RGB+pose/image | Main multimodal focused model |
| E9.5 | KSL+CASL+NSL research-v2 pose-only | ArcFace/CosFace + center loss upgrade for the pose-only winner |
| E9.6 | KSL+CASL+NSL research-v2 RGB/image-only | Metric-learning upgrade for RGB/image streams |
| E9.7 | KSL+CASL+NSL research-v2 RGB+pose/image | Metric-learning upgrade for the full multimodal model |

Submit the full focused sweep:

```bash
sbatch --parsable psc_jobs/90_kcn_joint_experiments_array.sbatch
```

Aggregate after jobs finish:

```bash
.venv/bin/python scripts/aggregate_kcn_focus_results.py
```

## What Should Not Be Committed

The repository should track source code, notebooks, docs, small metadata, and
manifest JSON files. Do not commit raw videos, extracted frames, checkpoints, or
large generated result folders unless a specific release artifact is requested.