social-data-attribution-w2/docs/SOC91_GPU_ENRICHMENT_HANDOFF.md

# SOC-91 PACE ICE GPU Enrichment — Full Handoff (2026-03-15)

## What the pipeline does

SOC-91 enriches ~56,514 deduplicated Dolma3 shards with WebOrganizer labels (topic + format classification) using GPU jobs on Georgia Tech's PACE ICE HPC cluster. The pipeline runs 4 classifiers per document: TopicClassifier, TopicClassifier-NoURL, FormatClassifier, FormatClassifier-NoURL. Output is labels-only Parquet sidecars uploaded to Cloudflare R2.

**Architecture:**
- A **launcher job** (`soc91_launcher`, sbatch wrapper around `launch_dual.sh`) runs on a CPU node and submits GPU enrichment tasks in batches of 5
- Each **enrichment task** (`soc91_enrich`) gets 1 GPU, reads a line from the manifest (~20 shard paths per line), downloads each shard from R2, runs 4 classifiers, uploads Parquet sidecar + `.done` marker + `.stats.json` back to R2
- **Idempotency**: before processing a shard, checks if `.done` marker exists in R2. Safe to resubmit any task
- **State file**: `logs/soc91_enrich/launcher_state_coe-ice.txt` tracks which manifest line the launcher will submit next. Launcher reads this on restart to resume
- **Manifest indexing**: `enrich_sidecar.py` reads manifest line N via `manifest.read_text().splitlines()[task_id]`. The `TASK_OFFSET` env var maps SLURM array indices to manifest line numbers (since SLURM array indices are capped at 1000)

## State at handoff (~22:35 EDT 2026-03-15)

- **R2 progress**: 19,348/56,514 shards done (34.2%)
- **Running GPUs**: 8 (severely underutilized, down from 144+ earlier in the week)
- **Pending jobs**: 0 (396 blocked pending jobs were just cancelled)
- **Launcher**: alive (job 4461161, ~4.1h into 8h walltime), at task 2,372/2,826
- **Launcher log shows**: it's now submitting again since pending slots were freed. Was stuck repeating `"At cap (404/400 active, 8 running), waiting..."` before the cancel

---

## Known problems

### Problem 1: Hidden maintenance reservation blocking all pending jobs

- All 396 pending jobs were stuck with reason `ReqNodeNotAvail, Reserved for maintenance`
- SLURM's `PrivateData = accounts,jobs,reservations,usage,users` setting hides reservation details from non-admin users. We cannot see when maintenance starts/ends or which nodes are affected
- Our jobs request `--time=16:00:00` (16h walltime). If maintenance starts within 16h, SLURM won't schedule any new jobs because it can't guarantee completion before maintenance begins
- Result: pending jobs consume submit slots (QOS limit = 500) but never run
- **Key question**: would shorter walltime (4h or 8h) allow jobs to schedule? Each GPU processes ~10 shards/hr, so a 4h job can complete its ~20 shards if the GPU is fast enough

### Problem 2: Submit slot exhaustion (QOSMaxSubmitJobPerUserLimit = 500)

- QOS `coe-ice` allows max 500 submitted jobs (running + pending combined)
- 396 dead-weight pending jobs + 8 running = 404 active, leaving almost no room
- Launcher has its own `MAX_ACTIVE=400` cap (line 9 of `launch_dual.sh`) and was stuck because active count exceeded it
- As running jobs completed, GPU count declined with no replacements: 144 → 138 → 132 → 127 → 122 → 115 → 12 → 8
- **Fix applied**: cancelled all 396 pending jobs via `scancel -u gmatlin3 --name=soc91_enrich -t PENDING`

### Problem 3: Shard selection bias (sequential manifest)

- The manifest (`r2_shard_manifest.txt`) is alphabetically sorted: 2,826 lines, ~20 shards each
- Launcher processes sequentially (task 0, 1, 2...), so all 19,348 completed shards are from `common_crawl` subcategories starting with letters A-S
- **Zero coverage** of: `olmocr_science_pdfs` (21,429 shards, 0%), `phase2_nonpool` (256 shards, 0%), and 7 common_crawl subcategories (software, software_development, sports_and_fitness, transportation, travel_and_tourism, social_life, fashion_and_beauty)
- 8 common_crawl subcategories are at 100% while others are at 0%
- **Fix built but not deployed**: `build_prioritized_manifest.py` creates a new manifest with 7 tiers by category completion rate, shuffled within tiers, skipping 100%-done categories. Dry-run verified: 39,294 remaining shards across ~1,965 lines

### Problem 4: Draining/drained nodes

- 6 GPU nodes currently unavailable (4 drained, 1 drained*, 1 draining)
- As running jobs on draining nodes complete, those GPUs become permanently unavailable until maintenance ends
- This causes the steady decline in running GPUs

### Resolved misdiagnosis: "Ghost GPU allocations"

- Initially appeared that 89 GPUs on H100/H200 nodes had zero jobs
- Root cause: `PrivateData = jobs` hides other users' jobs from `squeue`
- The allocations were real jobs from other users, not a bug

---

## What was done on 2026-03-15

1. Diagnosed the hidden maintenance reservation as root cause (not a misconfiguration on our end)
2. Built `manifest_coverage.py` — reports per-subcategory completion rates vs R2
3. Built `build_prioritized_manifest.py` — prioritized manifest with 7 tiers. Dry-run output:
   - T0: 28,879 shards (0% done, 16 categories including all olmocr + phase2)
   - T1: 2,130 shards (0-15% done)
   - T2: 5,350 shards (15-35% done)
   - T3: 2,176 shards (35-60% done)
   - T4: 180 shards (60-80% done)
   - T5: 538 shards (80-95% done)
   - T6: 41 shards (95-100% done)
   - Skipped: 8,130 shards from 100% complete categories
4. Cancelled 396 blocked pending jobs to free submit slots

---

## Action items for next session

1. **Test shorter walltime**: Submit a single test job with `--time=4:00:00` to see if it schedules. If it does, the maintenance window is >4h away and shorter walltimes bypass the scheduling block:
   ```
   ssh pace-ice "cd ~/dev/data-attribution-soc91 && sbatch --qos=coe-ice --time=4:00:00 --array=0-0 scripts/slurm/enrich_sidecar_gpu.sbatch"
   ```
   Check if it goes to RUNNING or PENDING with `squeue -u gmatlin3 -h -t PENDING -o '%i %r'`

2. **Deploy the prioritized manifest**:
   ```
   ssh pace-ice "cd ~/dev/data-attribution-soc91 && source ~/.r2_credentials && python3 scripts/slurm/build_prioritized_manifest.py"
   ```
   This writes `scripts/slurm/r2_shard_manifest_prioritized.txt`. Then update the launcher to use it by either:
   - Setting `MANIFEST=scripts/slurm/r2_shard_manifest_prioritized.txt` in the sbatch environment
   - Or editing `enrich_sidecar_gpu.sbatch` line 102: change `MANIFEST="${MANIFEST:-scripts/slurm/r2_shard_manifest.txt}"` to point at the prioritized manifest
   - Reset the state file: `echo 0 > logs/soc91_enrich/launcher_state_coe-ice.txt`

3. **Resubmit launcher** when the current one expires (~3.9h remaining on job 4461161):
   ```
   ssh pace-ice "cd ~/dev/data-attribution-soc91 && sbatch --qos=coe-ice --partition=ice-cpu --time=8:00:00 scripts/slurm/launcher.sbatch"
   ```
   If using shorter walltime for GPU jobs, also update `enrich_sidecar_gpu.sbatch` line 8 (`--time=16:00:00`) before resubmitting

4. **Monitor GPU recovery**: after cancelling pending jobs the launcher should be submitting new tasks. Verify with:
   ```
   ssh pace-ice "squeue -u gmatlin3 -h --name=soc91_enrich -t RUNNING | wc -l"
   ssh pace-ice "squeue -u gmatlin3 -h -r --name=soc91_enrich -t PENDING | wc -l"
   ssh pace-ice "tail -5 ~/dev/data-attribution-soc91/logs/soc91_launcher/4461161.out"
   ```

5. **Determine maintenance window**: try `scontrol show reservation` (may return nothing due to PrivateData), check https://pace.gatech.edu for announcements, or email pace-support@oit.gatech.edu

6. **R2 completion check**:
   ```
   ssh pace-ice "source ~/.r2_credentials && python3 -c \"import boto3,os;s3=boto3.client('s3',endpoint_url='https://0934ab8e84ac8f4e81decaf3eb121337.r2.cloudflarestorage.com',aws_access_key_id=os.environ['R2_ACCESS_KEY_ID'],aws_secret_access_key=os.environ['R2_SECRET_ACCESS_KEY'],region_name='auto');p=s3.get_paginator('list_objects_v2');d=sum(1 for pg in p.paginate(Bucket='soc127-dedup',Prefix='soc91-labels/') for o in pg.get('Contents',[]) if o['Key'].endswith('.done'));print(f'r2_done={d}/56514 ({d/56514*100:.1f}%)');\""
   ```

---

## Key files on cluster (`~/dev/data-attribution-soc91/`)

| File | Purpose |
|------|---------|
| `scripts/slurm/r2_shard_manifest.txt` | Current manifest (alphabetical, 2,826 lines, 56,514 shards) |
| `scripts/slurm/r2_shard_manifest_prioritized.txt` | Output of prioritized builder (not yet generated) |
| `scripts/slurm/build_prioritized_manifest.py` | Builds prioritized manifest from R2 .done state |
| `scripts/slurm/manifest_coverage.py` | Reports per-subcategory completion rates |
| `scripts/slurm/launch_dual.sh` | Launcher logic (MAX_ACTIVE=400, BATCH=5, state checkpoint) |
| `scripts/slurm/launcher.sbatch` | Launcher sbatch wrapper (CPU node, 18h walltime) |
| `scripts/slurm/enrich_sidecar_gpu.sbatch` | GPU enrichment job (1 GPU, 16h walltime, auto-detects VRAM/dtype) |
| `scripts/enrich_sidecar.py` | Enrichment worker (reads manifest by task_id, runs 4 classifiers) |
| `logs/soc91_enrich/launcher_state_coe-ice.txt` | Current state: 2372 (line number in manifest) |
| `logs/soc91_launcher/4461161.out` | Current launcher log |

## Key constants and infrastructure

| Item | Value |
|------|-------|
| R2 bucket | `soc127-dedup` |
| R2 output prefix | `soc91-labels/` |
| R2 endpoint | `https://0934ab8e84ac8f4e81decaf3eb121337.r2.cloudflarestorage.com` |
| R2 credentials | `source ~/.r2_credentials` on cluster |
| QOS | `coe-ice` (max submit: 500, max GPUs: 960) |
| SLURM partitions | `ice-gpu,coe-gpu,ice-bw-gpu` |
| GPU constraint | `-C nvidia-gpu` (any NVIDIA GPU) |
| Excluded nodes | `atl1-1-03-014-16-0` (bad GPU) |
| Username | `gmatlin3` |
| Worktree path | `~/dev/data-attribution-soc91` |
| Launcher MAX_ACTIVE | 400 (in launch_dual.sh) |
| Launcher BATCH | 5 tasks per submission |
| GPU job walltime | 16:00:00 (may need reduction) |
| Batch size | Auto-detected by VRAM (128/64/32/16) |
| Classifier max_length | 8192 tokens |
| Dtype | bf16 (auto-downgrades to fp16 for compute capability < 8.0) |

## SSH access pattern

All cluster interaction is via `ssh pace-ice "command"`. Each invocation is a fresh login shell. No state persists between calls. The user must have an active SSH control socket (ControlMaster). If SSH times out or hangs, the user needs to re-authenticate in a separate terminal with `ssh pace-ice`.

## Throughput baseline

- Each GPU processes ~10 shards/hr (varies by GPU type and shard size)
- At 144 GPUs: ~1,440 shards/hr (~39h for remaining 37,166 shards)
- At 8 GPUs: ~80 shards/hr (~464h, not viable)
- Target: get GPU count back above 100 by fixing the scheduling issue