social-data-attribution-w2/docs/SOC91_GPU_ENRICHMENT_HANDOFF.md

SOC-91 PACE ICE GPU Enrichment — Full Handoff (2026-03-15)

What the pipeline does

SOC-91 enriches ~56,514 deduplicated Dolma3 shards with WebOrganizer labels (topic + format classification) using GPU jobs on Georgia Tech's PACE ICE HPC cluster. The pipeline runs 4 classifiers per document: TopicClassifier, TopicClassifier-NoURL, FormatClassifier, FormatClassifier-NoURL. Output is labels-only Parquet sidecars uploaded to Cloudflare R2.

Architecture:

• A launcher job (soc91_launcher, sbatch wrapper around launch_dual.sh) runs on a CPU node and submits GPU enrichment tasks in batches of 5
• Each enrichment task (soc91_enrich) gets 1 GPU, reads a line from the manifest (~20 shard paths per line), downloads each shard from R2, runs 4 classifiers, uploads Parquet sidecar + .done marker + .stats.json back to R2
• Idempotency: before processing a shard, checks if .done marker exists in R2. Safe to resubmit any task
• State file: logs/soc91_enrich/launcher_state_coe-ice.txt tracks which manifest line the launcher will submit next. Launcher reads this on restart to resume
• Manifest indexing: enrich_sidecar.py reads manifest line N via manifest.read_text().splitlines()[task_id]. The TASK_OFFSET env var maps SLURM array indices to manifest line numbers (since SLURM array indices are capped at 1000)

State at handoff (~22:35 EDT 2026-03-15)

• R2 progress: 19,348/56,514 shards done (34.2%)
• Running GPUs: 8 (severely underutilized, down from 144+ earlier in the week)
• Pending jobs: 0 (396 blocked pending jobs were just cancelled)
• Launcher: alive (job 4461161, ~4.1h into 8h walltime), at task 2,372/2,826
• Launcher log shows: it's now submitting again since pending slots were freed. Was stuck repeating "At cap (404/400 active, 8 running), waiting..." before the cancel

---

Known problems

Problem 1: Hidden maintenance reservation blocking all pending jobs

• All 396 pending jobs were stuck with reason ReqNodeNotAvail, Reserved for maintenance
• SLURM's PrivateData = accounts,jobs,reservations,usage,users setting hides reservation details from non-admin users. We cannot see when maintenance starts/ends or which nodes are affected
• Our jobs request --time=16:00:00 (16h walltime). If maintenance starts within 16h, SLURM won't schedule any new jobs because it can't guarantee completion before maintenance begins
• Result: pending jobs consume submit slots (QOS limit = 500) but never run
• Key question: would shorter walltime (4h or 8h) allow jobs to schedule? Each GPU processes ~10 shards/hr, so a 4h job can complete its ~20 shards if the GPU is fast enough

Problem 2: Submit slot exhaustion (QOSMaxSubmitJobPerUserLimit = 500)

• QOS coe-ice allows max 500 submitted jobs (running + pending combined)
• 396 dead-weight pending jobs + 8 running = 404 active, leaving almost no room
• Launcher has its own MAX_ACTIVE=400 cap (line 9 of launch_dual.sh) and was stuck because active count exceeded it
• As running jobs completed, GPU count declined with no replacements: 144 → 138 → 132 → 127 → 122 → 115 → 12 → 8
• Fix applied: cancelled all 396 pending jobs via scancel -u gmatlin3 --name=soc91_enrich -t PENDING

Problem 3: Shard selection bias (sequential manifest)

• The manifest (r2_shard_manifest.txt) is alphabetically sorted: 2,826 lines, ~20 shards each
• Launcher processes sequentially (task 0, 1, 2...), so all 19,348 completed shards are from common_crawl subcategories starting with letters A-S
• Zero coverage of: olmocr_science_pdfs (21,429 shards, 0%), phase2_nonpool (256 shards, 0%), and 7 common_crawl subcategories (software, software_development, sports_and_fitness, transportation, travel_and_tourism, social_life, fashion_and_beauty)
• 8 common_crawl subcategories are at 100% while others are at 0%
• Fix built but not deployed: build_prioritized_manifest.py creates a new manifest with 7 tiers by category completion rate, shuffled within tiers, skipping 100%-done categories. Dry-run verified: 39,294 remaining shards across ~1,965 lines

Problem 4: Draining/drained nodes

• 6 GPU nodes currently unavailable (4 drained, 1 drained*, 1 draining)
• As running jobs on draining nodes complete, those GPUs become permanently unavailable until maintenance ends
• This causes the steady decline in running GPUs

Resolved misdiagnosis: "Ghost GPU allocations"

• Initially appeared that 89 GPUs on H100/H200 nodes had zero jobs
• Root cause: PrivateData = jobs hides other users' jobs from squeue
• The allocations were real jobs from other users, not a bug

---

What was done on 2026-03-15

1. Diagnosed the hidden maintenance reservation as root cause (not a misconfiguration on our end)
2. Built manifest_coverage.py — reports per-subcategory completion rates vs R2
3. Built build_prioritized_manifest.py — prioritized manifest with 7 tiers. Dry-run output:
   • T0: 28,879 shards (0% done, 16 categories including all olmocr + phase2)
   • T1: 2,130 shards (0-15% done)
   • T2: 5,350 shards (15-35% done)
   • T3: 2,176 shards (35-60% done)
   • T4: 180 shards (60-80% done)
   • T5: 538 shards (80-95% done)
   • T6: 41 shards (95-100% done)
   • Skipped: 8,130 shards from 100% complete categories
4. Cancelled 396 blocked pending jobs to free submit slots

---

Action items for next session

1. Test shorter walltime: Submit a single test job with --time=4:00:00 to see if it schedules. If it does, the maintenance window is >4h away and shorter walltimes bypass the scheduling block:

   ssh pace-ice "cd ~/dev/data-attribution-soc91 && sbatch --qos=coe-ice --time=4:00:00 --array=0-0 scripts/slurm/enrich_sidecar_gpu.sbatch"
   

   Check if it goes to RUNNING or PENDING with squeue -u gmatlin3 -h -t PENDING -o '%i %r'

2. Deploy the prioritized manifest:

   ssh pace-ice "cd ~/dev/data-attribution-soc91 && source ~/.r2_credentials && python3 scripts/slurm/build_prioritized_manifest.py"
   

   This writes scripts/slurm/r2_shard_manifest_prioritized.txt. Then update the launcher to use it by either:

   • Setting MANIFEST=scripts/slurm/r2_shard_manifest_prioritized.txt in the sbatch environment
   • Or editing enrich_sidecar_gpu.sbatch line 102: change MANIFEST="${MANIFEST:-scripts/slurm/r2_shard_manifest.txt}" to point at the prioritized manifest
   • Reset the state file: echo 0 > logs/soc91_enrich/launcher_state_coe-ice.txt

3. Resubmit launcher when the current one expires (~3.9h remaining on job 4461161):

   ssh pace-ice "cd ~/dev/data-attribution-soc91 && sbatch --qos=coe-ice --partition=ice-cpu --time=8:00:00 scripts/slurm/launcher.sbatch"
   

   If using shorter walltime for GPU jobs, also update enrich_sidecar_gpu.sbatch line 8 (--time=16:00:00) before resubmitting

4. Monitor GPU recovery: after cancelling pending jobs the launcher should be submitting new tasks. Verify with:

   ssh pace-ice "squeue -u gmatlin3 -h --name=soc91_enrich -t RUNNING | wc -l"
   ssh pace-ice "squeue -u gmatlin3 -h -r --name=soc91_enrich -t PENDING | wc -l"
   ssh pace-ice "tail -5 ~/dev/data-attribution-soc91/logs/soc91_launcher/4461161.out"
   

5. Determine maintenance window: try scontrol show reservation (may return nothing due to PrivateData), check https://pace.gatech.edu for announcements, or email pace-support@oit.gatech.edu

6. R2 completion check:

   ssh pace-ice "source ~/.r2_credentials && python3 -c \"import boto3,os;s3=boto3.client('s3',endpoint_url='https://0934ab8e84ac8f4e81decaf3eb121337.r2.cloudflarestorage.com',aws_access_key_id=os.environ['R2_ACCESS_KEY_ID'],aws_secret_access_key=os.environ['R2_SECRET_ACCESS_KEY'],region_name='auto');p=s3.get_paginator('list_objects_v2');d=sum(1 for pg in p.paginate(Bucket='soc127-dedup',Prefix='soc91-labels/') for o in pg.get('Contents',[]) if o['Key'].endswith('.done'));print(f'r2_done={d}/56514 ({d/56514*100:.1f}%)');\""
   

---

Key files on cluster (~/dev/data-attribution-soc91/)

File	Purpose
scripts/slurm/r2_shard_manifest.txt	Current manifest (alphabetical, 2,826 lines, 56,514 shards)
scripts/slurm/r2_shard_manifest_prioritized.txt	Output of prioritized builder (not yet generated)
scripts/slurm/build_prioritized_manifest.py	Builds prioritized manifest from R2 .done state
scripts/slurm/manifest_coverage.py	Reports per-subcategory completion rates
scripts/slurm/launch_dual.sh	Launcher logic (MAX_ACTIVE=400, BATCH=5, state checkpoint)
scripts/slurm/launcher.sbatch	Launcher sbatch wrapper (CPU node, 18h walltime)
scripts/slurm/enrich_sidecar_gpu.sbatch	GPU enrichment job (1 GPU, 16h walltime, auto-detects VRAM/dtype)
scripts/enrich_sidecar.py	Enrichment worker (reads manifest by task_id, runs 4 classifiers)
logs/soc91_enrich/launcher_state_coe-ice.txt	Current state: 2372 (line number in manifest)
logs/soc91_launcher/4461161.out	Current launcher log

Key constants and infrastructure

Item	Value
R2 bucket	soc127-dedup
R2 output prefix	soc91-labels/
R2 endpoint	https://0934ab8e84ac8f4e81decaf3eb121337.r2.cloudflarestorage.com
R2 credentials	source ~/.r2_credentials on cluster
QOS	coe-ice (max submit: 500, max GPUs: 960)
SLURM partitions	ice-gpu,coe-gpu,ice-bw-gpu
GPU constraint	-C nvidia-gpu (any NVIDIA GPU)
Excluded nodes	atl1-1-03-014-16-0 (bad GPU)
Username	gmatlin3
Worktree path	~/dev/data-attribution-soc91
Launcher MAX_ACTIVE	400 (in launch_dual.sh)
Launcher BATCH	5 tasks per submission
GPU job walltime	16:00:00 (may need reduction)
Batch size	Auto-detected by VRAM (128/64/32/16)
Classifier max_length	8192 tokens
Dtype	bf16 (auto-downgrades to fp16 for compute capability < 8.0)

SSH access pattern

All cluster interaction is via ssh pace-ice "command". Each invocation is a fresh login shell. No state persists between calls. The user must have an active SSH control socket (ControlMaster). If SSH times out or hangs, the user needs to re-authenticate in a separate terminal with ssh pace-ice.

Throughput baseline

• Each GPU processes ~10 shards/hr (varies by GPU type and shard size)
• At 144 GPUs: 1,440 shards/hr (39h for remaining 37,166 shards)
• At 8 GPUs: 80 shards/hr (464h, not viable)
• Target: get GPU count back above 100 by fixing the scheduling issue