social-data-attribution-w2/scripts/monitoring/pace_job_monitor.md

PACE Job Monitor — Agent Dispatch Template

Reusable prompt for dispatching a subagent to monitor SLURM enrichment jobs on PACE ICE. Invoke by passing this prompt to an Agent tool call with subagent_type: "general-purpose".

Parameters

Replace these before dispatching:

• {{JOB_ID}} — primary array job ID (e.g., 4441409)
• {{LAUNCHER_JOB_ID}} — launcher job ID (e.g., 4441677)
• {{TOTAL_TASKS}} — total array tasks in the full pipeline (e.g., 2826)
• {{TOTAL_SHARDS}} — total shards being processed (e.g., 56514)
• {{SHARDS_PER_TASK}} — shards per array task (e.g., 20)
• {{USERNAME}} — SLURM username (e.g., gmatlin3)
• {{WORKTREE_PATH}} — worktree path on PACE (e.g., ~/dev/data-attribution-soc91)

Agent Prompt

You are monitoring SLURM enrichment jobs on PACE ICE. Produce a detailed status
report with timing estimates broken down by GPU type, partition, and node.

IMPORTANT: Run all commands via `ssh pace-ice "command"`. Each SSH call is
independent (no persistent state). Use `2>/dev/null` on all ssh commands to
suppress connection messages.

### Data Collection Steps

Run these in parallel where possible:

1. **Task state counts:**
   ssh pace-ice "sacct -j {{JOB_ID}} --format=State --noheader -X 2>/dev/null | sort | uniq -c | sort -rn"

2. **Per-partition breakdown:**
   ssh pace-ice "sacct -j {{JOB_ID}} --format=Partition --noheader -X 2>/dev/null | sort | uniq -c | sort -rn"

3. **Completed/failed tasks with timing:**
   ssh pace-ice "sacct -j {{JOB_ID}} --format=JobID%20,Partition%12,State%12,Elapsed%12,ExitCode --noheader -X 2>/dev/null | grep -E 'COMPLETED|FAILED' | head -30"

4. **Node distribution (running tasks only):**
   ssh pace-ice "squeue -u {{USERNAME}} -j {{JOB_ID}} --format='%.18i %.12P %.30R' --noheader 2>/dev/null | awk '{print \$2, \$3}' | sort | uniq -c | sort -rn"

5. **GPU types on active nodes** (pick 3-5 representative nodes from step 4):
   ssh pace-ice "scontrol show node <NODE_NAME> 2>/dev/null | grep -E 'Gres|NodeName|Partitions'"

6. **Shard-level progress from task logs** (check stderr for processing output):
   ssh pace-ice "tail -20 ~/scratch/soc91/logs/soc91_enrich/{{JOB_ID}}_0.err 2>/dev/null"
   ssh pace-ice "tail -20 ~/scratch/soc91/logs/soc91_enrich/{{JOB_ID}}_0.out 2>/dev/null"
   (repeat for 2-3 tasks on different partitions/GPUs)

7. **Launcher status:**
   ssh pace-ice "sacct -j {{LAUNCHER_JOB_ID}} --format=JobID,State,Elapsed --noheader -X 2>/dev/null"
   ssh pace-ice "tail -15 ~/scratch/soc91/logs/soc91_launcher/{{LAUNCHER_JOB_ID}}.out 2>/dev/null"
   ssh pace-ice "cat {{WORKTREE_PATH}}/logs/soc91_enrich/launcher_state_coe-ice.txt 2>/dev/null"

8. **R2 completion check** (if tasks have completed, sample stats.json):
   ssh pace-ice "cd {{WORKTREE_PATH}} && source ~/.r2_credentials && python3 -c \"
import boto3, json, os
s3 = boto3.client('s3', endpoint_url='https://0934ab8e84ac8e4e81decaf3eb121337.r2.cloudflarestorage.com', aws_access_key_id=os.environ['R2_ACCESS_KEY_ID'], aws_secret_access_key=os.environ['R2_SECRET_ACCESS_KEY'])
paginator = s3.get_paginator('list_objects_v2')
count = 0
for page in paginator.paginate(Bucket='soc127-dedup', Prefix='soc91-labels/'):
    for obj in page.get('Contents', []):
        if obj['Key'].endswith('.done'):
            count += 1
print(f'Total .done markers in R2: {count}')
\""

### Report Format

Produce a structured report with these sections:

#### 1. Summary
- Total tasks: running / pending / completed / failed
- Overall progress percentage
- Estimated completion time

#### 2. GPU and Partition Breakdown
Table with: partition, GPU type, VRAM, tasks running, tasks completed,
avg elapsed time, throughput (docs/sec if available from logs)

#### 3. Node Distribution
Table with: node name, partition, GPU type, tasks running on it

#### 4. Shard-Level Progress
For sampled tasks: how many shards completed within each task,
docs processed, docs/sec rate, estimated time to task completion

#### 5. Launcher Status
Current state, checkpoint position, next batch to submit

#### 6. R2 Completion Status
Total .done markers (completed shards) in R2

#### 7. ETA Calculation
- Per-task average time (from completed tasks or extrapolation)
- Remaining tasks / concurrent slots = remaining waves
- Wall time estimate for current batch and full pipeline
- Factor in launcher submitting future batches

#### 8. Warnings
Flag any: failures, timeouts, error patterns in logs, stalled tasks,
disk/quota issues, launcher problems