social-data-attribution-w2/scripts/monitoring/pace_job_monitor.md

# PACE Job Monitor — Agent Dispatch Template

Reusable prompt for dispatching a subagent to monitor SLURM enrichment jobs on PACE ICE.
Invoke by passing this prompt to an Agent tool call with `subagent_type: "general-purpose"`.

## Parameters

Replace these before dispatching:

- `{{JOB_ID}}` — primary array job ID (e.g., `4441409`)
- `{{LAUNCHER_JOB_ID}}` — launcher job ID (e.g., `4441677`)
- `{{TOTAL_TASKS}}` — total array tasks in the full pipeline (e.g., `2826`)
- `{{TOTAL_SHARDS}}` — total shards being processed (e.g., `56514`)
- `{{SHARDS_PER_TASK}}` — shards per array task (e.g., `20`)
- `{{USERNAME}}` — SLURM username (e.g., `gmatlin3`)
- `{{WORKTREE_PATH}}` — worktree path on PACE (e.g., `~/dev/data-attribution-soc91`)

## Agent Prompt

```
You are monitoring SLURM enrichment jobs on PACE ICE. Produce a detailed status
report with timing estimates broken down by GPU type, partition, and node.

IMPORTANT: Run all commands via `ssh pace-ice "command"`. Each SSH call is
independent (no persistent state). Use `2>/dev/null` on all ssh commands to
suppress connection messages.

### Data Collection Steps

Run these in parallel where possible:

1. **Task state counts:**
   ssh pace-ice "sacct -j {{JOB_ID}} --format=State --noheader -X 2>/dev/null | sort | uniq -c | sort -rn"

2. **Per-partition breakdown:**
   ssh pace-ice "sacct -j {{JOB_ID}} --format=Partition --noheader -X 2>/dev/null | sort | uniq -c | sort -rn"

3. **Completed/failed tasks with timing:**
   ssh pace-ice "sacct -j {{JOB_ID}} --format=JobID%20,Partition%12,State%12,Elapsed%12,ExitCode --noheader -X 2>/dev/null | grep -E 'COMPLETED|FAILED' | head -30"

4. **Node distribution (running tasks only):**
   ssh pace-ice "squeue -u {{USERNAME}} -j {{JOB_ID}} --format='%.18i %.12P %.30R' --noheader 2>/dev/null | awk '{print \$2, \$3}' | sort | uniq -c | sort -rn"

5. **GPU types on active nodes** (pick 3-5 representative nodes from step 4):
   ssh pace-ice "scontrol show node <NODE_NAME> 2>/dev/null | grep -E 'Gres|NodeName|Partitions'"

6. **Shard-level progress from task logs** (check stderr for processing output):
   ssh pace-ice "tail -20 ~/scratch/soc91/logs/soc91_enrich/{{JOB_ID}}_0.err 2>/dev/null"
   ssh pace-ice "tail -20 ~/scratch/soc91/logs/soc91_enrich/{{JOB_ID}}_0.out 2>/dev/null"
   (repeat for 2-3 tasks on different partitions/GPUs)

7. **Launcher status:**
   ssh pace-ice "sacct -j {{LAUNCHER_JOB_ID}} --format=JobID,State,Elapsed --noheader -X 2>/dev/null"
   ssh pace-ice "tail -15 ~/scratch/soc91/logs/soc91_launcher/{{LAUNCHER_JOB_ID}}.out 2>/dev/null"
   ssh pace-ice "cat {{WORKTREE_PATH}}/logs/soc91_enrich/launcher_state_coe-ice.txt 2>/dev/null"

8. **R2 completion check** (if tasks have completed, sample stats.json):
   ssh pace-ice "cd {{WORKTREE_PATH}} && source ~/.r2_credentials && python3 -c \"
import boto3, json, os
s3 = boto3.client('s3', endpoint_url='https://0934ab8e84ac8e4e81decaf3eb121337.r2.cloudflarestorage.com', aws_access_key_id=os.environ['R2_ACCESS_KEY_ID'], aws_secret_access_key=os.environ['R2_SECRET_ACCESS_KEY'])
paginator = s3.get_paginator('list_objects_v2')
count = 0
for page in paginator.paginate(Bucket='soc127-dedup', Prefix='soc91-labels/'):
    for obj in page.get('Contents', []):
        if obj['Key'].endswith('.done'):
            count += 1
print(f'Total .done markers in R2: {count}')
\""

### Report Format

Produce a structured report with these sections:

#### 1. Summary
- Total tasks: running / pending / completed / failed
- Overall progress percentage
- Estimated completion time

#### 2. GPU and Partition Breakdown
Table with: partition, GPU type, VRAM, tasks running, tasks completed,
avg elapsed time, throughput (docs/sec if available from logs)

#### 3. Node Distribution
Table with: node name, partition, GPU type, tasks running on it

#### 4. Shard-Level Progress
For sampled tasks: how many shards completed within each task,
docs processed, docs/sec rate, estimated time to task completion

#### 5. Launcher Status
Current state, checkpoint position, next batch to submit

#### 6. R2 Completion Status
Total .done markers (completed shards) in R2

#### 7. ETA Calculation
- Per-task average time (from completed tasks or extrapolation)
- Remaining tasks / concurrent slots = remaining waves
- Wall time estimate for current batch and full pipeline
- Factor in launcher submitting future batches

#### 8. Warnings
Flag any: failures, timeouts, error patterns in logs, stalled tasks,
disk/quota issues, launcher problems
```