| # Agent runbook |
|
|
| For AI agents operating this stack. Humans: see README.md first. |
|
|
| ## Fast paths |
|
|
| **Is it up?** |
| ```bash |
| curl -s -o /dev/null -w '%{http_code}\n' http://NODE1:8004/health # want 200 |
| ``` |
|
|
| **Bring the pair up** |
| ```bash |
| bash scripts/04-launch-pair.sh # r1 first, then r0; health ≤ 7.5 min |
| ``` |
|
|
| **Bring it down** |
| ```bash |
| ssh NODE2 'docker rm -f qwen38-dflash2-tp2-r1' ; ssh NODE1 'docker rm -f qwen38-dflash2-tp2-r0' |
| ``` |
|
|
| **Quick health of both ranks** |
| ```bash |
| ssh NODE1 'docker logs --tail 3 qwen38-dflash2-tp2-r0 2>&1' | tail -3 |
| ssh NODE2 'docker logs --tail 3 qwen38-dflash2-tp2-r1 2>&1' | tail -3 |
| ``` |
|
|
| **Bench once** |
| ```bash |
| python3 bench/edit_bench.py http://NODE1:8004/v1 qwen3.8-27b-dflash2 |
| ``` |
|
|
| ## Failure signatures → root causes |
|
|
| | Symptom | Root cause | Fix | |
| |---|---|---| |
| | boot dies: unquantized LM head gate | lm_head quantized (wrong checkpoint) | run 02 + 03, relaunch | |
| | health 000 after >8 min | see logs; often wrong checkpoint or NCCL | `docker logs qwen38-dflash2-tp2-r0` | |
| | accept ~99.6%, tps low | K=7 (1 block) | set K=16 in launch | |
| | accept ~66%, tps below peak | K=24 | set K=16 | |
| | r0 up, r1 dead | r1 started after r0 | relaunch both, r1 first | |
| |
| ## Hard rules |
| |
| - K must be a multiple of 8. |
| - rank1 before rank0, always. |
| - Kill stale background launcher jobs before relaunching (zombie once rm -f'd |
| a healthy pair and relaunched the wrong checkpoint). |
| - Never edit config.json in place on hardlink copies — temp+os.replace. |
| - The original `unsloth-nvfp4/` checkpoint must stay pristine (it is the |
| rollback). If corrupted, re-download via 00. |
| |
