docs/runbook.md

SZL Holdings — On-Call Runbook

Doctrine v11 LOCKED 749/14/163 | SLSA L1 honest | Generated: 2026-06-03

Quick Reference

Flagship	Space URL	Lambda	Honest	Critical Endpoint
a11oy	https://szlholdings-a11oy.hf.space	/v1/lambda	/v1/honest	/api/a11oy/v4/fleet
sentra	https://szlholdings-sentra.hf.space	/api/sentra/v1/lambda	/api/sentra/v1/honest	/api/sentra/v1/verdict
amaru	https://szlholdings-amaru.hf.space	/api/amaru/v1/lambda	/api/amaru/v1/honest	/api/amaru/v1/brain
rosie	https://szlholdings-rosie.hf.space	/api/rosie/v1/lambda	/api/rosie/v1/honest	/api/rosie/v1/brain
killinchu	https://szlholdings-killinchu.hf.space	/api/killinchu/v1/lambda	/api/killinchu/v1/honest	/api/killinchu/v1/lambda

Incident Playbooks

INC-01: HF Space is DOWN (not RUNNING)

Symptoms: Space stage = BUILD_ERROR or STOPPED or APP_STARTING for >5 min

Diagnosis:

curl -s https://huggingface.co/api/spaces/SZLHOLDINGS/<flagship>/runtime | python3 -c "import json,sys; print(json.load(sys.stdin))"

Resolution steps:

1. Check HF Space logs via https://huggingface.co/spaces/SZLHOLDINGS//logs
2. If BUILD_ERROR: Check recent commits for Dockerfile issues. Look for broken COPY lines.
3. If the error is cache miss: [N/N] COPY --chown=user <file>: file listed in Dockerfile does not exist in Space. Remove that COPY line.
4. Push a minimal fix commit via huggingface_hub (NOT the NDJSON API which is unreliable)
5. Wait 2-3 min for rebuild; verify stage: RUNNING
6. Run smoke test: curl https://szlholdings-<flagship>.hf.space/api/<flagship>/v1/lambda
7. If still failing after 2 rebuild attempts, file GitHub Issue with incident label

Rollback:

# Get previous good commit SHA from HF commit log
curl -s https://huggingface.co/api/spaces/SZLHOLDINGS/<flagship>/commits/main?limit=10
# Revert to good SHA via huggingface_hub

---

INC-02: Endpoint returns 404 (regression)

Symptoms: Lambda, honest, or other CTO-signed endpoint returns 404

Diagnosis:

curl -sv https://szlholdings-<flagship>.hf.space/api/<flagship>/v1/lambda 2>&1 | tail -20

Common causes:

1. HF race condition: Multiple commits to same Space within 5 minutes caused file corruption
   • Check: curl -s https://huggingface.co/api/spaces/SZLHOLDINGS/<flagship>/commits/main?limit=5
   • Fix: Push the correct file content via huggingface_hub.upload_file()
2. Import failure: Module used by route fails to import, route never registers
   • Fix: Add try/except around import; check that module is COPY'd in Dockerfile
3. Mount ordering: Starlette /api/<flagship> mount shadows explicit routes
   • Fix: Register explicit routes BEFORE calling app.mount("/api/<flagship>", ...)

---

INC-03: Doctrine violation in live response

Symptoms: doctrine field ≠ v11, declarations ≠ 749, or sorries_total ≠ 163

Diagnosis:

curl -s https://szlholdings-<flagship>.hf.space/api/<flagship>/v1/lambda | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('doctrine'), d.get('declarations'), d.get('sorries_total'))"

Resolution:

1. Identify the commit that introduced the violation via HF commit log
2. Check: is DOCTRINE = "v10" or any other non-v11 value in serve.py/app.py?
3. Fix: Update DOCTRINE constant; commit with DCO trailers
4. Push via GitHub → HF sync or huggingface_hub.upload_file()
5. CRITICAL: Never change 749/14/163 — these are LOCKED

---

INC-04: GitHub Actions CI failing

Symptoms: Red check on main branch

Diagnosis:

gh run list --repo szl-holdings/<flagship> --limit 5
gh run view <run-id> --repo szl-holdings/<flagship> --log-failed

Common CI failures:

• gitleaks: Secret detected → do NOT push fix to public; rotate credential immediately
• trivy/grype: HIGH/CRITICAL CVE in base image → update base image pinning
• dco: Commit missing Signed-off-by: → rebase + amend with -s
• doctrine-grep: Doctrine violation pattern detected → fix inline

---

Escalation Matrix

Severity	Condition	Escalate To	SLA
Critical	Doctrine violation in live response	Founder immediately	15 min
Critical	Secret leaked in response/logs	Founder + rotate credentials	30 min
High	2+ flagships down simultaneously	On-call team	1 hour
High	Build failing for >30 min	On-call team	2 hours
Medium	Single flagship 404 on CTO endpoint	On-call team	4 hours
Low	CI failing but prod OK	Team	Next business day

Required DCO on All Fix Commits

Signed-off-by: Yachay <yachay@szlholdings.ai>
Co-Authored-By: Perplexity Computer Agent <agent@perplexity.ai>

Signed-off-by: Yachay yachay@szlholdings.ai
Co-Authored-By: Perplexity Computer Agent agent@perplexity.ai