docs/modules/risk_book.md

risk_book.md — DriftCall Consolidated Risk Register

Version: v1.0 Owner: Person B (Rewards & Tests; CLAUDE.md §2.2) Primary source: DESIGN.md §14 (12 risks) + CLAUDE.md §11 (escalation & stop conditions) Consumes from: training.md §5, vendors.md §5, audio.md §5 Consumed by: env.md, training.md, audio.md, deploy_env_space.md, deploy_demo_space.md, pitch_demo.md, evaluation.md

---

1. Purpose

The Risk Book is the single consolidated risk register for DriftCall's 48-hour onsite execution (Apr 25–26, 2026) plus the ≈18 hours of pre-onsite doc + smoke work. It unifies three previously-separate concerns:

1. DESIGN.md §14 static register — 12 risks known at spec-lock time, each with P × I × mitigation.
2. Module-level error modes — concrete exceptions declared in training.md §5, vendors.md §5, audio.md §5 that, when they fire at runtime, are RISK EVENTS this book owns routing for.
3. CLAUDE.md §11 escalate / stop conditions — the gates that halt agent dispatch or terminate the plan.

Without this consolidation every teammate would independently rediscover stop conditions, escalation paths, and who owns which mitigation. This doc centralizes the answer so that — under the hackathon clock — decisions are table-lookups, not debates.

The Risk Book is live. It is consulted at three moments:

• Onsite start (hour 0 of Apr 25): Person B runs Risk.assess(), reviews every entry's owner + trigger_signal with that owner, and blesses the register as the active contract.
• Runtime (continuously): the orchestrator and module owners call Risk.triage(signal) whenever a module-level error mode fires or an observable trigger is crossed. Triage returns the rooted mitigation + any escalation path.
• Post-mortem (hour 48+): the RiskLog (append-only) feeds the reward-hacking probe (DESIGN.md §13 deliverable #9) and the blog post (§15 Pitch).

Goal: every mitigation has a named owner and an observable trigger signal. No "we'll handle it when it comes up." If a risk fires and no one knows they own it, that is the design failure this book exists to prevent.

---

2. Interface

driftcall/risk.py (authored in Phase C by Person B) exposes the following. Phase D spec only — no code yet.

2.1 RiskEntry (frozen dataclass)

from __future__ import annotations
from dataclasses import dataclass
from enum import Enum
from typing import Optional

@dataclass(frozen=True)
class RiskEntry:
    id: str                       # "R01" ... "R15", "R01-STOP", "R06-STOP", "R99" (stable)
    description: str              # one-sentence human-readable risk
    probability: Probability      # Low | Med | High
    impact: Impact                # low | med | high | kills
    mitigation: str               # concrete action; cites module doc §
    owner: str                    # "A" | "B" | "C" | "D" | "orchestrator" | "team"
    trigger_signal: TriggerSignal # observable condition that fires mitigation
    stop_condition: Optional[StopCondition] = None  # None if not a hard-stop risk
    design_refs: tuple[str, ...] = ()                # e.g. ("DESIGN.md §14 #1", "training.md §7a")

2.2 Enums

class Probability(str, Enum):
    LOW = "Low"
    MED = "Med"
    HIGH = "High"

class Impact(str, Enum):
    LOW = "low"            # nuisance; deliverable unaffected
    MED = "med"            # one deliverable degraded
    HIGH = "high"          # ≥ 2 deliverables at risk
    KILLS = "kills"        # training or demo cannot ship

class TriggerSignal(str, Enum):
    # Training-loop signals (mirrors training.md §5)
    GRAD_NORM_INF            = "grad_norm_inf"
    POLICY_KL_OVER_10        = "policy_kl_over_10"
    R5_DROP_WITH_HACK_SPIKE  = "r5_drop_with_hack_spike"
    CHECKPOINT_IO_ERROR      = "checkpoint_io_error"
    OOM_AT_G8                = "oom_at_g8"
    STAGE1_R1_BELOW_0_4_AT_100 = "stage1_r1_below_0_4_at_step_100"
    # Vendor signals (mirrors vendors.md §5)
    UNEXPECTED_STATUS_VALUE  = "vendor_status_not_in_five"
    ERROR_ENVELOPE_VIOLATION = "vendor_response_field_outside_catalogue"
    # Audio signals (mirrors audio.md §5)
    KOKORO_LOAD_FAILED       = "kokoro_model_load_error"
    WHISPER_LOAD_FAILED      = "whisper_model_load_error"
    INDIC_VOICE_PACK_MISSING = "indic_voice_pack_missing"
    # Infra / process signals
    V100_DOWN                = "v100_unreachable"
    V100_DOWN_OVER_8H        = "v100_unavailable_over_8h"
    HF_HUB_OUTAGE            = "hf_hub_or_spaces_5xx"
    HF_HUB_OUTAGE_OVER_2H    = "hf_hub_outage_over_2h"
    WANDB_STARTUP_FAIL       = "wandb_init_failed_nonoffline"
    DOCKER_IMAGE_OVER_2GB    = "docker_image_over_2gb"
    OPENENV_VALIDATE_FAIL    = "openenv_validate_failure"
    GEMMA4_SMOKE_FAIL        = "gemma4_e2b_smoke_test_failed"
    MERGE_CONFLICT_CROSS_OWNER = "merge_conflict_a_and_b_owned_files"
    TEAM_MEMBER_DROP         = "team_member_dropped"
    TEAM_3PERSON_BELOW_GATE  = "three_person_cannot_meet_phase_d_gate"
    JUDGE_LANGUAGE_MISMATCH  = "judge_does_not_speak_indic"
    # Meta
    UNKNOWN                  = "unknown"

class StopCondition(str, Enum):
    ESCALATE_TO_USER = "escalate"   # pause dispatch; ping user
    HARD_STOP        = "hard_stop"  # terminate plan; re-plan with user

2.3 Assessment + triage API

class Risk:
    @staticmethod
    def assess() -> list[RiskEntry]:
        """Return the frozen register (§6 below, as data).
        Called once at onsite start by Person B. No side effects.
        Invariant: `len(assess()) >= 15` and every entry has a non-None owner."""

    @staticmethod
    def triage(signal: TriggerSignal, *, context: Optional[str] = None) -> TriageResult:
        """Look up the RiskEntry keyed by `signal`, return the handling plan.
        Idempotent. Never raises — if `signal == UNKNOWN` or no entry matches,
        returns a TriageResult pointing to R99 (unknown-class risk)."""

@dataclass(frozen=True)
class TriageResult:
    entry: RiskEntry
    action: str            # copy of entry.mitigation, ready to execute
    escalate: bool         # True iff entry.stop_condition == ESCALATE_TO_USER or HARD_STOP
    hard_stop: bool        # True iff entry.stop_condition == HARD_STOP
    log_line: str          # one-line summary for RiskLog append

class RiskLog:
    """Append-only audit log. One line per triage call during the 48h.
    Serialized to `risk_log.jsonl` at episode end; feeds reward-hacking probe."""
    def append(self, t: TriageResult, ts_iso: str) -> None: ...
    def to_jsonl(self, path: str) -> None: ...
    def summary(self) -> dict[str, int]:
        """{risk_id: fire_count} for the post-mortem report."""

2.4 Wiring points

Caller	Trigger	How
training/train_grpo.py callback	KLDivergenceExplosion, RewardCollapseError, CheckpointIOError, OutOfMemoryError	Risk.triage(signal) then honor hard_stop / escalate
driftcall/env.py	AudioDecodeError, ModelLoadError, UnsupportedLanguageError	Risk.triage(signal) for logging; env already has local mitigations
driftcall/vendors/*.py	envelope-shape test fires	Risk.triage(ERROR_ENVELOPE_VIOLATION) — always hard_stop
Orchestrator (this CLAUDE session)	openenv validate fail, smoke-test fail, team drop	Risk.triage(signal) before deciding to escalate
deploy_env_space.md health check	HF Hub 5xx, Docker image > 2 GB	Risk.triage(signal) for the deploy runbook

---

3. Behavior Spec

3.1 Invariants

1. Minimum 17 entries. len(Risk.assess()) >= 17 — 12 from DESIGN.md §14 (R01–R12) + 3 mandated here (R13–R15) + R01-STOP + R06-STOP + R99 + room for post-onsite additions. Asserted in tests/test_risk_book.py.
2. Every entry has a named owner. Owner ∈ {A, B, C, D, orchestrator, team}. No None, no "TBD". Asserted.
3. Every entry has an observable trigger signal. Either a module error class (training.md §5, vendors.md §5, audio.md §5) or an explicit infra condition. No "we'll notice it." Asserted.
4. Every CLAUDE.md §11 condition is represented. Both escalation items (5) AND hard-stop items (3) appear as StopCondition-bearing entries. Asserted by a coverage test.
5. Mitigations cite their source. design_refs is non-empty for every entry. This lets a reviewer trace any mitigation back to the canonical spec.

3.2 Triage loop (runtime)

on signal s fired at time t:
    1. triage = Risk.triage(s)
    2. append triage to RiskLog with ts=t
    3. if triage.hard_stop:
           orchestrator halts all Agent dispatch
           SendMessage team-lead: "HARD_STOP fired for {triage.entry.id}: {triage.log_line}"
           await user intervention
    4. elif triage.escalate:
           orchestrator pauses new dispatch in affected scope
           SendMessage team-lead: "ESCALATE {triage.entry.id}: {triage.log_line}"
           continue other independent work
    5. else:
           owner executes triage.action in their worktree
           orchestrator continues

3.3 Escalation path (from CLAUDE.md §11 verbatim)

Signal	Scope	First responder	Escalates to
Gemma 3n E2B smoke fails on V100	training	Person C	User (block; may downshift to Gemma 3 4B)
openenv validate fails 3×	deploy	Person D	User (block; may need schema deviation approval)
Stage-1 R1 < 0.4 at step 100	training	Person C → Person B	User (curriculum/reward redesign required)
Critic flags consistent DESIGN.md flaw	any phase	Orchestrator	User (update spec before continuing)
Merge conflict across A-owned and B-owned files	any phase	Orchestrator	User — ownership was wrong (process fix)
V100 unavailable > 8h during onsite	training	Person C	Hard stop — re-plan with user
HF Hub / Spaces outage > 2h	deploy	Person D	Hard stop
Team member drops AND 3-person cannot meet Phase D gate	all	Orchestrator	Hard stop

3.4 Decision thresholds

Impact	Probability	Action
kills	any	Immediate surface on first trigger; mitigation is non-negotiable
high	High or Med	Immediate surface; mitigation inline
high	Low	Pre-stage mitigation; surface on first trigger
med	High	Pre-stage mitigation; surface on repeat
med	Med or Low	Defer to next batch boundary; log only on first trigger
low	any	Defer; log only

"Surface" = SendMessage team-lead + RiskLog.append. "Defer" = RiskLog.append only, no interruption.

3.5 Retry semantics

Mitigations that retry (e.g., HF Hub upload with 3× backoff) do so at their own layer (training.md §5 CheckpointIOError). This book does NOT implement retries — it routes the final failure after retries are exhausted. A CheckpointIOError in Risk.triage means 3 retries already failed.

3.6 Post-mortem

At hour 48, Person B runs RiskLog.summary(). Any risk that fired ≥ 3× is a candidate spec bug — update DESIGN.md §14 + this doc in the post-hackathon merge (not during the 48h).

---

4. Data Structures

4.1 RiskEntry — frozen, immutable, append-once at assess().

Serialization: to_dict() / from_dict() round-trip stable. id is the primary key; changing any other field requires a new entry (versioning by file revision, not in-memory mutation). Enforced by frozen=True.

4.2 RiskLog — append-only in-memory list wrapped as immutable view per append.

• Storage format: JSONL (one TriageResult.to_dict() per line).
• Path: logs/risk_log.jsonl within the run directory (DESIGN.md §13 deliverable #6 traces).
• Never truncated during a run. A new run writes a new file (run_id prefix).

4.3 TriggerSignal — string Enum; stable wire format.

Adding a new signal requires (a) a new RiskEntry with that signal OR routing to R99 (unknown), and (b) a new line in the §2.2 enum above. Removing a signal is forbidden during the 48h.

4.4 StopCondition — three-valued sum type (None | ESCALATE | HARD_STOP).

None is the default (most risks have a local mitigation, not an escalation). ESCALATE and HARD_STOP are reserved for the CLAUDE.md §11 set.

---

5. Error Modes

These are the error classes this module itself can raise or encounter — distinct from the risks it catalogs.

Situation	Handling
Unknown-class risk (not in register) — Risk.triage(UNKNOWN) or a signal with no matching entry	Returns TriageResult(entry=R99, escalate=True, log_line="unknown signal: <raw>"). Does NOT raise. Orchestrator is notified via SendMessage. R99 is the catch-all so triage() is total.
Stop-condition met (V100 down > 8h, HF outage > 2h, team drop + gate-miss)	triage().hard_stop == True. Orchestrator halts dispatch. This is not an error in risk.py — it is the correct behavior.
Unresolvable mitigation — owner reports mitigation attempt failed	Owner calls Risk.triage(signal) a second time within 5 min with context="mitigation_failed". If stop_condition is already ESCALATE, upgrade path to HARD_STOP is explicitly a user decision (not automatic).
Duplicate registration — same id appears twice in assess()	AssertionError at import-time test (tests/test_risk_book.py::test_unique_ids). The register is a static data structure; duplicates are a code bug.
design_refs cites a non-existent section	Caught by tests/test_risk_book.py::test_design_refs_valid which greps the referenced files for the cited headings. Missing → test failure → fix before merge.
RiskLog write fails (disk full)	Log to stderr + continue. The run is more important than the log line. This is a silent-log fallback, not a silent-failure of the risk itself.
Risk.triage called before Risk.assess	Safe — assess() returns pure data; triage() internally calls it if not cached. No initialization-order coupling.

Policy: risk.py never crashes the run. If in doubt, route to R99 and escalate.

---

6. Dependencies

6.1 Consumes

• DESIGN.md §14 — the 12-risk table (R01–R12 below).
• CLAUDE.md §11 — escalation + stop conditions (R13–R15 and StopCondition wiring).
• docs/modules/training.md §5 — exception class names mapped to TriggerSignal values for training risks.
• docs/modules/vendors.md §5 — envelope-shape invariant mapped to ERROR_ENVELOPE_VIOLATION.
• docs/modules/audio.md §5 — model-load / voice-pack failures mapped to KOKORO_LOAD_FAILED, WHISPER_LOAD_FAILED, INDIC_VOICE_PACK_MISSING.

6.2 Consumed by

• env.md — catches audio errors; calls Risk.triage for logging. env.py already has local fallbacks (audio.md §5 table) so the triage call is informational.
• training.md — the training callback wiring (training.md §5) maps KLDivergenceExplosion, RewardCollapseError, CheckpointIOError, OutOfMemoryError, LanguageCohortCollapseError, WandBStartupError to triggers. Callback delegates to Risk.triage.
• audio.md — reports startup / missing-pack conditions but owns local mitigations.
• deploy_env_space.md, deploy_demo_space.md — the deploy runbooks reference R06 (ZeroGPU quota), R10 (Docker image size), R11 (openenv validate), R06-STOP (HF Hub/Spaces outage > 2h) for their Day-2 checklists.
• pitch_demo.md — references R06 (ZeroGPU) as the reason for the gradio share=True fallback demo path; references R12 (judge language) for the English-captions design choice.
• evaluation.md — references R13 (Stage 1 convergence) as the gate for final-eval vs retrain-decision.

6.3 Does not depend on

• The agent / LLM — this book is pure environment + process code.
• HF Hub at runtime — assess() reads from local Python data only.
• Any vendor module — vendors report errors to this book, not the other way.

---

7. Edge Cases

Minimum 5 required. Ten here for completeness.

7.1 Compound risk trigger (two risks fire in the same step)

Scenario: During Stage-3 training, at step 247, the V100 briefly disconnects (R01-V100-adjacent) AND train/policy_kl crosses 10.0 (R13 KL catastrophe) within a 30-second window.

Handling: Risk.triage is called twice — once per signal — in whatever order the callback observes them. Both append to RiskLog. The orchestrator sees two HARD_STOP / ESCALATE messages and treats them as independent incidents. RiskLog.summary() post-mortem will show both. We do NOT try to merge compound events into a single "meta-risk" entry — that invites silent de-duplication bugs.

7.2 New risk surfaced mid-event

Scenario: Hour 22 of onsite, Person D discovers the HF Hub deprecated huggingface-cli upload in favor of hf upload and the Dockerfile command silently no-ops. This is not in the register.

Handling: Person D calls Risk.triage(UNKNOWN, context="hf-cli-deprecated"). Returns R99 (unknown-class). Person D messages the team lead. A one-line entry is appended to the doc (not the frozen register) under §11 "Live additions". The frozen register gets updated in the post-hackathon PR — during the 48h we never mutate assess() because test suites pin its length + contents.

7.3 Mitigation fails (first attempt)

Scenario: R01 (V100 FP16 instability) fires. Mitigation: learning_rate 5e-6 → 2e-6, resume from last checkpoint. Person C does that. Step 30 later, grad_norm spikes inf again.

Handling: Person C re-triages with context="mitigation_failed_round_1". R01's stop_condition is None (not originally escalate). But two consecutive mitigation failures promote this to ESCALATE — the promotion is a human decision, encoded in the triage-loop §3.2 step 3 ("elif triage.escalate"). Person C SendMessage team lead with the promoted severity. Never silent, never automatic.

7.4 Team member reassign (A covers B)

Scenario: Person B gets a fever at hour 30. Person A picks up rewards + test plans. R09 (team drop) fires with TEAM_MEMBER_DROP signal.

Handling: R09's mitigation is "roles are additive — Person D covers A+env, Person C covers rewards". But in this specific case Person A takes B's load (A has capacity because env code is already in C-batch). Orchestrator updates the live-ownership mapping in memory (not in CLAUDE.md §2.2 — that is a spec, not a runtime roster). If 3-person coverage cannot make the Phase D gate on time (TEAM_3PERSON_BELOW_GATE signal), R09 promotes to HARD_STOP.

7.5 Mitigation conflicts with deliverable

Scenario: R03 (Hinglish Whisper noise) mitigation: "score R3/R4 on semantic match not exact string". This was a DESIGN.md §14 decision pre-locked. At hour 36, Person B proposes tightening R4 to exact-string to fix a specific reward-hack exploit found in the probe (DESIGN.md §13 #9).

Handling: This is a spec conflict, not a triage event. Person B updates rewards.md AND DESIGN.md §14 AND this doc's R03 entry. Critic-gated change. NOT a Risk.triage call — triage is runtime, the spec conflict is authoring. The rule: mitigation edits go through the doc workflow, not through the runtime log.

7.6 False-positive stop (V100 dropped, came back in 4 min)

Scenario: V100 SSH drops at hour 20 minute 8. Returns at hour 20 minute 12. V100_DOWN fires; V100_DOWN_OVER_8H does NOT fire (not enough elapsed time).

Handling: R01-adjacent V100_DOWN has stop_condition=None — it's a nuisance. RiskLog.append only. Resume work. The HARD_STOP is only on the _OVER_8H variant, which requires a wall-clock threshold. This matches CLAUDE.md §11 literally ("V100 unavailable for > 8h").

7.7 ZeroGPU quota depleted during demo day

Scenario: Saturday midday, demo Space has been hit by spectators; ZeroGPU quota for krrishchoudhary109 is exhausted.

Handling: R07 (HF Space ZeroGPU) trigger fires. Mitigation: fallback to gradio share=True local tunnel. Person D already has $20 A10G reserved — switch to that first (cheaper for real judge eval). If A10G pulls a 5xx, fall to share=True on the local V100 training box (after Stage-3 completes). RiskLog records which path was taken for the blog.

7.8 openenv validate fails on a subtle schema issue

Scenario: openenv validate . reports observation.tool_results[].schema_version type is str but validator expects Literal["v1", "v2", "v3"].

Handling: R11 (openenv validate fails) fires. Mitigation: "Validate early (pre-onsite hour 16 gate)." If we are past that gate and still failing, after 3 fix attempts OPENENV_VALIDATE_FAIL promotes to ESCALATE. Person D + Orchestrator decide whether to (a) update our models.md to use the Literal, (b) ask user for schema-deviation approval, or (c) drop the validator check — never (c) without user approval.

7.9 Reward-hacking spike during Stage 2

Scenario: train/R5_mean drops to -0.32 at step 180 of Stage 2. train/reward_mean stays flat — classic hack indicator. training.md §5 RewardCollapseError fires.

Handling: R05 (reward hacking on R2) trigger. Mitigation chain: (i) training halts, RewardCollapseError raised; (ii) Person B runs reward-hacking probe on last 200 episodes per DESIGN.md §13 #9; (iii) if a new exploit pattern is found, update anti_hack_penalty logic in rewards.py per rewards.md §3.6; (iv) resume from pre-regression checkpoint, NOT the current one. This case is the single most important runtime mitigation in the book — reward hacking propagates through the model weights and cannot be fixed by just resuming.

7.10 Judge doesn't speak Hindi

Scenario: Demo day, the judge hearing our booth is fluent only in English + Mandarin (neither Indic).

Handling: R12 pre-staged mitigation — every audio clip in the deck has English captions, and the demo Gradio UI auto-translates agent replies to English for readability. No runtime trigger; this is a design-time mitigation surfaced at the pitch-script level. RiskLog records which judges the English path fired for (instrumentation lives in the demo, not in this module).

---

8. Examples

Three concrete worked examples, runnable if implemented.

8.1 R01 — V100 FP16 gradient instability → detection + mitigation + rollback

Setup. Hour 6 of onsite. Stage-1 training starts. Fresh V100 box; Unsloth 2026.4.5 pinned; use_bias_correction_kl=True. Everything looks right.

Detection (step 14). Training callback reports grad_norm=inf. NonFiniteGradientError would be raised on the 4th consecutive skip. At step 14, this is the first occurrence.

signal = TriggerSignal.GRAD_NORM_INF
triage = Risk.triage(signal)
# → TriageResult(
#     entry=R01,
#     action="Unsloth 4-bit QLoRA + FP16 autocast; grad clip 1.0; loss-scale monitored every 10 steps; fallback to dtype=float16 explicit",
#     escalate=False, hard_stop=False,
#     log_line="R01 fired (step 14): grad_norm=inf; mitigation=fp16_autocast+gradclip1.0"
#   )
RiskLog.append(triage, ts_iso="2026-04-25T15:22:04+05:30")
# Person C inspects: loss-scale has halved twice in last 50 steps — precursor to underflow.
# Action (from training.md §7a mitigation 5): drop learning_rate 5e-6 → 2e-6, resume from last checkpoint.

Mitigation (step 15 onward). Person C: train(..., learning_rate=2e-6, resume_from=Path("checkpoints/stage1_step_10")). Resume starts cleanly.

Rollback guardrail. Training continues for 20 steps. grad_norm stays finite. train/skipped_updates==0 rolling mean. R01 does NOT fire again.

If it had fired again: Person C would triage with context="mitigation_failed_round_1". Second-round failure promotes R01 to escalation per §3.3; user is paged; we consider downshifting to Gemma 3 4B per CLAUDE.md §11 first-responder case.

8.2 R05 — Reward hacking spike on R2 → probe → reward re-weighting

Setup. Hour 28 of onsite, Stage-2 (single-drift) training. train/R5_mean has floated near 0.0 for 170 steps. Then across steps 171–180, it drops to -0.31.

Detection (step 180). The training callback computes R5_mean_10 (10-step moving mean) = -0.31 AND checks train/hallucinated_field_count simultaneously (training.md §7d) — hallucinations spiked. train/reward_mean is flat (not dropping proportionally). RewardCollapseError is raised.

signal = TriggerSignal.R5_DROP_WITH_HACK_SPIKE
triage = Risk.triage(signal)
# → TriageResult(
#     entry=R05,
#     action="R2 requires specific field-name OR correct follow-up call; R5 penalizes bare assertions. "
#            "Halt training; surface to Person B for probe inspection (training.md §7d).",
#     escalate=True, hard_stop=False,
#     log_line="R05 fired (step 180): R5_mean_10=-0.31; reward_mean flat; halt + probe"
#   )

Probe (hour 28 + 30 min). Person B runs python3 eval/reward_hack_probe.py --episodes-from step_170_to_180.jsonl --top-k-suspicious 20. Output reveals: the agent learned to emit "drift detected on unknown_field" on EVERY turn, which scores R2 = +0.3 via the substring "drift" token match AND R5 = -0.3 for bare assertion — net 0.0, but the hallucinated R2 credit still feeds advantage variance and the model is "rewarded by gradient" even if the mean reward is unchanged.

Fix (hour 29). rewards.md §3.6 update: tighten R2 to require the drift-pattern's canonical error_code token (from vendors.md §5.2) as substring, not a generic "drift" token. Bump config_sha256.

Resume (hour 30). Person C resumes from the pre-regression checkpoint checkpoints/stage2_step_160 — NOT 170 — because weights from 160 onward already encoded the exploit pattern in latent gradient directions. Stage-2 restarts with updated rewards. Runs clean for 20 steps; R05 does not re-fire.

Book-keeping. The reward-hacking probe result is appended to reward_hacks.jsonl (DESIGN.md §13 deliverable #9). The blog post cites this incident as "one of three exploits caught by the probe, all fixed without silent re-weighting".

8.3 Stop-condition trigger — V100 down ≥ 8h

Setup. Hour 14 of onsite. V100 box drops SSH at hour 14:02. Root cause: data-center UPS maintenance neither DGX ops nor the hackathon organizers flagged.

Detection (continuous). Orchestrator watchdog ssh -o ConnectTimeout=5 v100-box echo ok fails. Signal V100_DOWN fires at hour 14:02.

triage = Risk.triage(TriggerSignal.V100_DOWN)
# → R01-adjacent entry, non-stop, log_only.
# RiskLog.append. Continue non-training work (Person B on test plans, Person D on Dockerfile).

Escalation (hour 14:02 → hour 22:02). Watchdog flips every 2 min. At wall-clock 8 hours elapsed without successful SSH, signal V100_DOWN_OVER_8H fires. CLAUDE.md §11 says: hard stop.

triage = Risk.triage(TriggerSignal.V100_DOWN_OVER_8H)
# → TriageResult(
#     entry=R01-STOP variant (id="R01-STOP"),
#     action="V100 unreachable > 8h — terminate plan; re-plan with user.",
#     escalate=True, hard_stop=True,
#     log_line="R01-STOP fired: V100 down 8h+; HARD_STOP"
#   )
# Orchestrator:
#   - halts all Agent dispatch immediately
#   - SendMessage team-lead: "HARD_STOP R01-STOP: V100 down ≥ 8h. Plan must be re-scoped with user."
#   - awaits user decision: (a) cancel onsite, (b) pivot to Colab T4 at reduced scale, (c) submit baseline-only.
# Person C preserves whatever checkpoint is on last-good-rsync to HF Hub.

Resolution. User decides: pivot to Colab T4 for Stage-1 only; skip Stage-3; record the incident in the blog. The blog-post honesty about this is, per DESIGN.md §14 #9-adjacent, an asset not a liability.

---

9. Open Questions

1. Should Risk.triage auto-promote to ESCALATE on second-round mitigation failure, or stay manual? Current spec: manual (§7.3). Rationale: automated promotion has surprised us in prior incidents. But manual requires an attentive operator. DECISION during Batch D3 critic round: stay manual; revisit post-hackathon.

2. Do we include supply-chain risks (PyPI malicious package)? Currently out of scope — we pin every dep in requirements.txt, and pip-audit is a Phase C C1 task. If a malicious package lands during onsite, it becomes R99 (unknown). Not worth a dedicated entry for a 48h event.

3. RiskLog public vs private. We write risk_log.jsonl to the run directory. Do we push it to HF Hub as part of the training traces (DESIGN.md §13 #6)? Pros: full transparency for judges. Cons: the log names V100 incidents which is operationally sensitive. Proposed resolution: push with operator identifiers scrubbed to team / A..D only. Confirm with Person D before Phase C5.

4. R14 threshold specifics. Currently "Stage 1 doesn't converge (R1 < 0.4 at step 100)". Is step 100 the right checkpoint, or should it be step 75? DESIGN.md §14 risk #4 says Stage 1 targets 100 steps total, so step 100 = end-of-stage — already too late to recover in-stage. Proposed: STAGE1_R1_BELOW_0_4_AT_STEP_75 is the actual signal, but we call it "at step 100" in prose to match CLAUDE.md §11. Confirm with Person C during their training callback implementation.

5. Judge language (R12) beyond Hindi. Our mitigation is "English captions on every audio clip." What if the judge is fluent only in Mandarin and English? Captions are still English. Demo voice reply is Hindi. Acceptable? YES — English is the hackathon lingua franca; Mandarin judges on an India-first hackathon is low-probability. Not worth adding a second caption track.

---

10. The Register (data for §2.3 Risk.assess())

Fifteen entries. R01–R12 are DESIGN.md §14 verbatim, augmented with owner + trigger signal. R13–R15 are CLAUDE.md §11 adds.

R01 — V100 FP16 gradient instability (Gemma 4 is BF16-native)

• Probability: Med
• Impact: kills (training)
• Mitigation: Unsloth 4-bit QLoRA + FP16 autocast; max_grad_norm=1.0; loss-scale monitored every 10 steps; fallback to dtype=torch.float16 explicit at FastModel.from_pretrained; if instability persists, learning_rate 5e-6 → 2e-6 and resume from last checkpoint.
• Owner: C (Training & Data)
• Trigger signal: GRAD_NORM_INF (training callback observes grad_norm is inf or NaN; training.md §5 NonFiniteGradientError after 3 consecutive skips).
• Stop condition: None (local mitigation). See R01-STOP for the > 8h variant.
• design_refs: DESIGN.md §14 #1, training.md §5, training.md §7a, §7c.

R01-STOP — V100 unavailable for more than 8 hours

• Probability: Low
• Impact: kills (training AND demo)
• Mitigation: HARD STOP. Preserve last-good checkpoint to HF Hub. Re-plan with user: (a) cancel onsite, (b) pivot to Colab T4 with reduced scope, (c) submit baseline-only.
• Owner: Orchestrator → user
• Trigger signal: V100_DOWN_OVER_8H (wall-clock ≥ 8h since first V100_DOWN event, per CLAUDE.md §11).
• Stop condition: HARD_STOP.
• design_refs: CLAUDE.md §11 (hard-stop #1).

R02 — TRL GRPOTrainer KL catastrophe

• Probability: Med
• Impact: kills (training)
• Mitigation: Pin TRL ≥ 0.23; use_bias_correction_kl=True (invariant asserted in build_grpo_config); beta=0.04; training callback raises KLDivergenceExplosion if policy_kl 10-step mean > 10.0; halt + dump last 20 rollout groups to debug/kl_explosion_dump.jsonl and escalate to user.
• Owner: C (Training & Data)
• Trigger signal: POLICY_KL_OVER_10 (training.md §5 KLDivergenceExplosion).
• Stop condition: ESCALATE_TO_USER (halt training; no silent recovery attempt; root-cause is almost always config-invariant violation).
• design_refs: DESIGN.md §14 #2, training.md §5, training.md §7c, TRL issue #4637.

R03 — Whisper transcription errors on Hinglish code-mixing

• Probability: High
• Impact: med (noisy observation)
• Mitigation: Use faster-whisper-small with language="hi"; accept noise — it's realistic; score R3/R4 on semantic match not exact string. TTS/ASR is ONLY at env boundary (DESIGN.md §9.4), never in training loop — so noisy ASR cannot break gradient. Reflected in DriftCallObservation.last_confidence; agent may CLARIFY to re-prompt.
• Owner: C (Audio subsystem; env-side consumer is A)
• Trigger signal: observable as low TranscriptResult.confidence in observation; not an exception. Does not fire Risk.triage unless confidence < 0.3 on ≥ 30% of episodes over a 50-episode window (indicating systemic degradation, not natural variance).
• Stop condition: None.
• design_refs: DESIGN.md §14 #3, audio.md §5, audio.md §7.1, §7.4, rewards.md (semantic-match specification).

R04 — 200–500 GRPO steps too few for 3-stage curriculum

• Probability: Med
• Impact: med (Stage 3 undertrains)
• Mitigation: Compressed curriculum: Stage 1 → 100, Stage 2 → 200, Stage 3 → 100 (total 400). Prioritize Stage 2 depth. If time budget slips further, drop Stage 3 entirely and ship Stage-2-final as the LoRA — this is still better than untrained.
• Owner: C (Training & Data)
• Trigger signal: not a runtime signal per se; checked at each stage boundary via train/step vs plan. Triggers Risk.triage only if Stage-3 has < 50 steps available at its start.
• Stop condition: None.
• design_refs: DESIGN.md §14 #4, DESIGN.md §10.3, training.md §6.

R05 — Reward hacking on R2 (spam "drift detected!")

• Probability: High
• Impact: high (R2 collapses as a signal)
• Mitigation: R2 requires specific error_code field-name substring OR correct follow-up call (rewards.md §3.6); R5 penalizes bare assertions (-0.3); penalties stack additively to -1.0 floor. Runtime: RewardCollapseError raises on R5_mean_10 ≤ -0.3 AND hallucinated_field_count spike AND reward_mean flat — halt and run reward-hacking probe (DESIGN.md §13 deliverable #9); resume from pre-regression checkpoint, not the current one.
• Owner: B (Rewards & Tests)
• Trigger signal: R5_DROP_WITH_HACK_SPIKE (training.md §5 RewardCollapseError).
• Stop condition: ESCALATE_TO_USER (halt + probe; does not auto-hard-stop, but requires B attention).
• design_refs: DESIGN.md §14 #5, DESIGN.md §7.3, training.md §7d, rewards.md §3.6.

R06 — HF Space ZeroGPU quota exhausted

• Probability: Low
• Impact: med (demo degrades)
• Mitigation: $20 A10G budget reserved for paid fallback; secondary fallback gradio share=True locally from the V100 training box after Stage-3 completes. Pre-generated demo audio means a static deck is the tertiary fallback.
• Owner: D (Deploy & Story)
• Trigger signal: HF Space returns quota-exceeded HTTP; surfaced by deploy-demo-space.md health probe.
• Stop condition: None.
• design_refs: DESIGN.md §14 #6, deploy_demo_space.md §4.

R06-STOP — HF Hub / Spaces outage persisting > 2 hours

• Description: Hugging Face Hub or Spaces serving infrastructure is unreachable for > 2 hours during onsite; env Space / demo Space / Hub model pushes cannot be verified or redeployed.
• P: Low · I: kills
• Mitigation: pause all HF-dependent tasks; fall back to local gradio share=True demo and local-only Docker testing; if not restored by onsite hour −2 (2 hours before pitch), orchestrator declares HARD_STOP and team pivots to pre-recorded pitch video.
• Owner: orchestrator
• Trigger signal: HF_HUB_OUTAGE_OVER_2H
• Stop condition: HARD_STOP
• Design refs: CLAUDE.md §11 Hard-stop #2, DESIGN.md §13 (deployment), deploy_env_space.md §9 OQ3

R07 — Indic Whisper quality too poor for live demo

• Probability: Med
• Impact: med (live demo weak)
• Mitigation: Fallback — English-only briefs for live demo; Indic briefs in recorded video. Decision point is hour 40 (after Stage-3 complete) based on whisper quality measured during final eval.
• Owner: D (Deploy & Story), decision co-signed by C.
• Trigger signal: final-eval Indic-cohort R4 < 0.3 (a proxy for "agent can't parse Indic transcripts reliably").
• Stop condition: None.
• design_refs: DESIGN.md §14 #7, audio.md §1.1, pitch_demo.md.

R08 — Kokoro Indic voice quality insufficient

• Probability: Low
• Impact: med (demo sounds bad)
• Mitigation: Pre-generate ALL demo audio offline with careful voice-pack selection; A/B each generated clip against AI4Bharat reference clips before shipping; if still poor, use AI4Bharat's TTS for demo-only clips (training is text-only so this has no training-side impact).
• Owner: D (Deploy & Story), audio pipeline owned by C.
• Trigger signal: INDIC_VOICE_PACK_MISSING (audio.md §5) OR subjective quality fail at pre-demo rehearsal.
• Stop condition: None.
• design_refs: DESIGN.md §14 #8, audio.md §4.3.1 (voice-pack fallback chain), audio.md §7.2.

R09 — Team member drops / sick

• Probability: Med
• Impact: high (≥ 8h slip)
• Mitigation: Roles are additive by design — Person D covers A + env if A drops; Person C covers rewards if B drops; Person A covers demo if D drops. Plan survives 3-person execution. If 3-person execution cannot hit Phase D gate on time, escalate.
• Owner: Orchestrator (re-planning); affected teammate for hand-off.
• Trigger signal: TEAM_MEMBER_DROP (self-reported via SendMessage to team-lead).
• Stop condition: ESCALATE_TO_USER. Promotes to HARD_STOP only if TEAM_3PERSON_BELOW_GATE also fires.
• design_refs: DESIGN.md §14 #9, CLAUDE.md §11 (hard-stop #3), CLAUDE.md §2.2.

R10 — Env Docker image too large for free CPU tier

• Probability: Low
• Impact: med (env Space fails to deploy)
• Mitigation: Trim Whisper/Kokoro models to int8; use Alpine base (or python:3.11-slim with aggressive layer pruning); target < 2 GB total (audio.md §6.4 already budgets ~450 MB for weights). Measured at docker build via docker images → image size.
• Owner: D (Deploy & Story)
• Trigger signal: DOCKER_IMAGE_OVER_2GB — Dockerfile build prints image size; CI lint fails if > 2 GB.
• Stop condition: None.
• design_refs: DESIGN.md §14 #10, deploy_env_space.md, audio.md §6.4.

R11 — openenv validate fails on our spec

• Probability: Med
• Impact: high (disqualification risk)
• Mitigation: Validate early — pre-onsite hour 16 gate (CLAUDE.md §8 smoke-test checklist expansion). Keep openenv's known-good example envs side-by-side for diffing. After 3 fix attempts without success, escalate to user for schema-deviation approval.
• Owner: D (Deploy & Story), env schema owned by A.
• Trigger signal: OPENENV_VALIDATE_FAIL after 3 fix attempts.
• Stop condition: ESCALATE_TO_USER (CLAUDE.md §11 escalate #2).
• design_refs: DESIGN.md §14 #11, CLAUDE.md §11 (escalate #2), env.md.

R12 — Judge doesn't speak Hindi / Indic, misses the nuance

• Probability: Med
• Impact: med (weaker 30% score component)
• Mitigation: Pitch deck has English captions on every audio clip; demo UI auto-translates agent replies to English; pitch script explicitly calls out schema-drift which is language-agnostic as the primary technical contribution (Indic is the substrate, drift is the innovation).
• Owner: D (Deploy & Story)
• Trigger signal: JUDGE_LANGUAGE_MISMATCH — observed at booth (not automated).
• Stop condition: None (design-time mitigation, not runtime).
• design_refs: DESIGN.md §14 #12, pitch_demo.md, DESIGN.md §15.

R13 — Gemma 3n E2B smoke test fails on V100 (NEW per CLAUDE.md §11)

• Probability: Med
• Impact: kills (block; may need downshift to Gemma 3 4B)
• Mitigation: Pre-onsite smoke test per DESIGN.md §16.A.1 — run BEFORE Batch D1 kickoff (CLAUDE.md §8 checklist). If fails: diagnose via nvidia-smi, CUDA driver version, Unsloth pin. Two fallbacks in priority order: (i) downshift to unsloth/gemma-3-4b-it-bnb-4bit — same GRPO pipeline, known-working on V100; (ii) use unsloth/gemma-2-2b-it-bnb-4bit — smaller, older, very stable. Either downshift requires updating DESIGN.md §0 model ref AND escalation to user (architecture-level decision).
• Owner: C (Training & Data); escalation via Orchestrator.
• Trigger signal: GEMMA4_SMOKE_FAIL.
• Stop condition: ESCALATE_TO_USER (CLAUDE.md §11 escalate #1). Hard-stop only if all three candidates (E2B, 3-4B, 2-2B) fail — then hardware is the problem.
• design_refs: CLAUDE.md §11 (escalate #1), DESIGN.md §16.A.1, training.md §6.3.

R14 — Stage 1 training doesn't converge (NEW per CLAUDE.md §11)

• Probability: Med
• Impact: high (reward/curriculum redesign required; Stage 2/3 blocked)
• Mitigation: Detect at step 100 (end-of-Stage-1): if eval/R1 < 0.4 on the 50-episode held-out set, halt. Diagnose via per-reward breakdown: is R1 flat (tool-use not learned) or R4 flat (format not learned)? Fix candidates: (i) raise learning_rate 5e-6 → 1e-5 for Stage 1 only (Stage 2/3 stay at 5e-6); (ii) over-weight R4 during Stage 1 only (rewards.md temporary override); (iii) extend Stage 1 from 100 → 150 steps by borrowing from Stage 3 budget. All three are reversible; pick one, re-run Stage 1.
• Owner: C (Training & Data), reward-side decisions co-signed by B.
• Trigger signal: STAGE1_R1_BELOW_0_4_AT_100.
• Stop condition: ESCALATE_TO_USER (CLAUDE.md §11 escalate #3).
• design_refs: CLAUDE.md §11 (escalate #3), training.md §6, DESIGN.md §10.3.

R15 — Merge conflict across owned files — ownership violation (NEW per CLAUDE.md §11)

• Probability: Low
• Impact: med (short slip; trust hit)
• Mitigation: First rule — orchestrator resolves merge conflicts, never delegates to an agent (CLAUDE.md §5.4). If a conflict touches both A-owned (env/vendors) and B-owned (rewards/tests) files, the ownership was wrong somewhere. Orchestrator: (i) halt both agents, (ii) identify the overlap file, (iii) reassign that file to a single owner, (iv) update CLAUDE.md §2.2 if the reassignment is permanent, (v) restart the affected batch.
• Owner: Orchestrator.
• Trigger signal: MERGE_CONFLICT_CROSS_OWNER.
• Stop condition: ESCALATE_TO_USER (process-level; CLAUDE.md §11 escalate #5). Never hard-stop — this is a process bug, not a technical one.
• design_refs: CLAUDE.md §11 (escalate #5), CLAUDE.md §5.4, CLAUDE.md §2.2.

R99 — Unknown-class risk (not in register)

• Probability: Low (by construction — the register aims to be exhaustive)
• Impact: unknown (treated as high until classified)
• Mitigation: Risk.triage(UNKNOWN) routes here. Log line to risk_log.jsonl; SendMessage team-lead; human classifies within 15 min: (a) add a new entry to this doc (post-hackathon PR) and use R99's mitigation template for now; (b) map to an existing entry if the signal was mislabeled.
• Owner: Orchestrator → team lead.
• Trigger signal: UNKNOWN or any signal not matched by R01–R15.
• Stop condition: ESCALATE_TO_USER (default-pessimistic).
• design_refs: CLAUDE.md §11 (catch-all), §7.2 of this doc.

---

11. Live additions (populated during the 48h)

Append-only notes about risks discovered after hour 0 of onsite. These go here NOT into Risk.assess() because the register's length is pinned in tests. Post-hackathon PR will promote these into numbered entries.

(empty at v1.0)

---

12. Cross-doc consistency table

Risk	DESIGN.md §14 #	CLAUDE.md §11	training.md §5	vendors.md §5	audio.md §5	rewards.md
R01	#1	—	NonFiniteGradientError	—	—	—
R01-STOP	—	hard-stop #1	—	—	—	—
R02	#2	—	KLDivergenceExplosion	—	—	—
R03	#3	—	—	—	confidence field	§3.6 semantic match
R04	#4	—	—	—	—	—
R05	#5	—	RewardCollapseError	—	—	§3.6
R06	#6	—	—	—	—	—
R06-STOP	—	hard-stop #2	—	—	—	—
R07	#7	—	—	—	—	—
R08	#8	—	—	—	INDIC_VOICE_PACK_MISSING	—
R09	#9	hard-stop #3	—	—	—	—
R10	#10	—	—	—	§6.4 budget	—
R11	#11	escalate #2	—	—	—	—
R12	#12	—	—	—	—	—
R13	—	escalate #1	—	—	—	—
R14	—	escalate #3	—	—	—	§3 (curriculum impact)
R15	—	escalate #5	—	—	—	—
R99	—	catch-all	—	—	—	—

Every cell in this table corresponds to an actual citation in the source doc. If a cell is filled but the cited section does not actually describe that risk → spec-drift bug → fix the cite.