Spaces:

saumilyajj
/

driftcall

Sleeping

File size: 48,447 Bytes

f2df60e

# datasets — Four-Layer Dataset Strategy + HF Hub Publication

**Module path:** `driftcall/data/` (loaders) + `data/` (on-disk artifacts)
**Owner:** Person C (Training & Data)
**Implements:** DESIGN.md §8 (Dataset Strategy — §§8.1, 8.2, 8.3, 8.4, 8.5, 8.6)
**Consumed by:** `driftcall/task_generator.py` (L1), `driftcall/drift_injector.py` (L2 drift patterns), `driftcall/vendors/*.py` (L2 API schemas), `driftcall/audio/*.py` (L3 audio), `training/train_grpo.py` (L4 SFT warmup).
**Status:** Design spec — no code yet.

---

## 1. Purpose

`datasets` is the **authoring-and-loading contract** for every piece of static data DriftCall depends on. It is *not* a training-data-pipeline module (that is `training.md`) and it does *not* compose rewards (that is `rewards.md`). It does exactly four things:

1. **Defines** the four dataset layers per DESIGN.md §8.2 — task-brief templates, vendor API schemas + drift patterns, voice audio, and the optional SFT warmup corpus — as on-disk files with frozen YAML/JSON schemas.
2. **Loads** each file through a deterministic, lazy, singleton loader that NFC-normalizes all Indic strings at load time (cross-references `docs/modules/task_generator.md` §3.4 invariant #8).
3. **Validates** each file at load time: schema shape, type constraints, license header presence, train/val leakage, and consistency cross-references (e.g., every `drift_slot_tags` token in templates.yaml is targetable by ≥ 1 pattern in drift_patterns.yaml).
4. **Publishes** the public-facing bundle `<team>/driftcall-indic-briefs` to the Hugging Face Hub per DESIGN.md §8.6, packaged from a deterministic `enumerate_variants()` walk (see `docs/modules/task_generator.md` §2.2).

**No file in `data/` is ever written at runtime.** All four layers are authored before Phase C ships. Runtime only reads. The only exception is the dataset-packaging script (`training/data_export.py`) which writes `train/briefs.jsonl` + `val/briefs.jsonl` once, pre-publication.

Supervision for GRPO comes from the 5 reward functions (DESIGN.md §8.1) — these files exist to parameterize the environment, not to teach the policy. L4 (SFT warmup) is optional per DESIGN.md §8.2 row 4.

---

## 2. Interface

### 2.1 Directory layout (on disk, shipped inside the env Docker image per DESIGN.md §11.1)

```
data/
├── task_briefs/
│   ├── templates.yaml                # L1 — hand-authored + procedural expansion source (§8.3)
│   └── i18n.yaml                     # L1 — Indic localized strings (cities, weekdays, dish names)
├── drift_patterns/
│   └── drifts.yaml                   # L2 — 20 drift patterns (§6.3, §8.2 row 2)
├── api_schemas/                      # L2 — frozen JSON Schema per vendor per version
│   ├── airline/
│   │   ├── v1.json
│   │   ├── v2.json
│   │   └── v3.json
│   ├── cab/
│   │   ├── v1.json
│   │   ├── v2.json
│   │   └── v3.json
│   ├── restaurant/
│   │   ├── v1.json
│   │   ├── v2.json
│   │   └── v3.json
│   ├── hotel/
│   │   ├── v1.json
│   │   ├── v2.json
│   │   └── v3.json
│   └── payment/
│       ├── v1.json
│       └── v2.json
├── audio/                            # L3 — synthesized + real voice clips (§8.2 row 3, §9)
│   ├── synth/                        # Kokoro-82M output, generated lazily; gitignored
│   │   └── .gitkeep
│   ├── real/                         # AI4Bharat IndicVoices-R held-out subset for pitch demo
│   │   └── MANIFEST.jsonl            # (utterance_id, path, language, license, sha256)
│   └── LICENSES.md                   # per-clip license attribution
└── sft_warmup/                       # L4 — optional Sarvam-M synthesized trajectories (§8.2 row 4)
    ├── trajectories.jsonl            # 200–500 correct rollouts
    └── LICENSES.md
```

**Publication structure** (HF Hub dataset repo `<team>/driftcall-indic-briefs`, DESIGN.md §8.6):

```
driftcall-indic-briefs/
├── README.md                         # model card — provenance, license, stats, reward caveats
├── train/briefs.jsonl                # 15,000 sampled episodes (seed, stage, language_weights, GoalSpec)
├── val/briefs.jsonl                  #    500 held-out episodes — seeds disjoint from train
├── drift_patterns.yaml               # exact copy of data/drift_patterns/drifts.yaml (20 patterns)
├── api_schemas/                      # exact copy of data/api_schemas/
└── LICENSE                           # bundle license (Apache 2.0 by default; see §3.4)
```

### 2.2 Per-file contracts

| File | Format | Authored by | Runtime writer | Schema anchor |
|---|---|---|---|---|
| `data/task_briefs/templates.yaml` | YAML | Hand (20 seeds) | none | `Template` (§4.1 task_generator.md) |
| `data/task_briefs/i18n.yaml` | YAML | Hand | none | `Mapping[LanguageCode, Mapping[str, str]]` |
| `data/drift_patterns/drifts.yaml` | YAML | Hand | none | `DriftPattern` (§4.2 drift_injector.md) |
| `data/api_schemas/<domain>/v<N>.json` | JSON Schema 2020-12 | Hand | none | `APISchema` (§4.4 below) |
| `data/audio/real/MANIFEST.jsonl` | JSONL | Hand (curated from IndicVoices-R) | none | `AudioClipManifest` (§4.5) |
| `data/audio/synth/*.wav` | WAV 16kHz mono | `audio/tts_kokoro.py` (lazy) | `audio/tts_kokoro.py` | n/a — generated |
| `data/sft_warmup/trajectories.jsonl` | JSONL | Sarvam-M via HF Inference (offline) | `training/sft_generator.py` (one-shot) | `SFTTrajectory` (§4.6) |

### 2.3 Loaders (all return frozen dataclasses, all NFC-normalize Indic strings)

```python
from __future__ import annotations
from pathlib import Path
from driftcall.data.models import (
    TemplateLibrary, I18nLibrary,
    DriftPatternLibrary, APISchemaRegistry,
    AudioManifest, SFTCorpus,
)

# L1 — task briefs
def load_templates(path: Path | str = "data/task_briefs/templates.yaml") -> TemplateLibrary: ...
def load_i18n(path: Path | str = "data/task_briefs/i18n.yaml") -> I18nLibrary: ...

# L2 — drift patterns + api schemas
def load_drift_patterns(path: Path | str = "data/drift_patterns/drifts.yaml") -> DriftPatternLibrary: ...
def load_api_schemas(root: Path | str = "data/api_schemas") -> APISchemaRegistry: ...

# L3 — audio manifest (paths + licenses only; actual WAVs resolved on-demand)
def load_audio_manifest(path: Path | str = "data/audio/real/MANIFEST.jsonl") -> AudioManifest: ...

# L4 — optional SFT warmup
def load_sft_corpus(path: Path | str = "data/sft_warmup/trajectories.jsonl") -> SFTCorpus: ...
```

Each loader is implemented as a **module-level lazy singleton** — the first call reads + validates + freezes; subsequent calls return the same instance. Not thread-safe for write (there is no write); safe for concurrent read.

### 2.4 HF Hub publication commands

Packaging runs *once*, pre-event. The script is `training/data_export.py` (see `docs/modules/training.md` for its interface — this module only defines the on-disk shape of what it writes).

**Immutability.** The published bundle is IMMUTABLE after publication. Re-running `hf upload` against the same `data/publication/` tree produces a byte-identical bundle (invariant #6). Adding rows to `val/briefs.jsonl` requires a MINOR-version bump (v1.1, v1.2, …) and a new publication seed; the `train/` split NEVER silently mutates between versions — a version bump either adds disjoint val rows or re-publishes train+val together, never partial mutation of train.

**Seed selection (deterministic, locked).** Train and val seeds are drawn by `training/data_export.py` using these two exact expressions:

```python
import random
# Train: 15,000 seeds sampled without replacement from [0, 20_000_000).
train_seeds = random.Random(20260425).sample(range(0, 20_000_000), 15_000)
# Val: deterministic slice of 500 contiguous seeds in the reserved range.
val_seeds = list(range(20_000_000, 20_000_500))
```

Both lists are byte-identical across re-runs. The publication meta-seed `20260425` is locked; changing it requires a major-version bump and a new repo name or subfolder.

```bash
# Generate the sampled briefs by walking enumerate_variants() (see task_generator.md §2.2)
python3 training/data_export.py \
    --out-train data/publication/train/briefs.jsonl \
    --out-val   data/publication/val/briefs.jsonl \
    --n-train   15000 \
    --n-val     500 \
    --seed      20260425        # frozen publication seed; NOT a training seed

# Copy the static L2 artifacts verbatim
cp  data/drift_patterns/drifts.yaml  data/publication/drift_patterns.yaml
cp -r  data/api_schemas              data/publication/api_schemas

# Upload (see DRIFTCALL/CLAUDE.md §6 command table and huggingface-skills:hf-cli).
# NOTE: `hf` is the modern CLI replacing the deprecated `huggingface-cli`.
hf upload <org>/driftcall-indic-briefs \
    data/publication/ . \
    --repo-type dataset \
    --hf-org <org> \
    --commit-message "v1.0 publication — locked 2026-04-25"
```

The publication seed `20260425` is fixed and recorded in the README. Re-running the script produces a byte-identical bundle (determinism contract inherited from `task_generator.generate` per `docs/modules/task_generator.md` §3.1).

> **Doc-sync flag:** `DRIFTCALL/CLAUDE.md` §6 still lists the deprecated `huggingface-cli upload` command; update that table to `hf upload` in the same PR that lands this doc (captured as Open Question #1 / CLAUDE.md sync item).

---

## 3. Behavior Spec

### 3.1 Authoring conventions

**NFC normalization.** Every string value in every YAML/JSON file is NFC-normalized before it is committed. The loaders re-normalize defensively at load time (invariant #8, `docs/modules/task_generator.md` §3.4). Editors used during authoring (VS Code, vim) must be configured to save NFC — a pre-commit hook (`ruff`-adjacent script) runs `python -c "import unicodedata, sys; ..."` to reject NFD commits.

**License headers.** Every hand-authored file begins with a YAML comment block declaring SPDX identifier, author, year, and upstream attribution if the content is derived from a public dataset (§8.5):

```yaml
# SPDX-License-Identifier: Apache-2.0
# Copyright 2026 DriftCall Team
# Derived-from: AmazonScience/MASSIVE (intent taxonomy, Apache-2.0)
# See data/LICENSES.md for full attribution chain.
```

JSON files carry the same metadata in a `$comment` field at root (JSON Schema 2020-12 permits `$comment` per RFC 7159 conventions).

**Seed determinism.** Every numeric or stochastic sampling decision in template/drift authoring threads through a fixed seed: the publication seed `20260425`, the template-expansion seed `42`, or the curriculum-language seeds declared in DESIGN.md §10.3. No wall-clock, no `random.random()`, no host-machine entropy.

**No PII.** Authored strings never contain real names, phone numbers, email addresses, booking reference numbers, card PANs, or IP addresses. The `from` / `to` fields use IATA codes; the `pickup` / `drop` fields use fictional neighborhood landmarks. A CI lint (`grep -En '[0-9]{10}' data/`) runs before every commit and fails on any 10-digit run outside the allowed IATA/timestamp contexts.

**Eval-set held out from training.** The 500-episode val set uses seeds drawn from a reserved range (seed ∈ `[20_000_000, 20_000_500)`); training always draws seeds from `[0, 20_000_000)`. The publication script asserts disjointness at write time (see §5 leak detection). The exact seed-selection expressions are specified in §2.4.

**Canonical JSON key ordering.** Every row in `train/briefs.jsonl` and `val/briefs.jsonl` is serialized with:

```python
json.dumps(row, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
```

This is enforced as an invariant precondition for byte-identical re-runs (see §3.5 invariant #6). `ensure_ascii=False` preserves Devanagari / Tamil / Kannada script without `\uXXXX` escaping; `sort_keys=True` canonicalizes key order; `separators=(",", ":")` eliminates whitespace variance across Python/libc versions.

**Per-row data lineage.** Every `BriefRow` carries the full six-tuple `(template_id, seed, stage, language, domain, generator_version)` plus three corpus-version hashes (`catalogue_hash`, `templates_sha256`, `i18n_sha256`). This is enforced as an invariant (§3.5 invariant #9) so that any published row is re-derivable from the triple `(seed, stage, library@hash)` alone.

### 3.2 Lazy singleton loaders

```python
# sketch of the module-level pattern, mirrored in every loader
_LIBRARY: TemplateLibrary | None = None
_LIBRARY_LOCK = threading.Lock()

def load_templates(path: Path | str = "data/task_briefs/templates.yaml") -> TemplateLibrary:
    global _LIBRARY
    if _LIBRARY is None:
        with _LIBRARY_LOCK:
            if _LIBRARY is None:
                _LIBRARY = _load_and_validate_templates(Path(path))
    return _LIBRARY
```

The singleton is **path-keyed** — if a test passes a different `path`, a fresh instance is built (still cached in a per-path dict). Production callers always use the default path.

### 3.3 Schema validation at load time

Each loader does three passes:

1. **YAML/JSON parse.** Failure → `MalformedYAMLError` / `MalformedJSONError` with line/column.
2. **Type + shape validation** against the dataclass schema in §4. Failure → `DatasetSchemaError` naming the offending key.
3. **Cross-file consistency** check (loader-specific):
   - `load_drift_patterns` asserts `pattern.id` values are unique, exactly 20 patterns total, `drift_type ∈ {schema,policy,tnc,pricing,auth}`, and every `from_version`/`to_version` references an existing schema file in `data/api_schemas/<domain>/`.
   - `load_templates` asserts every `drift_slot_tags` token is matched by ≥ 1 `DriftPattern.mutation` key or value (`airline.total_fare_inr` must be targetable, else why tag it).
   - `load_api_schemas` asserts each `v<N>.json` validates as JSON Schema 2020-12 against the meta-schema via `jsonschema.Draft202012Validator.check_schema`.
   - `load_audio_manifest` asserts every referenced `path` exists on disk and its sha256 matches the recorded hash.

Failures here abort env startup; HTTP 503 is served until the data is fixed (mirrors `DriftCatalogueError` handling in `docs/modules/drift_injector.md` §5).

### 3.4 License compatibility check

Per §8.5 the public datasets we reference carry mixed licenses:

| Upstream | License | Redistributable in our bundle? |
|---|---|---|
| AI4Bharat IndicVoices-R | Apache-2.0 | Yes, with attribution |
| MASSIVE (Amazon) | Apache-2.0 | Yes, with attribution |
| Schema-Guided Dialogue (SGD) | CC-BY-SA | Inspiration only — derived schema patterns, not verbatim rows |
| MTOP (Facebook) | MIT-style (see original repo) | Inspiration only — derived Hindi task phrasings, not verbatim rows |
| APIs.guru | CC0 | Yes, no attribution required but recorded |

The bundle license (`LICENSE` at the root of `<team>/driftcall-indic-briefs`) is **Apache-2.0**. Because CC-BY-SA is copyleft-adjacent, we never copy SGD or MTOP rows verbatim — only *inspiration* (intent labels, schema shapes). A CI check enforces that no string in `train/briefs.jsonl` or `val/briefs.jsonl` appears verbatim (≥ 10-token suffix match) in a cached SGD/MTOP export. See §5 for the error mode and §7 edge case #3 for the exact detection rule.

**Full verbatim license text (MANDATORY).** The root `LICENSE` file MUST contain the **full verbatim Apache 2.0 license text** as published at https://www.apache.org/licenses/LICENSE-2.0.txt — NOT a URL, NOT a one-line SPDX identifier, NOT a summary. The same requirement applies to `data/audio/LICENSES.md` and `data/sft_warmup/LICENSES.md` (both must embed the full Apache-2.0 text plus per-clip / per-trajectory attribution rows). CI check `tests/data/test_license_text.py` verifies that the byte length of each `LICENSE` file is ≥ 11,000 (Apache-2.0 full text is ~11,357 bytes) and that the canonical "Apache License / Version 2.0, January 2004" header string is present.

**`LICENSES.md` schema (L3 audio + L4 SFT warmup).** Both `data/audio/LICENSES.md` and `data/sft_warmup/LICENSES.md` follow the same markdown format:

1. A preamble (5–15 lines) naming the bundle and linking back to the root `LICENSE`.
2. The full verbatim Apache-2.0 text (as above).
3. A single markdown table with exactly these columns, one row per clip (L3) or per trajectory (L4):

```markdown
| utterance_id | upstream_source      | upstream_license | attribution_required | notes                         |
|--------------|----------------------|------------------|----------------------|-------------------------------|
| iv_r_kn_0451 | IndicVoices-R        | Apache-2.0       | yes                  | speaker consent verified      |
| sft_00042    | Sarvam-M (synthesis) | Apache-2.0       | no                   | rollout seed 42, stage 2      |
```

For L4 the `utterance_id` column is replaced by `trajectory_id` but the other four columns are identical. Loaders do not parse these tables at runtime; they are human-audit artifacts enforced only by pre-commit schema check `scripts/check_licenses_md.py`.

### 3.5 Invariants (enforced by tests)

1. Every string value in every loaded library is NFC (`unicodedata.is_normalized("NFC", s) == True`).
2. `load_drift_patterns()` returns exactly 20 patterns (matches `docs/modules/drift_injector.md` §4.4 and DESIGN.md §6.3).
3. `load_api_schemas()` returns exactly `{airline:v1,v2,v3 + cab:v1,v2,v3 + restaurant:v1,v2,v3 + hotel:v1,v2,v3 + payment:v1,v2}` = **14 schemas across 5 domains** (matches DESIGN.md §8.6 bundle enumeration and §5 vendor catalogue).
4. `load_templates()` library satisfies: every template has ≥ 1 variant in every `LanguageCode` (`hi`, `ta`, `kn`, `en`, `hinglish`); every **primary-domain** pattern's `mutation` field set is a subset of the union of `drift_slot_tags` across that domain's templates. The two transversal payment-auth patterns (`payment.auth_scope_upgrade`, `payment.mfa_required`) are EXEMPT from this subset check — they mutate shared payment fields (`token`, `scope`, `mfa_code`) that are intentionally not present in primary-domain goal templates and therefore cannot appear in `drift_slot_tags`.
5. Publication invariant: train seed set ∩ val seed set = ∅.
6. Publication invariant: running `data_export.py` twice with the same seed produces byte-identical `train/briefs.jsonl` + `val/briefs.jsonl` (SHA-256 match). Enforced via canonical JSON dump (§3.1): `json.dumps(row, ensure_ascii=False, sort_keys=True, separators=(",", ":"))`.
7. Every file in `data/` begins with an SPDX license header (YAML comment or JSON `$comment`).
8. No 10-digit digit-run in any authored string outside the timestamp / IATA allowed contexts (PII guard).
9. **Per-row data lineage.** Every `BriefRow` (§4.7) in the published `train/` and `val/` splits carries all of: `template_id`, `seed`, `stage`, `language`, `domain`, `generator_version`, `catalogue_hash`, `templates_sha256`, `i18n_sha256`. At eval-load time, `catalogue_hash` / `templates_sha256` / `i18n_sha256` must match the currently-loaded library hashes, else `CatalogueHashMismatchError` is raised (§5).
10. **Bundle immutability.** After publication (§2.4), the train split SHA-256 MUST match across all future re-runs of `hf upload`; adding val rows requires a minor-version bump, never a silent train-split mutation.

---

## 4. Data Structures

All types are frozen dataclasses, immutable after load. Mappings are wrapped in `types.MappingProxyType`.

### 4.1 `TemplateLibrary` (re-exported from `task_generator.models` — single source of truth)

```python
@dataclass(frozen=True)
class TemplateLibrary:
    templates: tuple[Template, ...]                                      # exactly 20 at v1.0
                                                                         # (4 domains × 5 templates);
                                                                         # ≥ 20 after minor-version bumps
    cities_by_domain: Mapping[Domain, tuple[str, ...]]                   # 10 per domain
    i18n: Mapping[LanguageCode, Mapping[str, str]]                       # merged from i18n.yaml
    source_sha256: str                                                   # hash of templates.yaml bytes
```

The `templates` tuple length is **exactly 20** at v1.0 publication (4 domains × 5 templates per domain). Post-v1.0 minor-version bumps may grow this count monotonically; the invariant `len(templates) >= 20` and `len(templates) % 5 == 0` holds across all future versions. `load_templates` asserts `len(templates) == 20` at v1.0 via the `generator_version` check.

Authoritative schema lives in `docs/modules/task_generator.md` §4. This module re-exports the type so callers of `load_templates` receive the same object that `task_generator.generate` consumes.

### 4.2 `I18nLibrary`

```python
@dataclass(frozen=True)
class I18nLibrary:
    strings: Mapping[LanguageCode, Mapping[str, str]]
    # e.g., strings["hi"]["BLR"] = "बेंगलुरु"
    # strings["ta"]["Monday"] = "திங்கள்"
    source_sha256: str
```

Merged into `TemplateLibrary.i18n` by `load_templates`, but exposed independently for pure-i18n use cases (e.g., the Gradio demo UI localizing labels).

### 4.3 `DriftPatternLibrary`

```python
@dataclass(frozen=True)
class DriftPatternLibrary:
    patterns: Mapping[str, DriftPattern]                                 # keyed by DriftPattern.id
    by_domain: Mapping[str, tuple[str, ...]]                             # domain → pattern_ids
    by_type:   Mapping[str, tuple[str, ...]]                             # drift_type → pattern_ids
    source_sha256: str
```

`DriftPattern` itself is defined in `docs/modules/drift_injector.md` §4.2 (see the `DriftPattern` dataclass snippet). This module owns *loading*, `drift_injector` owns *applying*.

### 4.4 `APISchemaRegistry`

```python
@dataclass(frozen=True)
class APISchema:
    domain: str                       # "airline" | "cab" | "restaurant" | "hotel" | "payment"
    version: str                      # "v1" | "v2" | "v3"
    schema: Mapping[str, Any]         # parsed JSON Schema 2020-12 document
    source_sha256: str

@dataclass(frozen=True)
class APISchemaRegistry:
    schemas: Mapping[str, Mapping[str, APISchema]]
    # schemas["airline"]["v2"] = APISchema(...)

    def get(self, domain: str, version: str) -> APISchema: ...
    def versions(self, domain: str) -> tuple[str, ...]: ...              # ordered v1,v2,v3
```

Each `v<N>.json` is a valid JSON Schema 2020-12 document describing the tool-response shape for that domain at that drift version. Vendors (DESIGN.md §5) validate outgoing responses against these schemas at test time; the injector (`docs/modules/drift_injector.md` §3) consults version transitions via these files.

### 4.5 `AudioManifest`

```python
@dataclass(frozen=True)
class AudioClip:
    utterance_id: str                 # stable; matches a curated IndicVoices-R clip id
    path: Path                        # relative to data/audio/
    language: LanguageCode
    source: Literal["real_indicvoices_r"]   # manifest is authored-only; synth clips
                                            # are lazily generated and NEVER recorded here
    license: str                      # SPDX identifier
    sha256: str
    duration_s: float                 # ≤ 20.0 (DESIGN.md §9 upper bound)

@dataclass(frozen=True)
class AudioManifest:
    clips: tuple[AudioClip, ...]
    source_sha256: str                # hash of MANIFEST.jsonl bytes
```

The `source` field is a single-value `Literal` — the manifest is **authored-only**. Synth clips generated on-demand by `audio/tts_kokoro.py` are **never** recorded in the manifest (they are transient, gitignored under `data/audio/synth/`). This keeps the manifest auditable and its SHA-256 stable across Kokoro model-weight updates.

### 4.6 `SFTCorpus` (L4, optional)

```python
@dataclass(frozen=True)
class SFTTrajectory:
    episode_id: int
    goal_seed: int                    # same seed space as train/; NEVER a val seed (§3.1)
    turns: tuple[Mapping[str, Any], ...]   # role/content pairs, JSON-serializable
    stage: Literal[1, 2, 3]
    reward_breakdown: Mapping[str, float]  # R1..R5 + total, from the env at synthesis time
    generation_batch_id: str          # uuid4 per invocation of sft_generator.py
    generation_index: int             # monotonic within a batch, 0..N-1

@dataclass(frozen=True)
class SFTCorpus:
    trajectories: tuple[SFTTrajectory, ...]
    generator: Literal["sarvam-m-hf-inference"]
    generation_seed: int
    target_count: int                 # from --target-count CLI flag
    source_sha256: str
```

Consumed by `training/train_grpo.py` only when `--sft-warmup-steps > 0` is passed. Absent by default; loader raises a non-fatal warning if the file is missing (training proceeds without warmup).

**Atomic append + restart recovery (`training/sft_generator.py`):**

- Each trajectory is appended to `data/sft_warmup/trajectories.jsonl` as a single canonical-JSON line followed by `os.fsync(fd)` on the file descriptor, ensuring durability before the next Sarvam-M API call. Partial-write recovery is therefore line-granular.
- Every row carries `generation_batch_id` (uuid4, generated once per invocation of `sft_generator.py`) and `generation_index` (monotonic integer 0..N-1 within that batch).
- On restart, `sft_generator.py` reads the existing `trajectories.jsonl`, reconstructs `(seed, generation_index)` pairs already completed, and resumes from the next uncompleted seed in its deterministic seed list. This tolerates Sarvam-M rate-limit drops, OOM kills, and SIGKILL.
- After all generation completes, the script performs a **final count validation**: if `len(trajectories) != target_count`, it raises `PartialSFTCorpusError` (§5). The loader `load_sft_corpus` also performs this check at load time and raises the same error if the on-disk row count does not match the `target_count` field.
- Edge case #11 (§7) walks through a concrete partial-generation-recovery scenario.

### 4.7 `BriefRow` — canonical publication-row contract

Every line of `train/briefs.jsonl` and `val/briefs.jsonl` in the published HF Hub bundle is exactly one serialized `BriefRow`. This dataclass is the single-source-of-truth schema for everything an offline consumer (a judge re-running eval, a third party reproducing our experiments) needs to re-derive the episode from `(seed, library@hash)` alone.

```python
from __future__ import annotations
from dataclasses import dataclass
from typing import Literal
from driftcall.models import GoalSpec, DriftEvent, LanguageCode

@dataclass(frozen=True)
class BriefRow:
    episode_id: str                    # deterministic from seed + stage (e.g. "s2_ep_00000042")
    seed: int                          # original episode seed (train: [0, 20_000_000),
                                       #                         val:   [20_000_000, 20_000_500))
    stage: Literal[1, 2, 3]            # curriculum stage at publication time
    language: LanguageCode             # "hi" | "ta" | "kn" | "en" | "hinglish"
    domain: Literal["airline", "cab", "restaurant", "hotel"]
    template_id: str                   # e.g. "airline.book.budget_timewindow"
    goal: GoalSpec                     # full GoalSpec (slots + constraints + seed_utterance)
    drift_schedule: tuple[DriftEvent, ...]   # schedule pre-computed by drift_injector
    catalogue_hash: str                # sha256(drift_patterns/drifts.yaml bytes)
    templates_sha256: str              # sha256(task_briefs/templates.yaml bytes)
    i18n_sha256: str                   # sha256(task_briefs/i18n.yaml bytes)
    generator_version: str             # e.g. "driftcall-1.0.0" — semver of the generator
    created_ts_ist: str                # ISO 8601 with +05:30 offset, e.g. "2026-04-25T10:30:00+05:30"
```

Serialization is always canonical: `json.dumps(asdict(row), ensure_ascii=False, sort_keys=True, separators=(",", ":"))`. A concrete JSONL line example is given in §8.5.

At eval-load time, the loader re-hashes the currently-loaded `drifts.yaml` / `templates.yaml` / `i18n.yaml` and compares against `catalogue_hash` / `templates_sha256` / `i18n_sha256`. Any mismatch raises `CatalogueHashMismatchError` (§5) — this prevents silent semantic drift where a consumer runs `train/briefs.jsonl` against a newer catalogue and gets different episodes.

---

## 5. Error Modes

All exceptions subclass `DatasetError(Exception)`. Each is raised exactly once and unit-tested.

| Exception | Trigger | Where raised |
|---|---|---|
| `DatasetFileMissingError` | `data/<path>` absent on disk | every loader |
| `MalformedYAMLError` | YAML parse failure (syntax) | `load_templates`, `load_i18n`, `load_drift_patterns` |
| `MalformedJSONError` | JSON parse failure (syntax) | `load_api_schemas`, `load_audio_manifest`, `load_sft_corpus` |
| `DatasetSchemaError` | type/shape validation failure (missing required key, wrong type, extra unknown key) | every loader |
| `UnknownLanguageKeyError` | a language key ∉ `LanguageCode = {"hi","ta","kn","en","hinglish"}` appears in `templates.yaml` or `i18n.yaml` | `load_templates`, `load_i18n` |
| `LicenseConflictError` | a CC-BY-SA or GPL-licensed row appears in publication bundle while bundle is Apache-2.0; or verbatim ≥ 10-token suffix matches a CC-BY-SA upstream row | publication script (see §3.4) |
| `TrainValLeakError` | train and val seed sets intersect; or an `SFTTrajectory.goal_seed` sits in the val reserved range `[20_000_000, 20_000_500)` | publication script, `load_sft_corpus` |
| `DriftPatternOrphanError` | `drift_patterns.yaml` references a `from_version`/`to_version` not present in `data/api_schemas/<domain>/` | `load_drift_patterns` |
| `ChecksumMismatchError` | `AudioClip.sha256` does not match the on-disk file's hash | `load_audio_manifest` |
| `UnicodeNFDError` | any loaded string fails `unicodedata.is_normalized("NFC", s)` | every loader |
| `PIIDetectedError` | a 10-digit run appears outside allowed contexts in authored text | every text-bearing loader; also CI lint |
| `DuplicateDriftPatternIdError` | two entries in `drifts.yaml` share an `id` | `load_drift_patterns` |
| `CatalogueHashMismatchError` | a `BriefRow` in `train/briefs.jsonl` or `val/briefs.jsonl` carries `catalogue_hash` / `templates_sha256` / `i18n_sha256` that does not match the currently-loaded library (drifts.yaml / templates.yaml / i18n.yaml) hashes | eval-load path (consumers of published bundle) |
| `PartialSFTCorpusError` | `len(SFTCorpus.trajectories) != target_count` at final-count validation; raised by `training/sft_generator.py` post-generation and by `load_sft_corpus` at load time | `load_sft_corpus`, `training/sft_generator.py` |

**No silent fallbacks.** If `data/sft_warmup/trajectories.jsonl` is missing, `load_sft_corpus` raises `DatasetFileMissingError`; the training script is the one that treats this as non-fatal (falls back to no-SFT warmup). Loaders themselves never substitute defaults.

---

## 6. Dependencies

### 6.1 Reads

- `data/task_briefs/templates.yaml`, `data/task_briefs/i18n.yaml`
- `data/drift_patterns/drifts.yaml`
- `data/api_schemas/**/*.json`
- `data/audio/real/MANIFEST.jsonl` + the `.wav` files it references
- `data/sft_warmup/trajectories.jsonl` (optional)

### 6.2 Imports

- `driftcall.models` — `GoalSpec`, `LanguageCode`, `Domain`
- Python stdlib: `json`, `hashlib`, `pathlib`, `unicodedata`, `threading`, `dataclasses`, `typing`, `types`
- Third-party: `PyYAML`, `jsonschema` (for JSON Schema 2020-12 meta-validation)

### 6.3 Consumers

Consuming modules and the exact function they call:

- `docs/modules/task_generator.md` — `load_templates()` in `task_generator.generate()`'s lazy-singleton `_get_library()`.
- `docs/modules/drift_injector.md` — `load_drift_patterns()` in the injector's module-level registry; consults DESIGN.md §6.3 pattern catalogue.
- `docs/modules/vendors.md` — `load_api_schemas()` at vendor import time; each vendor asserts its own response shape against the schema in test fixtures.
- `docs/modules/audio.md` — `load_audio_manifest()` for the pitch demo (§9.5 IndicVoices-R clip playback).
- `docs/modules/training.md` — `load_sft_corpus()` behind `--sft-warmup-steps` flag; also invokes `training/data_export.py` which calls `task_generator.enumerate_variants()` to produce the publication briefs.

### 6.4 Publishes to

- HF Hub dataset repo `<team>/driftcall-indic-briefs` (one-time, pre-event, Phase C5 per `DRIFTCALL/CLAUDE.md` §4.1).

### 6.5 Non-dependencies (explicit)

- Does **not** import from `env.py`, `rewards.py`, `app.py`, or the training entrypoint. Pure data layer.
- Does **not** hit the network at runtime. Every file is local. Publication script is a separate, explicitly-invoked entrypoint.
- Does **not** depend on GPU, CUDA, or PyTorch. CPU-only.

---

## 7. Edge Cases

1. **Missing template variant for a rare language.** `templates.yaml` is authored with `hinglish` + `hi` + `en` + `ta` but an author forgets `kn` for one template. `load_templates` runs per-template check `set(variants.keys()) == LanguageCode.values` and raises `DatasetSchemaError: template 'restaurant.order.veg' missing language_variants['kn']`. The generator's `NoVariantForLanguageError` (task_generator.md §5) never has a chance to fire because loading fails first. Fix: author supplies the missing variant; loader re-runs.

2. **Unicode NFD in author contribution.** A collaborator pastes a Kannada weekday name from macOS clipboard (NFD by default for composed characters). `load_i18n` re-normalizes to NFC *before* equality/hashing; the assertion `unicodedata.is_normalized("NFC", value)` fires post-normalization as a defense against Python/ICU bugs. In practice the round-trip succeeds and NFC is stored. The pre-commit hook separately catches NFD at commit time so CI never sees it.

3. **License incompatibility (CC-BY-SA row smuggled into an Apache-2.0 bundle).** An author, inspired by an SGD row, copies 20 tokens verbatim into a template variant. Publication CI runs a suffix-array check over cached SGD/MTOP exports looking for ≥ 10-token verbatim matches; on hit, `LicenseConflictError("variant in 'airline.book.budget_timewindow' matches SGD row sgd_5432:0 (≥ 10 tokens)")` raises. Fix: rewrite the variant. We keep only *inspiration*, never verbatim text, from CC-BY-SA sources. The threshold (10 tokens) is a pragmatic choice: below that length we treat overlap as incidental linguistic reuse; at or above we flag.

4. **Empty language cohort in a stage mix.** A future curriculum config passes `language_weights = {"en": 1.0, "hi": 0.0, "ta": 0.0, "kn": 0.0, "hinglish": 0.0}`. This is valid at the task-generator level (task_generator.md §3.2 — non-negative weights summing to 1 are legal). `datasets` does not re-validate curriculum config; it only asserts the *library* has variants for all 5 languages. Downstream (`task_generator`) will simply never draw `hi`/`ta`/`kn`/`hinglish`. No error in this module.

5. **Train/val episode-id collision at publication time.** `data_export.py` draws 15,000 seeds for train and 500 for val. If the RNG accidentally maps a train seed into `[20_000_000, 20_000_500)` (the val reserved range) — which cannot happen given the seed-space partitioning in §3.1 — the assertion `train_seeds.isdisjoint(val_seeds)` raises `TrainValLeakError` with the offending seed. Safeguard: train seeds are drawn from `[0, 20_000_000)` and val seeds from `[20_000_000, 20_000_500)`. The two ranges are non-overlapping by construction; the assertion is a defense against future range edits.

6. **Drift-pattern-id orphan (trace references pattern not in YAML).** A test fixture or cached trace references `drift_pattern_id='airline.mysterious_fee'` but `drifts.yaml` has no such entry (it was renamed or removed). `load_drift_patterns` does not look at traces — it only checks internal consistency. The *trace consumer* (`rewards.r2_drift_detection` in `docs/modules/rewards.md`) raises `UnknownDriftPatternError` at scoring time per drift_injector.md §5. If the orphan is discovered during dataset publication, the publication script emits `DriftPatternOrphanError` and aborts.

7. **JSON Schema file that is valid JSON but not valid JSON Schema 2020-12.** `data/api_schemas/cab/v3.json` is hand-edited and accidentally drops the `$schema` keyword or uses an unknown keyword. `load_api_schemas` runs `jsonschema.Draft202012Validator.check_schema(schema)` and on failure raises `DatasetSchemaError("cab/v3.json: not a valid JSON Schema 2020-12: <error>")`. The env refuses to serve `reset()` until fixed.

8. **Audio clip on disk does not match manifest sha256.** `data/audio/real/MANIFEST.jsonl` lists `kn_greeting_03.wav` with `sha256=abc...`. The file gets re-encoded (e.g., by a well-intentioned ffmpeg pass). `load_audio_manifest` re-hashes every referenced WAV and raises `ChecksumMismatchError("kn_greeting_03.wav: expected abc..., got def...")`. Fix: either revert the WAV, or regenerate the manifest after an audit trail commit.

9. **SFT corpus contains a val-reserved seed.** Sarvam-M synthesis inadvertently uses a seed in `[20_000_000, 20_000_500)`. `load_sft_corpus` raises `TrainValLeakError`. The training script may be configured to treat this as fatal (default) or to filter out those trajectories (`--sft-tolerate-leak`); the loader itself always raises.

10. **PyYAML silently deduplicating keys.** If `drifts.yaml` has two entries with the same `id`, the YAML parse is valid but one wins. `load_drift_patterns` builds a set of ids during validation and raises `DuplicateDriftPatternIdError` on collision, with both source line numbers.

11. **Partial SFT corpus recovery (L4 restart).** `training/sft_generator.py` is mid-run at trajectory 137 of a target 300 when the host OOM-kills the process (Sarvam-M inference peak memory). On restart, the script re-opens `data/sft_warmup/trajectories.jsonl`, reads the existing 137 rows (each fsync'd atomically per §4.6), reconstructs the completed `(generation_batch_id, generation_index)` pairs, and resumes from index 137 of the same batch. It does NOT start a new `generation_batch_id` — batch id is rehydrated from the last row. When generation finally reaches 300, the script validates `len(rows) == target_count`; if a Sarvam-M response was silently truncated (say, only 298 rows written), `PartialSFTCorpusError("expected 300, got 298")` is raised and the operator must decide whether to re-run the missing two or ship a corpus with a smaller `target_count`. `load_sft_corpus` performs the same count check at load time.

---

## 8. Examples

### 8.1 Full `templates.yaml` entry for `airline.book.budget_timewindow`

```yaml
# SPDX-License-Identifier: Apache-2.0
# Copyright 2026 DriftCall Team
# Derived-from: AmazonScience/MASSIVE (intent taxonomy, Apache-2.0)

- template_id: airline.book.budget_timewindow
  domain: airline
  intent: book_flight
  min_stage: 1
  required_slots: [from, to, when]
  optional_slots: [seat_pref]
  constraints_template:
    budget_inr:
      distribution: uniform
      low: 3000
      high: 15000
      step: 500
    time_window:
      choices: [morning, afternoon, evening, late_night]
  drift_slot_tags: [price, total_fare_inr]
  # Language keys: ISO short codes matching LanguageCode = Literal["hi","ta","kn","en","hinglish"]
  language_variants:
    hinglish:
      - "Bhai {when} ko {to} jaana hai, cheapest flight {time_window} mein, {budget_inr} rupees max"
      - "{when} ko {from} se {to} ka ticket book kar de, under {budget_inr}, {time_window} ke baad"
    hi:
      - "मुझे {when} को {from} से {to} जाना है, {budget_inr} रुपये से कम में"
    ta:
      - "{when} அன்று {from} லிருந்து {to} க்கு டிக்கெட் வேண்டும், {budget_inr} ரூபாய்க்கு கீழ்"
    kn:
      - "{when} ರಂದು {from} ಇಂದ {to} ಗೆ ಅಗ್ಗದ ವಿಮಾನ ಟಿಕೆಟ್ ಬೇಕು, {budget_inr} ರೂಪಾಯಿಗಳ ಒಳಗೆ"
    en:
      - "Book the cheapest flight from {from} to {to} on {when}, budget under ₹{budget_inr}, departing {time_window}"
```

This is the single source-of-truth entry for the Stage-1 airline booking template; mirror of DESIGN.md §8.3 and `docs/modules/task_generator.md` §4.1.

### 8.2 Full `drift_patterns.yaml` entry for `airline.price_rename`

```yaml
# SPDX-License-Identifier: Apache-2.0
# Copyright 2026 DriftCall Team

- id: airline.price_rename
  drift_type: schema
  domain: airline
  from_version: v1
  to_version: v2
  description: "field 'price' renamed to 'total_fare_inr'; 'currency' removed"
  mutation:
    rename: {price: total_fare_inr}
    remove: [currency]
  detection_hints:
    - "total_fare_inr"
    - "price"
    - "rename"
```

`load_drift_patterns` will (a) parse this, (b) check `id` uniqueness, (c) confirm `from_version=v1` + `to_version=v2` both exist as `data/api_schemas/airline/v1.json` + `data/api_schemas/airline/v2.json`, (d) confirm `detection_hints` is non-empty, (e) wrap `mutation` in `MappingProxyType`. Matches `docs/modules/drift_injector.md` §4.3 byte-for-byte.

### 8.3 `data/api_schemas/airline/v2.json`

```json
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://driftcall.dev/schemas/airline/v2.json",
  "$comment": "SPDX-License-Identifier: Apache-2.0. v2 = post-price_rename drift (DESIGN.md §5.1).",
  "title": "Airline search result (v2)",
  "type": "object",
  "required": ["flight_id", "from", "to", "depart", "total_fare_inr", "seats_left"],
  "additionalProperties": false,
  "properties": {
    "flight_id":       {"type": "string", "pattern": "^[0-9A-Z]{2}-[0-9]{4}$"},
    "from":            {"type": "string", "pattern": "^[A-Z]{3}$"},
    "to":              {"type": "string", "pattern": "^[A-Z]{3}$"},
    "depart":          {"type": "string", "format": "date-time"},
    "total_fare_inr":  {"type": "integer", "minimum": 0},
    "seats_left":      {"type": "integer", "minimum": 0}
  }
}
```

Note that `price` and `currency` from v1 are absent (drift `airline.price_rename` applied). Vendors (`docs/modules/vendors.md`) validate their emitted `airline.search` responses against whichever version the injector has installed in `state.schema_versions['airline']`. This schema also serves as the R2 structural detection surface: a tool call that keys into `price` after drift returns `KeyError` / 422, which is a detection-positive signal per DESIGN.md §7.1 R2.

### 8.4 `MANIFEST.jsonl` row for a curated IndicVoices-R clip (L3)

```json
{"utterance_id": "iv_r_kn_0451", "path": "real/kn/iv_r_kn_0451.wav", "language": "kn", "source": "real_indicvoices_r", "license": "Apache-2.0", "sha256": "b7f1a9c2e5d4...", "duration_s": 4.82}
```

Referenced only by the pitch demo. Training never touches this file — DRIFTCALL/CLAUDE.md §9 "Do not put TTS/ASR in the training loop".

### 8.5 Canonical `BriefRow` JSONL line (single row from `train/briefs.jsonl`)

One line from the published bundle — canonical JSON (sorted keys, no whitespace, UTF-8 preserved for Devanagari):

```json
{"catalogue_hash":"3f9a8e7c2b1d4e5f6a0b9c8d7e6f5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d9e8f","created_ts_ist":"2026-04-25T10:30:00+05:30","domain":"airline","drift_schedule":[{"description":"'price' field renamed to 'total_fare_inr'","domain":"airline","drift_type":"schema","from_version":"v1","pattern_id":"airline.price_rename","to_version":"v2","turn":4}],"episode_id":"s2_ep_00000042","generator_version":"driftcall-1.0.0","goal":{"constraints":{"budget_inr":8000,"time_window":"evening"},"domain":"airline","intent":"book_flight","language":"hinglish","seed_utterance":"Bhai Friday ko Bangalore jaana hai, cheapest flight evening mein, 8000 rupees max","slots":{"from":"HYD","to":"BLR","when":"2026-04-30"}},"i18n_sha256":"a1b2c3d4e5f60718293a4b5c6d7e8f901234567890abcdef1234567890abcdef","language":"hinglish","seed":42,"stage":2,"template_id":"airline.book.budget_timewindow","templates_sha256":"b2c3d4e5f60718293a4b5c6d7e8f901234567890abcdef1234567890abcdef12"}
```

Note: keys are alphabetically sorted (`catalogue_hash`, `created_ts_ist`, `domain`, …), strings are NFC-normalized, no embedded spaces. The 64-hex hashes are full sha256 hex digests.

### 8.6 `README.md` YAML frontmatter (HF Hub dataset card)

The published `<org>/driftcall-indic-briefs/README.md` begins with the following YAML frontmatter. The HF Dataset Viewer reads this block to auto-configure splits, license, and task tags.

```yaml
---
license: apache-2.0
language: [hi, ta, kn, en]
size_categories: [10K<n<100K]
task_categories: [conversational, text-generation]
pretty_name: DriftCall Indic Briefs
configs:
  - config_name: default
    data_files:
      - split: train
        path: train/briefs.jsonl
      - split: val
        path: val/briefs.jsonl
dataset_info:
  features:
    - { name: episode_id, dtype: string }
    - { name: seed, dtype: int64 }
    - { name: stage, dtype: int32 }
    - { name: language, dtype: string }
    - { name: domain, dtype: string }
    - { name: template_id, dtype: string }
  splits:
    - { name: train, num_examples: 15000 }
    - { name: val, num_examples: 500 }
---
```

The body of `README.md` follows below the frontmatter: dataset description, licensing chain (full Apache-2.0 text is in the separate `LICENSE` file per §3.4), provenance (`generator_version`, `catalogue_hash`), reward-caveat paragraph, and usage example. The frontmatter's `features` block lists only the top-level flat columns; nested structs (`goal`, `drift_schedule`) are auto-inferred by the HF Datasets library on first load.

---

## 9. Open Questions

1. **HF org name not yet finalized.** `<org>` placeholder in `<org>/driftcall-indic-briefs` depends on `DRIFTCALL/CLAUDE.md` §8 kickoff-checklist item "HF org name locked". The publication script parameterizes the org via `--hf-org`; no code change needed once locked, just a CLI arg at publication time. Does not block Phase D. **Sync note:** `DRIFTCALL/CLAUDE.md` §6 command table still lists the deprecated `huggingface-cli upload` — when the org name is locked, update that table to the modern `hf upload` in the same PR.

2. **SFT warmup corpus size — 200 vs 500 trajectories.** DESIGN.md §8.2 row 4 quotes the range "200–500". The exact count depends on Sarvam-M's cost/latency budget during one-shot synthesis. Recommend 200 as a floor (sufficient for format priming per §10 training convergence target) and 500 as a ceiling if inference time permits. Resolution: Person C chooses during Phase C4; does not affect loader or schema.

3. **Audio manifest curation count.** DESIGN.md §9 implies a handful of real IndicVoices-R clips for pitch demo realism, but does not specify exact count. Recommend 20 curated clips (4 per language × 5 languages), balanced by speaker gender and dialect region. Resolution: Person D curates during Phase C5; this module only ensures the manifest format is stable.

### 9.1 Resolved

- **License-cache implementation (previously Open Q #4).** `data/.license_cache/{sgd,mtop}.idx` is a sqlite3 FTS5 index built by `scripts/build_license_cache.py` at dev time. Schema: `CREATE VIRTUAL TABLE licensed_text USING fts5(chunk_text, source_id);` with 5-gram tokenization. CI invokes this index (read-only) on each PR to verify that no `seed_utterance` or template variant in the publication bundle substring-matches any upstream CC-BY-SA text (≥ 10-token threshold, §3.4). The index is built once per upstream corpus version and committed to the repo so re-builds are only needed when SGD or MTOP themselves publish a new version. Determinism + reviewability win over per-PR rebuild cost.

---

**This doc tells you HOW the four dataset layers are shaped, loaded, validated, and published. Do not write loaders before a fresh critic returns `NOTHING_FURTHER`. Do not commit `data/*.yaml` without the pre-commit NFC + PII + license-header guards running. Do not ship the HF Hub bundle without the train/val disjointness and verbatim-match checks green.**