tools/schema_v2_synthetic_augment/README.md

# schema_v2_synthetic_augment

Independent schema v2 synthetic augmentation generator for numeric-title and
path-context samples. It does not rewrite DMHY source records or modify the
template application pipeline.

```powershell
cargo run --release --manifest-path tools\schema_v2_synthetic_augment\Cargo.toml --bin schema_v2_synthetic_augment -- `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --label-schema-file label_schema.json `
  --numeric-title-seeds data\synthetic_numeric_titles.txt `
  --path-prefix-seeds data\synthetic_path_prefixes.txt `
  --limit-templates 3000 `
  --output data\schema_v2_synthetic_aug.jsonl `
  --manifest-output data\schema_v2_synthetic_aug.manifest.json
```

The output is char-tokenized JSONL (`tokens == list(filename)`) with schema v2
labels and `tokenizer_variant: "char"`.

Validate a generated JSONL/manifest pair with the Rust validator:

```powershell
cargo run --release --manifest-path tools\schema_v2_synthetic_augment\Cargo.toml --bin validate_synthetic_aug_jsonl -- `
  --input data\schema_v2_synthetic_aug.jsonl `
  --manifest data\schema_v2_synthetic_aug.manifest.json
```

Smoke validation example:

```powershell
cargo run --release --manifest-path tools\schema_v2_synthetic_augment\Cargo.toml --bin validate_synthetic_aug_jsonl -- `
  --input reports\schema_v2_synthetic_aug.smoke.jsonl `
  --manifest reports\schema_v2_synthetic_aug.smoke.manifest.json
```

Path seeds are typed: plain or `series|...` entries generate `path_series`
rows; `movie|...` / `film|...` / `theatrical|...` entries generate
`path_movie` rows; `special|...` entries generate `path_special` rows; and
`confuser|...` entries generate `path_confuser` rows. The manifest reports each
count separately so training mixtures can control these distributions and avoid
mixing TV-series and movie/special path priors.

## Plan refinements before scaling

### Separate TV-series and movie/special path generation

Do not treat TV anime and theatrical/movie releases as interchangeable when
wrapping filenames in synthetic paths. A TV-style filename wrapped in a movie
folder, or a movie/special-style filename wrapped in `Season 01` / `S01` / `第2季`
folders, creates noisy path labels and can teach the parser the wrong prior.

Before expanding beyond the small validation set, split path generation by media
kind:

- `series` / TV rows:
  - may use season directories such as `Season 01`, `Season 1`, `S01`, `01`,
    `第1季`, `第2季`;
  - may emit `PATH_SEASON` only for those directory-level season components;
  - should be paired with ordinary episodic base filenames.
- `movie` / theatrical rows:
  - should use movie-appropriate directories such as `Movie`, `Movies`,
    `Gekijouban`, `劇場版`, `Films`, `BDMovie`, or a direct title folder;
  - must not synthesize TV season directories unless the seed explicitly models
    a collection folder where the season-like component is a `TAG`, not
    `PATH_SEASON`;
  - should be paired with movie/special base filenames, not ordinary TV episode
    templates.
- `special` rows:
  - should cover `SPs`, `Specials`, `Extras`, `OVA`, `OAD`, `NCOP`, `NCED`,
    `PV`, `CM`, `Trailer`, `Menu`;
  - numbered specials keep their number inside `SPECIAL`, not `EPISODE`.
- `confuser` rows:
  - keep hard negatives such as `Gekijouban`, `Movie`, `2004`, and `TV` as
    `TAG` when they are non-season directories;
  - four-digit year directories remain `TAG`, never `PATH_SEASON`.
- noisy library/cache/download prefixes:
  - include a small controlled set of random-looking or operational directories
    such as `_tmp`, `.staging`, `rclone_cache`, `@@watching`, `0xA1B2`,
    `[NAS-01]`, `queue_42`, and `!done`;
  - these segments should remain `O` unless they are an explicit movie/special/
    confuser tag segment;
  - the goal is to teach slash/path robustness without making the model overfit
    to clean media-library roots only.

Implemented generator/manifest requirements for this split:

- base filename media classifier: `series_episode`, `movie`, `special`,
  `unknown`;
- path seed kinds: `series|...`, `movie|...`, `special|...`, `confuser|...`;
- cross-kind combinations are rejected by default:
  - `movie` base + `series` path = drop;
  - `series_episode` base + `movie` path = drop;
  - `confuser` rows remain explicit hard negatives and may wrap any base kind;
- manifest stats include `path_movie_rows`, `path_confuser_rows`,
  `dropped_media_kind_mismatch`, and per-kind seed counts.

### Improve numeric-title seed quality

The numeric title list should not look like a small set of mechanical templates
with only the number swapped. Keep the bucket coverage from the original plan,
but make the names feel closer to anime, light-novel, and web-novel titles. The
seeds still do not need to be real works; they need structural diversity and a
clear label intent: every number inside the title remains part of `TITLE_*`.

Seed curation guidelines:

- Mix short canonical numeric titles with long light-novel / web-novel style
  titles.
- Include Japanese romanization, English, and CJK-flavored forms where useful,
  but keep each seed internally style-consistent. Avoid unnatural mixed-language
  constructions such as romanized Japanese titles with a stray Chinese `年` year
  suffix, or English phrases glued to Japanese grammar.
- Vary separators and numeric forms: `7-nin`, `100-nin`, `3-gatsu`, `12-gatsu`,
  `91 Days`, `No.6`, `Area 88`, `86`, `Level E`, `2.43`, `22/7`, `Dai 7`,
  `1000-nen`, `2001-nen`, `第7`, `1000年`, `2001年`.
- Add long-title patterns, but keep the whole seed in one coherent naming style:
  - romanized Japanese / light-novel style:
    - `7-nin no Maou to Hazure Skill no Slow Life`
    - `100-nin no Classmate ni Kokuhaku Sareta Ore`
    - `3-gatsu no Guild de 12-kai Me no Tensei`
    - `No.6 Chiku no Saigo no Mahoutsukai`
    - `86-banme no Yuusha wa Season 2 wo Shinjinai`
    - `2001-nen ni Netoge no Yome ga Isekai kara Kita`
  - English-style titles:
    - `91 Days After Becoming the Villainess Butler`
    - `Area 88 and the Last Sky Witch`
    - `The Level E Roommate Has 100 Secrets`
    - `500 Days Until the Dungeon Closes`
  - CJK-style titles:
    - `第7王子与100人的魔法公会`
    - `2001年的异世界网游新娘`
    - `1000年后醒来的第86号勇者`
- Explicitly include misleading but title-owned season-like tokens, for example
  `S01 no Rakuen 86`, `Season 03 Chronicle 91`, `Part 2 no No.6`, and
  `Episode 0 no 1000-nen Maou`; these must remain `TITLE_*` when used as the
  title seed.
- Avoid overusing obvious placeholders like `Chronicle`, `Report`, `Club`, and
  `Patrol`; replace some with more natural genre terms such as `Maou`,
  `Yuusha`, `Akuyaku Reijou`, `Guild`, `Dungeon`, `Mahoutsukai`, `Tensei`,
  `Slow Life`, `Netoge`, `Haishin`, `Kouhai`, `Senpai`, `Kyoushitsu`, and
  `Gakuen`.

Acceptance checks for seed quality:

- each required numeric bucket has multiple non-near-duplicate examples;
- long-title seeds do not collapse into the same phrase skeleton;
- title-owned numeric and season-like tokens are verified to stay contiguous
  `TITLE_*` in generated rows;
- fixed cases such as `vcb_numeric_title_nced` improve without introducing new
  false `EPISODE` / `SEASON` spans inside titles.