schema_v2_synthetic_augment

Independent schema v2 synthetic augmentation generator for numeric-title and path-context samples. It does not rewrite DMHY source records or modify the template application pipeline.

cargo run --release --manifest-path tools\schema_v2_synthetic_augment\Cargo.toml --bin schema_v2_synthetic_augment -- `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --label-schema-file label_schema.json `
  --numeric-title-seeds data\synthetic_numeric_titles.txt `
  --path-prefix-seeds data\synthetic_path_prefixes.txt `
  --limit-templates 3000 `
  --output data\schema_v2_synthetic_aug.jsonl `
  --manifest-output data\schema_v2_synthetic_aug.manifest.json

The output is char-tokenized JSONL (tokens == list(filename)) with schema v2 labels and tokenizer_variant: "char".

Validate a generated JSONL/manifest pair with the Rust validator:

cargo run --release --manifest-path tools\schema_v2_synthetic_augment\Cargo.toml --bin validate_synthetic_aug_jsonl -- `
  --input data\schema_v2_synthetic_aug.jsonl `
  --manifest data\schema_v2_synthetic_aug.manifest.json

Smoke validation example:

cargo run --release --manifest-path tools\schema_v2_synthetic_augment\Cargo.toml --bin validate_synthetic_aug_jsonl -- `
  --input reports\schema_v2_synthetic_aug.smoke.jsonl `
  --manifest reports\schema_v2_synthetic_aug.smoke.manifest.json

Path seeds are typed: plain or series|... entries generate path_series rows; movie|... / film|... / theatrical|... entries generate path_movie rows; special|... entries generate path_special rows; and confuser|... entries generate path_confuser rows. The manifest reports each count separately so training mixtures can control these distributions and avoid mixing TV-series and movie/special path priors.

Plan refinements before scaling

Separate TV-series and movie/special path generation

Do not treat TV anime and theatrical/movie releases as interchangeable when wrapping filenames in synthetic paths. A TV-style filename wrapped in a movie folder, or a movie/special-style filename wrapped in Season 01 / S01 / 第2季 folders, creates noisy path labels and can teach the parser the wrong prior.

Before expanding beyond the small validation set, split path generation by media kind:

series / TV rows:
- may use season directories such as Season 01, Season 1, S01, 01, 第1季, 第2季;
- may emit PATH_SEASON only for those directory-level season components;
- should be paired with ordinary episodic base filenames.
movie / theatrical rows:
- should use movie-appropriate directories such as Movie, Movies, Gekijouban, 劇場版, Films, BDMovie, or a direct title folder;
- must not synthesize TV season directories unless the seed explicitly models a collection folder where the season-like component is a TAG, not PATH_SEASON;
- should be paired with movie/special base filenames, not ordinary TV episode templates.
special rows:
- should cover SPs, Specials, Extras, OVA, OAD, NCOP, NCED, PV, CM, Trailer, Menu;
- numbered specials keep their number inside SPECIAL, not EPISODE.
confuser rows:
- keep hard negatives such as Gekijouban, Movie, 2004, and TV as TAG when they are non-season directories;
- four-digit year directories remain TAG, never PATH_SEASON.
noisy library/cache/download prefixes:
- include a small controlled set of random-looking or operational directories such as _tmp, .staging, rclone_cache, @@watching, 0xA1B2, [NAS-01], queue_42, and !done;
- these segments should remain O unless they are an explicit movie/special/ confuser tag segment;
- the goal is to teach slash/path robustness without making the model overfit to clean media-library roots only.

Implemented generator/manifest requirements for this split:

base filename media classifier: series_episode, movie, special, unknown;
path seed kinds: series|..., movie|..., special|..., confuser|...;
cross-kind combinations are rejected by default:
- movie base + series path = drop;
- series_episode base + movie path = drop;
- confuser rows remain explicit hard negatives and may wrap any base kind;
manifest stats include path_movie_rows, path_confuser_rows, dropped_media_kind_mismatch, and per-kind seed counts.

Improve numeric-title seed quality

The numeric title list should not look like a small set of mechanical templates with only the number swapped. Keep the bucket coverage from the original plan, but make the names feel closer to anime, light-novel, and web-novel titles. The seeds still do not need to be real works; they need structural diversity and a clear label intent: every number inside the title remains part of TITLE_*.

Seed curation guidelines:

Mix short canonical numeric titles with long light-novel / web-novel style titles.
Include Japanese romanization, English, and CJK-flavored forms where useful, but keep each seed internally style-consistent. Avoid unnatural mixed-language constructions such as romanized Japanese titles with a stray Chinese 年 year suffix, or English phrases glued to Japanese grammar.
Vary separators and numeric forms: 7-nin, 100-nin, 3-gatsu, 12-gatsu, 91 Days, No.6, Area 88, 86, Level E, 2.43, 22/7, Dai 7, 1000-nen, 2001-nen, 第7, 1000年, 2001年.
Add long-title patterns, but keep the whole seed in one coherent naming style:
- romanized Japanese / light-novel style:
  - 7-nin no Maou to Hazure Skill no Slow Life
  - 100-nin no Classmate ni Kokuhaku Sareta Ore
  - 3-gatsu no Guild de 12-kai Me no Tensei
  - No.6 Chiku no Saigo no Mahoutsukai
  - 86-banme no Yuusha wa Season 2 wo Shinjinai
  - 2001-nen ni Netoge no Yome ga Isekai kara Kita
- English-style titles:
  - 91 Days After Becoming the Villainess Butler
  - Area 88 and the Last Sky Witch
  - The Level E Roommate Has 100 Secrets
  - 500 Days Until the Dungeon Closes
- CJK-style titles:
  - 第7王子与100人的魔法公会
  - 2001年的异世界网游新娘
  - 1000年后醒来的第86号勇者
Explicitly include misleading but title-owned season-like tokens, for example S01 no Rakuen 86, Season 03 Chronicle 91, Part 2 no No.6, and Episode 0 no 1000-nen Maou; these must remain TITLE_* when used as the title seed.
Avoid overusing obvious placeholders like Chronicle, Report, Club, and Patrol; replace some with more natural genre terms such as Maou, Yuusha, Akuyaku Reijou, Guild, Dungeon, Mahoutsukai, Tensei, Slow Life, Netoge, Haishin, Kouhai, Senpai, Kyoushitsu, and Gakuen.

Acceptance checks for seed quality:

each required numeric bucket has multiple non-near-duplicate examples;
long-title seeds do not collapse into the same phrase skeleton;
title-owned numeric and season-like tokens are verified to stay contiguous TITLE_* in generated rows;
fixed cases such as vcb_numeric_title_nced improve without introducing new false EPISODE / SEASON spans inside titles.