tools/rust_dmhy_template_apply/README.md

# Rust DMHY Template Apply

Multi-core Rust implementation of the DMHY template recipe apply stage.

Build template recipes from the repository root:

```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --cluster `
  --input datasets\AnimeName\dmhy_list.jsonl `
  --summary-output reports\dmhy_template_clusters.full_top5000.summary.json `
  --samples-output reports\dmhy_template_clusters.full_top5000.samples.jsonl `
  --clusters-output reports\dmhy_template_clusters.full_top5000.jsonl `
  --recipes-output reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --review-output reports\dmhy_template_review.full_top5000.jsonl `
  --top 5000 `
  --recipe-top 5000 `
  --review-top 5000 `
  --min-count 2 `
  --recipe-min-count 25 `
  --threads 24
```

Apply template recipes from the repository root:

```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --input datasets\AnimeName\dmhy_list.jsonl `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --output reports\dmhy_weak.template_generated.top5000.rust.jsonl `
  --manifest-output reports\dmhy_weak.template_generated.top5000.rust.manifest.json
```

Audit low-frequency recipe output from the repository root:

```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --audit-low-frequency `
  --input datasets\AnimeName\dmhy_list.jsonl `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --audit-output reports\dmhy_low_frequency_audit.rust.jsonl `
  --audit-max-count 50 `
  --threads 24
```

Verify the generated training output has no low-frequency blocking warnings:

```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --verify-generated-output `
  --input reports\dmhy_weak.template_generated.top5000.rust.jsonl `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --audit-max-count 50
```

Review ambiguous title-boundary decisions with a local LM Studio model or with
Pi RPC mode. This is an optional final-pass reviewer: it writes a decision cache
JSONL and does not modify recipes or training data directly.

```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --review-title-boundaries-lmstudio `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --title-boundary-decisions-output reports\dmhy_title_boundary_lmstudio_decisions.jsonl `
  --lmstudio-base-url http://127.0.0.1:1234/v1 `
  --lmstudio-model qwen `
  --title-boundary-min-similarity 0.25 `
  --title-boundary-max-similarity 0.85 `
  --limit 50
```

```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --review-title-boundaries-pi-rpc `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_decisions.jsonl `
  --pi-rpc-model openai/gpt-5.5 `
  --title-boundary-min-similarity 0.25 `
  --title-boundary-max-similarity 0.85 `
  --threads 4 `
  --limit 50
```

For stratified audits, select heuristic strata explicitly:

```powershell
# High-similarity adjacent-title merges. The old high-similarity keep heuristic
# was removed after Pi RPC review found it over-preserved title continuations.
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --review-title-boundaries-pi-rpc `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_adjacent_highsim.jsonl `
  --title-boundary-heuristic-contains merge_adjacent_title_continuation `
  --title-boundary-min-similarity 0.85 `
  --title-boundary-max-similarity 1.0 `
  --threads 4 `
  --limit 20

# Structural keep candidates normally excluded from ambiguous merge review.
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --review-title-boundaries-pi-rpc `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_structural_keep.jsonl `
  --title-boundary-heuristic-contains keep_structural_boundary `
  --title-boundary-min-similarity 0.0 `
  --title-boundary-max-similarity 1.0 `
  --threads 4 `
  --limit 20
```

Optional controls:

```powershell
--threads 24
--limit 50000
--limit-templates 1000
--min-count 10
--confidence high
--expand sample --sample-per-template 100
--keep-encoding-noise
```

The output record schema is `filename`, `tokens`, `labels`, `template_id`, and
`template`, plus optional `source_filename`, `path_trimmed`, and
`dropped_title_candidate_positions`. Clustered recipe rows also include
`title_spans` and `title_boundary_decisions` metadata so downstream synthetic
augmentation can distinguish one logical title span from repeated/path title
slots.

For low-frequency templates (`count <= --audit-max-count`, default `50`), apply
uses a conservative gate: records with `no_title`, `multiple_title_spans`,
`path_retained`, or `hash_labeled` audit warnings are skipped from the training
JSONL and left in the audit/review files. This keeps common templates stable
while preventing rare ambiguous path/title cases from polluting the generated
dataset.