tools/rust_dmhy_template_apply/README.md

Rust DMHY Template Apply

Multi-core Rust implementation of the DMHY template recipe apply stage.

Build template recipes from the repository root:

cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --cluster `
  --input datasets\AnimeName\dmhy_list.jsonl `
  --summary-output reports\dmhy_template_clusters.full_top5000.summary.json `
  --samples-output reports\dmhy_template_clusters.full_top5000.samples.jsonl `
  --clusters-output reports\dmhy_template_clusters.full_top5000.jsonl `
  --recipes-output reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --review-output reports\dmhy_template_review.full_top5000.jsonl `
  --top 5000 `
  --recipe-top 5000 `
  --review-top 5000 `
  --min-count 2 `
  --recipe-min-count 25 `
  --threads 24

Apply template recipes from the repository root:

cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --input datasets\AnimeName\dmhy_list.jsonl `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --output reports\dmhy_weak.template_generated.top5000.rust.jsonl `
  --manifest-output reports\dmhy_weak.template_generated.top5000.rust.manifest.json

Audit low-frequency recipe output from the repository root:

cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --audit-low-frequency `
  --input datasets\AnimeName\dmhy_list.jsonl `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --audit-output reports\dmhy_low_frequency_audit.rust.jsonl `
  --audit-max-count 50 `
  --threads 24

Verify the generated training output has no low-frequency blocking warnings:

cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --verify-generated-output `
  --input reports\dmhy_weak.template_generated.top5000.rust.jsonl `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --audit-max-count 50

Review ambiguous title-boundary decisions with a local LM Studio model or with Pi RPC mode. This is an optional final-pass reviewer: it writes a decision cache JSONL and does not modify recipes or training data directly.

cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --review-title-boundaries-lmstudio `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --title-boundary-decisions-output reports\dmhy_title_boundary_lmstudio_decisions.jsonl `
  --lmstudio-base-url http://127.0.0.1:1234/v1 `
  --lmstudio-model qwen `
  --title-boundary-min-similarity 0.25 `
  --title-boundary-max-similarity 0.85 `
  --limit 50

cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --review-title-boundaries-pi-rpc `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_decisions.jsonl `
  --pi-rpc-model openai/gpt-5.5 `
  --title-boundary-min-similarity 0.25 `
  --title-boundary-max-similarity 0.85 `
  --threads 4 `
  --limit 50

For stratified audits, select heuristic strata explicitly:

# High-similarity adjacent-title merges. The old high-similarity keep heuristic
# was removed after Pi RPC review found it over-preserved title continuations.
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --review-title-boundaries-pi-rpc `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_adjacent_highsim.jsonl `
  --title-boundary-heuristic-contains merge_adjacent_title_continuation `
  --title-boundary-min-similarity 0.85 `
  --title-boundary-max-similarity 1.0 `
  --threads 4 `
  --limit 20

# Structural keep candidates normally excluded from ambiguous merge review.
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
  --review-title-boundaries-pi-rpc `
  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
  --title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_structural_keep.jsonl `
  --title-boundary-heuristic-contains keep_structural_boundary `
  --title-boundary-min-similarity 0.0 `
  --title-boundary-max-similarity 1.0 `
  --threads 4 `
  --limit 20

Optional controls:

--threads 24
--limit 50000
--limit-templates 1000
--min-count 10
--confidence high
--expand sample --sample-per-template 100
--keep-encoding-noise

The output record schema is filename, tokens, labels, template_id, and template, plus optional source_filename, path_trimmed, and dropped_title_candidate_positions. Clustered recipe rows also include title_spans and title_boundary_decisions metadata so downstream synthetic augmentation can distinguish one logical title span from repeated/path title slots.

For low-frequency templates (count <= --audit-max-count, default 50), apply uses a conservative gate: records with no_title, multiple_title_spans, path_retained, or hash_labeled audit warnings are skipped from the training JSONL and left in the audit/review files. This keeps common templates stable while preventing rare ambiguous path/title cases from polluting the generated dataset.