Remove stale diagnostics reports

AGENTS.md

@@ -23,11 +23,11 @@ and ONNX export workspace used by MiruPlay as `tools/anime_parser`.
 ## Setup
 
 ```bash
-python -m pip install -r requirements.txt
+uv sync
 ```
 
-For local GPU training, install a CUDA-compatible PyTorch build first, then
-install the remaining requirements.
+Use `uv run`, `uv add`, and `uv sync` for environment operations. Do not use
+global `pip` for repository work.
 
 If the dataset submodule is missing, initialize it:
 
@@ -40,31 +40,31 @@ git submodule update --init --recursive
 Run a parser smoke check:
 
 ```bash
-python inference.py --model-dir . "Witch.Hat.Atelier.S01E07.1080p.NF.WEB-DL.JPN.AAC2.0.H.264.MSubs-ToonsHub"
+uv run python inference.py --model-dir . "Witch.Hat.Atelier.S01E07.1080p.NF.WEB-DL.JPN.AAC2.0.H.264.MSubs-ToonsHub"
 ```
 
-Run the lightweight training pipeline check:
+Run fixed real-world parser regression:
 
 ```bash
-python test_train_small.py --limit-samples 5000 --epochs 2
+uv run python evaluate_parser_cases.py --model-dir . --case-file data/parser_regression_cases.json --output case_metrics.json
 ```
 
-Train the default regex tokenizer from the dataset submodule:
+Benchmark PyTorch and ONNX Runtime inference:
 
 ```bash
-python train.py --data-file datasets/AnimeName/dmhy_weak.jsonl --vocab-file datasets/AnimeName/vocab.json --save-dir checkpoints/dmhy-finetune --init-model-dir . --epochs 1 --batch-size 128 --learning-rate 0.0003 --warmup-steps 300 --seed 42
+uv run python benchmark_inference.py --model-dir . --onnx exports/anime_filename_parser.onnx --case-file data/parser_regression_cases.json --repeat 20 --warmup 20 --torch-threads 1 --ort-threads 1 --output benchmark_results.json
 ```
 
-Train the character tokenizer only when that variant is intentional:
+Train the current default character tokenizer:
 
 ```bash
-python train.py --tokenizer char --data-file datasets/AnimeName/dmhy_weak_char.jsonl --vocab-file datasets/AnimeName/vocab.char.json --save-dir checkpoints/dmhy-weak-char --epochs 1 --batch-size 64 --learning-rate 0.0003 --warmup-steps 300 --max-seq-length 128 --seed 42
+uv run python train.py --tokenizer char --data-file datasets/AnimeName/dmhy_weak_char.jsonl --vocab-file datasets/AnimeName/vocab.char.json --save-dir checkpoints/dmhy-char-full --init-model-dir . --epochs 2 --batch-size 256 --learning-rate 0.00008 --warmup-steps 300 --max-seq-length 128 --train-split 0.98 --num-workers 4 --checkpoint-steps 1000 --save-total-limit 3 --parse-eval-limit 2048 --case-eval-file data/parser_regression_cases.json --seed 52 --experiment-name dmhy-char-full
 ```
 
 Export for Android:
 
 ```bash
-python export_onnx.py --model-dir checkpoints/dmhy-finetune/final --android-assets-dir ../../scraper/src/main/assets/anime_parser
+uv run python export_onnx.py --model-dir . --max-length 128 --android-assets-dir ../../scraper/src/main/assets/anime_parser
 ```
 
 ## Codex-Controlled Colab Training
@@ -131,10 +131,14 @@ land under `MyDrive/AniFileBERT/worker/jobs/<job-id>/`.
 
 - For parser or tokenizer changes, run `python inference.py --model-dir . ...`
   with at least one realistic filename.
+- Run `uv run python evaluate_parser_cases.py --model-dir . --case-file data/parser_regression_cases.json --output case_metrics.json`
+  before publishing parser changes.
 - For dataset alignment, tokenizer, model, or training-loop changes, run
   `python test_train_small.py --limit-samples 5000 --epochs 2` when practical.
 - For export changes, run `python export_onnx.py ...` and confirm the exporter
   reports a small PyTorch/ONNX logits difference.
+- For performance-sensitive inference changes, run `uv run python benchmark_inference.py ...`
+  and update `benchmark_results.json` plus the README performance table.
 - Full training is expensive; do not start long multi-epoch runs unless the
   task explicitly requires it.
 
@@ -149,6 +153,9 @@ land under `MyDrive/AniFileBERT/worker/jobs/<job-id>/`.
   `datasets/AnimeName/dmhy_weak_char.jsonl` for character tokenization.
   Treat `mixed_train.jsonl`, `ab_mix_100k.jsonl`, and other alternate JSONL
   files as legacy unless a task explicitly asks to inspect them.
+- The published default checkpoint is the character tokenizer variant with
+  `max_seq_length=128`. Keep `vocab.json`, `vocab.char.json`, `config.json`,
+  ONNX export, Android assets, and docs synchronized.
 - Large binary artifacts are tracked through Git LFS by `.gitattributes`.
   Preserve LFS handling for `.safetensors`, `.onnx`, `.bin`, and related model
   files.
@@ -156,6 +163,8 @@ land under `MyDrive/AniFileBERT/worker/jobs/<job-id>/`.
   repository root as described in `MAINTENANCE.md`.
 - When updating `datasets/AnimeName`, commit the submodule pointer in this repo
   and then update the parent MiruPlay submodule pointer.
+- Push LFS objects before pushing Git commits when model or ONNX artifacts
+  changed: `git lfs push origin main --all`, then `git push origin main`.
 
 ## Coding Notes
 

diagnostics_report.md

@@ -1,277 +0,0 @@
-# Anime Filename Parser Diagnostics Report
-
-## 根因分析
-
-当前症状不是 learning rate 问题，而是训练、验证、推理没有在同一个结构化输入空间里工作。
-
-最高优先级根因是 tokenizer/data 配置错位：你给出的训练命令使用 `dmhy_weak_char.jsonl` 和 `vocab.char.json`，但没有传 `--tokenizer char`。旧版 `train.py` 默认 `regex`，因此 char 数据会被当作 regex 训练配置保存，checkpoint metadata 会写成 `tokenizer_variant=regex`。推理时 `load_tokenizer()` 按 checkpoint metadata 重新加载 regex tokenizer，于是 `[LoliHouse]` 这类结构 token 会作为一个整体进入模型，而 char 训练数据里它是 `[`, `L`, `o`, ..., `]`。这会直接导致 group/title 边界漂移。
-
-第二个根因是 word-level 数据和当前 `AnimeTokenizer` 也不完全一致。`dmhy_weak.jsonl` 里示例 token 是 `[`, `LoliHouse`, `]`，但当前 regex tokenizer 对原始文件名会输出 `[LoliHouse]`。这说明 word-level 数据名义上是 regex，但不是严格由当前 inference tokenizer 重放得到的 token 序列。
-
-第三个根因是 char 训练命令没有设置 `--max-seq-length 128`。在抽样 5,000 条 char 数据中，默认 64 长度会截断 2,058 条，占 41.16%。episode/source/resolution 往往在后半段，默认长度会让模型训练和推理都丢失结构锚点。
-
-第四个根因是评估指标误导。低 validation loss 和 token accuracy 会被大量 `O`、`I-TITLE` 稀释；真实任务需要 entity-level F1、字段 exact match，以及结构案例回归。
-
-## 问题优先级
-
-P0: 训练命令必须显式或自动使用 char tokenizer。已修改 `train.py`，现在会从数据集 metadata 自动识别 `char`，并把 char 默认 max length 提升到 128。
-
-P0: 不允许 tokenizer variant 与 dataset metadata 不一致。已修改 `train.py`，检测到 dataset `tokenizer_variant` 与选择的 tokenizer 不一致会报错。
-
-P0: 推理必须使用 checkpoint 保存的 tokenizer 和 max length。已修改 `inference.py`，默认读取 `model.config.max_seq_length`，并新增 `--debug` 输出 token/label/score/UNK/截断信息。
-
-P1: 从旧 checkpoint fine-tune 到不同 vocab 时，不能按 ID 盲目 `resize_token_embeddings()`。已修改为按 token 字符串重映射 embedding，未匹配 token 再随机初始化。
-
-P1: 数据集存在 BIO/边界质量问题。char 抽样 5,000 条发现 468 个 `ORPHAN_I`，典型是标题被括号 `O` 打断后仍继续 `I-TITLE`。`B-X -> O` 本身是合法 BIO，但在 group/title/source 频繁出现时是边界告警。
-
-P2: 当前 `BertForTokenClassification` 独立逐 token 解码，不能约束非法转移。建议后续加 CRF 或 constrained BIO decoder。
-
-## 自动诊断结果
-
-新增脚本：
-
-```bash
-python diagnose_pipeline.py --data-file datasets/AnimeName/dmhy_weak_char.jsonl --vocab-file datasets/AnimeName/vocab.char.json --model-dir checkpoints/dmhy-finetune/final --sample-limit 5000 --eval-limit 128 --output diagnostics_report.md
-```
-
-char 数据抽样结果：
-
-- tokenizer variant: `char`
-- vocab size: 6,199
-- UNK rate: 0.0000%
-- O-label ratio: 37.47%
-- p95 length: 101, p99 length: 125
-- default max length 64 truncation: 41.16%
-- `ORPHAN_I`: 468
-- regex checkpoint 直接评 char 数据时 entity F1: 0.0832
-
-word 数据抽样结果保存在 `diagnostics_report_word.md`：
-
-- tokenizer variant: `regex`
-- vocab size: 8,000
-- UNK rate: 6.9158%
-- default max length 64 truncation: 0%
-- 当前 regex checkpoint 在抽样 word 数据上 entity F1: 0.9549
-- 但 model checkpoint vocab 是 3,000，诊断 vocab 是 8,000，继续 fine-tune 必须重映射 embedding
-
-## Tokenizer Split 示例
-
-输入：
-
-```text
-[LoliHouse] Yomi no Tsugai - 07 [WebRip 1080p HEVC-10bit AAC ASSx2]
-```
-
-char tokenizer：
-
-```text
-[, L, o, l, i, H, o, u, s, e, ],  , Y, o, m, i,  , n, o,  , T, s, u, g, a, i,  , -,  , 0, 7, ...
-```
-
-当前 regex tokenizer：
-
-```text
-[LoliHouse],  , Yomi,  , no,  , Tsugai,  , -,  , 07,  , [WebRip 1080p HEVC-10bit AAC ASSx2]
-```
-
-这两个 token 序列不是同一个标注空间。char label 不能直接套到 regex token 上，regex 模型也不能在 char token 序列上解释 logits。
-
-## BIO 与边界问题
-
-真实非法 BIO：
-
-```text
-... ( O, K I-TITLE, a I-TITLE ...
-```
-
-示例：
-
-```text
-[LoliHouse] Kanteishi (Kari) - 07 [WebRip 1080p HEVC-10bit AAC]
-```
-
-`(` 被标为 `O`，后面的 `Kari` 继续 `I-TITLE`，形成 `O -> I-TITLE`。这会让模型学习到标题可以跨越被标为非实体的括号，边界自然会漂。
-
-结构边界告警：
-
-```text
-[KissSub][Shunkashuutou Daikousha - Haru no Mai][06][1080P][GB][MP4]
-```
-
-`KissSub` 是 `B-GROUP`，右括号是 `O`，这是合法 BIO；但如果 tokenizer 在推理时把 `[KissSub]` 合成一个 token，模型就无法只给内部文字打 `GROUP`，只能把整个 bracket token 判成一个类别。
-
-## Confusion 分析
-
-故意用 char 数据评估 regex checkpoint，entity F1 只有 0.0832。主要混淆：
-
-- `O -> TITLE`: 930
-- `SOURCE -> TITLE`: 236
-- `EPISODE -> TITLE`: 228
-- `GROUP -> TITLE`: 86
-
-这与实际症状一致：模型把结构锚点和 meta 区域吸进 title，group/title 边界混淆，episode 被 title 或 O 吞掉。
-
-## 已修改的代码
-
-`train.py`
-
-- `--tokenizer` 默认从数据集 metadata/vocab 名称/样本结构自动推断。
-- char 数据默认 `max_seq_length >= 128`。
-- dataset metadata 与 tokenizer 不一致会直接报错。
-- fine-tune 到新 vocab 时按 token 字符串重映射 embedding，避免 token ID 语义错位。
-- checkpoint 保存正确的 `tokenizer_variant` 和 `max_seq_length`。
-
-`inference.py`
-
-- 新增 `--debug`，输出 tokenizer variant、token IDs、labels、scores、UNK rate、truncation、entity spans。
-- 默认使用 checkpoint `max_seq_length`。
-- 修正推理截断逻辑，保留 `[SEP]`，与训练一致。
-- 默认使用 constrained BIO Viterbi 解码，阻止 `O -> I-X` 这类非法转移；可用 `--no-constrained-bio` 查看原始 greedy 输出。
-- 新增 rule-assisted parsing，兜底修复高置信结构锚点：leading group bracket、` - 07`、`S01E07`、resolution、source。
-- 可用 `--no-rule-assist` 关闭规则兜底，只看模型原始输出。
-
-`diagnose_pipeline.py`
-
-- 自动检查 token/label 长度。
-- 输出 BIO 违规样本与边界告警。
-- 输出 tokenizer split 示例。
-- 输出 train/inference tokenizer 对比。
-- 输出实体、label、空格 label、UNK、截断统计。
-- 可选加载 checkpoint 做 confusion 和 seqeval entity-level F1。
-
-## 修改后的 Pipeline
-
-推荐 char-level pipeline：
-
-```bash
-python diagnose_pipeline.py ^
-  --data-file datasets/AnimeName/dmhy_weak_char.jsonl ^
-  --vocab-file datasets/AnimeName/vocab.char.json ^
-  --sample-limit 20000 ^
-  --output diagnostics_report.md
-
-python train.py ^
-  --tokenizer char ^
-  --data-file datasets/AnimeName/dmhy_weak_char.jsonl ^
-  --vocab-file datasets/AnimeName/vocab.char.json ^
-  --save-dir checkpoints/dmhy-char ^
-  --epochs 10 ^
-  --batch-size 128 ^
-  --learning-rate 0.0003 ^
-  --warmup-steps 300 ^
-  --max-seq-length 128 ^
-  --seed 42
-
-python inference.py ^
-  --model-dir checkpoints/dmhy-char/final ^
-  --debug ^
-  "[LoliHouse] Yomi no Tsugai - 07 [WebRip 1080p HEVC-10bit AAC ASSx2]"
-```
-
-如果继续使用 word/regex pipeline，必须先重新生成数据，使 `sample["tokens"] == AnimeTokenizer.tokenize(sample["filename"])` 对绝大多数样本成立；否则验证集仍然是训练 token 空间，真实 inference 是另一个 token 空间。
-
-## 最合理的 Tokenizer 方案
-
-当前任务更适合 char-level 或 deterministic hybrid tokenizer，不适合通用 subword tokenizer。
-
-char-level 优点：
-
-- train/inference 最容易完全一致。
-- 不会把 `[LoliHouse]`、`[WebRip ...]` 这类结构块压成单 token。
-- 对未知标题、组名、罗马音、中文、日文都没有 OOV。
-- 更适合学习括号、空格、连字符、集数位置这些结构信号。
-
-char-level 缺点：
-
-- 序列更长，必须用 `max_seq_length=128`。
-- 逐 token softmax 容易出现 BIO 非法转移，建议加 CRF。
-
-word-level/regex 优点：
-
-- 序列短，训练快。
-- 当前已有 checkpoint 在同 token 空间验证集上 F1 较高。
-
-word-level/regex 缺点：
-
-- 如果 bracket protection 把整段合并，内部 label 无法表达。
-- 数据生成 tokenizer 和 inference tokenizer 稍有不一致就会严重错位。
-- OOV 对新番标题和组名仍然明显。
-
-结论：短期用 char-level + rule-assisted parsing；中期改为 hybrid tokenizer：保留结构符号 `[ ] ( ) - _ . space` 为独立 token，英文数字连续串可作为片段但必须能映射回字符 offset，并在 label alignment 上以 offset 为准；长期加 BERT + CRF。
-
-## 建议训练配置
-
-首选：
-
-```bash
-python train.py --tokenizer char ^
-  --data-file datasets/AnimeName/dmhy_weak_char.jsonl ^
-  --vocab-file datasets/AnimeName/vocab.char.json ^
-  --save-dir checkpoints/dmhy-char ^
-  --epochs 10 --batch-size 128 ^
-  --learning-rate 0.0003 --warmup-steps 300 ^
-  --max-seq-length 128 --seed 42
-```
-
-不要从 regex checkpoint 直接当作同构模型继续训练 char；如果要迁移，当前代码会按 token 字符串 remap embedding，但多数 char token 与 regex token 共享有限，最好从头训练 char 模型或只迁移 encoder 非 embedding 层。
-
-必须新增评估：
-
-- entity-level F1 by field
-- field exact match: `group/title/episode/resolution/source`
-- full parse exact match
-- episode recall
-- boundary errors: group-title, title-episode, episode-meta
-- inference debug sample set，固定 50-200 个真实文件名回归
-
-## 真实案例分析
-
-输入：
-
-```text
-[LoliHouse] Yomi no Tsugai - 07 [WebRip 1080p HEVC-10bit AAC ASSx2]
-```
-
-旧 regex checkpoint 原始模型输出：
-
-```json
-{
-  "entities": [
-    {"type": "TITLE", "text": "[LoliHouse] Yomi no Tsugai"},
-    {"type": "EPISODE", "text": "07"}
-  ]
-}
-```
-
-问题点：
-
-- `[LoliHouse]` 被 tokenizer 合成一个 token。
-- 模型把该 token 判成 `B-TITLE`，无法只把内部 `LoliHouse` 判成 `GROUP`。
-- `Yomi` 和 `Tsugai` 在 3,000 vocab checkpoint 中是 `[UNK]`，但模型仍高置信输出 `I-TITLE`，说明 loss/置信度不能代表字段正确性。
-
-修改后带规则辅助的最终输出：
-
-```json
-{
-  "group": "LoliHouse",
-  "title": "Yomi no Tsugai",
-  "episode": 7,
-  "source": "WebRip",
-  "resolution": "1080p"
-}
-```
-
-这只是上线兜底；真正修复仍应训练一个 train/inference token 完全一致的 char 或 hybrid 模型。
-
-## 架构建议
-
-最推荐的重构路线：
-
-1. `BERT encoder + CRF`：约束 `O -> I-X`、`B-X -> I-Y` 等非法/低质量转移。
-2. char-level NER：保证 token-label alignment 不受 subword split 影响。
-3. rule-assisted parser：先抽取高置信结构锚点，再让模型负责模糊 title/group 边界。
-4. offset-based dataset：每条数据保存 raw filename、entity spans、tokens、offset_mapping、labels，训练时由 tokenizer 统一生成 labels。
-
-当前代码已先实现“无训练 CRF”的 constrained BIO decoding，作为上线前的轻量保护。完整 BERT+CRF 仍建议作为下一阶段训练架构重构。
-
-不要只优化 loss。这个任务的目标函数应更接近真实解析准确率：字段级 exact match + episode recall + title boundary F1。

diagnostics_report_word.md

@@ -1,2678 +0,0 @@
-# Anime Filename Parser Diagnostics Report
-
-## Executive Summary
-
-- Dataset: `datasets\AnimeName\dmhy_weak.jsonl`
-- Inspected rows: 5,000
-- Dataset tokenizer variant: `regex`
-- Diagnosed tokenizer variant: `regex`
-- Vocab: `datasets\AnimeName\vocab.json` (8,000 tokens)
-- Max sequence length checked: 64
-- O-label ratio: 38.12%
-- Truncation risk: 0/5,000 rows (0.00%)
-- UNK rate after selected tokenizer: 6.9158%
-- BIO warnings collected: 9,711
-
-Primary finding: this task is structural filename parsing. Tokenizer/preprocessing identity is more important than lowering token loss.
-
-## Label And Entity Statistics
-
-### Label distribution
-- `O`: 32,517 (38.12%)
-- `I-TITLE`: 30,321 (35.54%)
-- `B-TITLE`: 5,593 (6.56%)
-- `B-EPISODE`: 5,000 (5.86%)
-- `B-SOURCE`: 4,032 (4.73%)
-- `I-GROUP`: 2,459 (2.88%)
-- `B-GROUP`: 2,299 (2.69%)
-- `B-RESOLUTION`: 1,765 (2.07%)
-- `B-SEASON`: 1,269 (1.49%)
-- `B-SPECIAL`: 57 (0.07%)
-
-### Entity count
-- `TITLE`: 6,061 (29.59%)
-- `EPISODE`: 5,000 (24.41%)
-- `SOURCE`: 4,032 (19.68%)
-- `GROUP`: 2,299 (11.22%)
-- `RESOLUTION`: 1,765 (8.62%)
-- `SEASON`: 1,269 (6.20%)
-- `SPECIAL`: 57 (0.28%)
-
-### Length distribution
-```json
-{
-  "raw_tokens": {
-    "min": 3,
-    "p50": 17,
-    "p90": 28,
-    "p95": 31,
-    "p99": 39,
-    "max": 54
-  },
-  "aligned_tokens": {
-    "min": 3,
-    "p50": 17,
-    "p90": 28,
-    "p95": 31,
-    "p99": 39,
-    "max": 54
-  }
-}
-```
-
-### Whitespace labels
-- `I-TITLE`: 10,539 (48.98%)
-- `O`: 10,484 (48.72%)
-- `I-GROUP`: 411 (1.91%)
-- `B-TITLE`: 84 (0.39%)
-
-## BIO Violations And Boundary Drift
-
-### Violation counts
-- `B_DIRECT_TO_O`: 9,243 (95.18%)
-- `ORPHAN_I`: 468 (4.82%)
-
-### Boundary drift heuristics
-- none
-
-### Sample violations
-```json
-[
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 8,
-    "prev_label": "B-EPISODE",
-    "label": "O",
-    "token": ".",
-    "row": 1,
-    "file_id": 1,
-    "filename": "Witch.Hat.Atelier.S01E07.1080p.NF.WEB-DL.JPN.AAC2.0.H.264.MSubs-ToonsHub",
-    "context_tokens": [
-      ".",
-      "Atelier",
-      ".",
-      "S01",
-      "E07",
-      ".",
-      "1080p",
-      ".",
-      "NF",
-      ".",
-      "WEB-DL"
-    ],
-    "context_labels": [
-      "I-TITLE",
-      "I-TITLE",
-      "O",
-      "B-SEASON",
-      "B-EPISODE",
-      "O",
-      "B-RESOLUTION",
-      "O",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 10,
-    "prev_label": "B-RESOLUTION",
-    "label": "O",
-    "token": ".",
-    "row": 1,
-    "file_id": 1,
-    "filename": "Witch.Hat.Atelier.S01E07.1080p.NF.WEB-DL.JPN.AAC2.0.H.264.MSubs-ToonsHub",
-    "context_tokens": [
-      ".",
-      "S01",
-      "E07",
-      ".",
-      "1080p",
-      ".",
-      "NF",
-      ".",
-      "WEB-DL",
-      ".",
-      "JP"
-    ],
-    "context_labels": [
-      "O",
-      "B-SEASON",
-      "B-EPISODE",
-      "O",
-      "B-RESOLUTION",
-      "O",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 12,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": ".",
-    "row": 1,
-    "file_id": 1,
-    "filename": "Witch.Hat.Atelier.S01E07.1080p.NF.WEB-DL.JPN.AAC2.0.H.264.MSubs-ToonsHub",
-    "context_tokens": [
-      "E07",
-      ".",
-      "1080p",
-      ".",
-      "NF",
-      ".",
-      "WEB-DL",
-      ".",
-      "JP",
-      "N",
-      "."
-    ],
-    "context_labels": [
-      "B-EPISODE",
-      "O",
-      "B-RESOLUTION",
-      "O",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE",
-      "O",
-      "O"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 14,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": ".",
-    "row": 1,
-    "file_id": 1,
-    "filename": "Witch.Hat.Atelier.S01E07.1080p.NF.WEB-DL.JPN.AAC2.0.H.264.MSubs-ToonsHub",
-    "context_tokens": [
-      "1080p",
-      ".",
-      "NF",
-      ".",
-      "WEB-DL",
-      ".",
-      "JP",
-      "N",
-      ".",
-      "AAC",
-      "2"
-    ],
-    "context_labels": [
-      "B-RESOLUTION",
-      "O",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE",
-      "O",
-      "O",
-      "B-SOURCE",
-      "O"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 16,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": "N",
-    "row": 1,
-    "file_id": 1,
-    "filename": "Witch.Hat.Atelier.S01E07.1080p.NF.WEB-DL.JPN.AAC2.0.H.264.MSubs-ToonsHub",
-    "context_tokens": [
-      "NF",
-      ".",
-      "WEB-DL",
-      ".",
-      "JP",
-      "N",
-      ".",
-      "AAC",
-      "2",
-      ".",
-      "0"
-    ],
-    "context_labels": [
-      "B-SOURCE",
-      "O",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE",
-      "O",
-      "O",
-      "B-SOURCE",
-      "O",
-      "O",
-      "O"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 19,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": "2",
-    "row": 1,
-    "file_id": 1,
-    "filename": "Witch.Hat.Atelier.S01E07.1080p.NF.WEB-DL.JPN.AAC2.0.H.264.MSubs-ToonsHub",
-    "context_tokens": [
-      ".",
-      "JP",
-      "N",
-      ".",
-      "AAC",
-      "2",
-      ".",
-      "0",
-      ".",
-      "H.264",
-      "."
-    ],
-    "context_labels": [
-      "O",
-      "B-SOURCE",
-      "O",
-      "O",
-      "B-SOURCE",
-      "O",
-      "O",
-      "O",
-      "O",
-      "B-SOURCE",
-      "O"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 24,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": ".",
-    "row": 1,
-    "file_id": 1,
-    "filename": "Witch.Hat.Atelier.S01E07.1080p.NF.WEB-DL.JPN.AAC2.0.H.264.MSubs-ToonsHub",
-    "context_tokens": [
-      "2",
-      ".",
-      "0",
-      ".",
-      "H.264",
-      ".",
-      "MSubs",
-      "-",
-      "ToonsHub"
-    ],
-    "context_labels": [
-      "O",
-      "O",
-      "O",
-      "O",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE",
-      "O",
-      "O"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 26,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": "-",
-    "row": 1,
-    "file_id": 1,
-    "filename": "Witch.Hat.Atelier.S01E07.1080p.NF.WEB-DL.JPN.AAC2.0.H.264.MSubs-ToonsHub",
-    "context_tokens": [
-      "0",
-      ".",
-      "H.264",
-      ".",
-      "MSubs",
-      "-",
-      "ToonsHub"
-    ],
-    "context_labels": [
-      "O",
-      "O",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE",
-      "O",
-      "O"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 2,
-    "prev_label": "B-GROUP",
-    "label": "O",
-    "token": "]",
-    "row": 2,
-    "file_id": 2,
-    "filename": "[LoliHouse] Maid-san wa Taberu Dake - 07 [WebRip 1080p HEVC-10bit AAC ASSx2]",
-    "context_tokens": [
-      "[",
-      "LoliHouse",
-      "]",
-      " ",
-      "Maid",
-      "-",
-      "san",
-      " "
-    ],
-    "context_labels": [
-      "O",
-      "B-GROUP",
-      "O",
-      "O",
-      "B-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 17,
-    "prev_label": "B-EPISODE",
-    "label": "O",
-    "token": " ",
-    "row": 2,
-    "file_id": 2,
-    "filename": "[LoliHouse] Maid-san wa Taberu Dake - 07 [WebRip 1080p HEVC-10bit AAC ASSx2]",
-    "context_tokens": [
-      "Dake",
-      " ",
-      "-",
-      " ",
-      "07",
-      " ",
-      "[WebRip 1080p HEVC-10bit AAC ASSx2]"
-    ],
-    "context_labels": [
-      "I-TITLE",
-      "O",
-      "O",
-      "O",
-      "B-EPISODE",
-      "O",
-      "O"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 2,
-    "prev_label": "B-GROUP",
-    "label": "O",
-    "token": "]",
-    "row": 3,
-    "file_id": 3,
-    "filename": "[ANi] 異世界悠閒農家 2 - 06 [1080P][Baha][WEB-DL][AAC AVC][CHT]",
-    "context_tokens": [
-      "[",
-      "ANi",
-      "]",
-      " ",
-      "異",
-      "世",
-      "界",
-      "悠"
-    ],
-    "context_labels": [
-      "O",
-      "B-GROUP",
-      "O",
-      "O",
-      "B-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 13,
-    "prev_label": "B-SEASON",
-    "label": "O",
-    "token": " ",
-    "row": 3,
-    "file_id": 3,
-    "filename": "[ANi] 異世界悠閒農家 2 - 06 [1080P][Baha][WEB-DL][AAC AVC][CHT]",
-    "context_tokens": [
-      "閒",
-      "農",
-      "家",
-      " ",
-      "2",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]"
-    ],
-    "context_labels": [
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "O",
-      "B-SEASON",
-      "O",
-      "O",
-      "O",
-      "B-EPISODE",
-      "O",
-      "B-RESOLUTION"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 17,
-    "prev_label": "B-EPISODE",
-    "label": "O",
-    "token": " ",
-    "row": 3,
-    "file_id": 3,
-    "filename": "[ANi] 異世界悠閒農家 2 - 06 [1080P][Baha][WEB-DL][AAC AVC][CHT]",
-    "context_tokens": [
-      "2",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "context_labels": [
-      "B-SEASON",
-      "O",
-      "O",
-      "O",
-      "B-EPISODE",
-      "O",
-      "B-RESOLUTION",
-      "B-SOURCE",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 21,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": "[AAC AVC]",
-    "row": 3,
-    "file_id": 3,
-    "filename": "[ANi] 異世界悠閒農家 2 - 06 [1080P][Baha][WEB-DL][AAC AVC][CHT]",
-    "context_tokens": [
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "context_labels": [
-      "B-EPISODE",
-      "O",
-      "B-RESOLUTION",
-      "B-SOURCE",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 2,
-    "prev_label": "B-GROUP",
-    "label": "O",
-    "token": "]",
-    "row": 4,
-    "file_id": 4,
-    "filename": "[ANi] 木頭風紀委員和迷你裙 JK 的故事 - 06 [1080P][Baha][WEB-DL][AAC AVC][CHT]",
-    "context_tokens": [
-      "[",
-      "ANi",
-      "]",
-      " ",
-      "木",
-      "頭",
-      "風",
-      "紀"
-    ],
-    "context_labels": [
-      "O",
-      "B-GROUP",
-      "O",
-      "O",
-      "B-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 24,
-    "prev_label": "B-EPISODE",
-    "label": "O",
-    "token": " ",
-    "row": 4,
-    "file_id": 4,
-    "filename": "[ANi] 木頭風紀委員和迷你裙 JK 的故事 - 06 [1080P][Baha][WEB-DL][AAC AVC][CHT]",
-    "context_tokens": [
-      "事",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "context_labels": [
-      "I-TITLE",
-      "O",
-      "O",
-      "O",
-      "B-EPISODE",
-      "O",
-      "B-RESOLUTION",
-      "B-SOURCE",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 28,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": "[AAC AVC]",
-    "row": 4,
-    "file_id": 4,
-    "filename": "[ANi] 木頭風紀委員和迷你裙 JK 的故事 - 06 [1080P][Baha][WEB-DL][AAC AVC][CHT]",
-    "context_tokens": [
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "context_labels": [
-      "B-EPISODE",
-      "O",
-      "B-RESOLUTION",
-      "B-SOURCE",
-      "B-SOURCE",
-      "O",
-      "B-SOURCE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 2,
-    "prev_label": "B-GROUP",
-    "label": "O",
-    "token": "]",
-    "row": 5,
-    "file_id": 5,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][05][1080P][GB][MP4]",
-    "context_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " "
-    ],
-    "context_labels": [
-      "O",
-      "B-GROUP",
-      "O",
-      "O",
-      "B-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 19,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": "[MP4]",
-    "row": 5,
-    "file_id": 5,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][05][1080P][GB][MP4]",
-    "context_tokens": [
-      "Mai",
-      "]",
-      "[05]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "context_labels": [
-      "I-TITLE",
-      "O",
-      "B-EPISODE",
-      "B-RESOLUTION",
-      "B-SOURCE",
-      "O"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 2,
-    "prev_label": "B-GROUP",
-    "label": "O",
-    "token": "]",
-    "row": 6,
-    "file_id": 6,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][06][1080P][GB][MP4]",
-    "context_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " "
-    ],
-    "context_labels": [
-      "O",
-      "B-GROUP",
-      "O",
-      "O",
-      "B-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 19,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": "[MP4]",
-    "row": 6,
-    "file_id": 6,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][06][1080P][GB][MP4]",
-    "context_tokens": [
-      "Mai",
-      "]",
-      "[06]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "context_labels": [
-      "I-TITLE",
-      "O",
-      "B-EPISODE",
-      "B-RESOLUTION",
-      "B-SOURCE",
-      "O"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 2,
-    "prev_label": "B-GROUP",
-    "label": "O",
-    "token": "]",
-    "row": 7,
-    "file_id": 7,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][06][1080P][BIG5][MP4]",
-    "context_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " "
-    ],
-    "context_labels": [
-      "O",
-      "B-GROUP",
-      "O",
-      "O",
-      "B-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 19,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": "[MP4]",
-    "row": 7,
-    "file_id": 7,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][06][1080P][BIG5][MP4]",
-    "context_tokens": [
-      "Mai",
-      "]",
-      "[06]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "context_labels": [
-      "I-TITLE",
-      "O",
-      "B-EPISODE",
-      "B-RESOLUTION",
-      "B-SOURCE",
-      "O"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 2,
-    "prev_label": "B-GROUP",
-    "label": "O",
-    "token": "]",
-    "row": 8,
-    "file_id": 8,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][05][1080P][BIG5][MP4]",
-    "context_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " "
-    ],
-    "context_labels": [
-      "O",
-      "B-GROUP",
-      "O",
-      "O",
-      "B-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 19,
-    "prev_label": "B-SOURCE",
-    "label": "O",
-    "token": "[MP4]",
-    "row": 8,
-    "file_id": 8,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][05][1080P][BIG5][MP4]",
-    "context_tokens": [
-      "Mai",
-      "]",
-      "[05]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "context_labels": [
-      "I-TITLE",
-      "O",
-      "B-EPISODE",
-      "B-RESOLUTION",
-      "B-SOURCE",
-      "O"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 2,
-    "prev_label": "B-GROUP",
-    "label": "O",
-    "token": "]",
-    "row": 9,
-    "file_id": 9,
-    "filename": "[Airota][Sousou no Frieren][29][1080p AVC AAC][CHT]",
-    "context_tokens": [
-      "[",
-      "Airota",
-      "]",
-      "[",
-      "Sousou",
-      " ",
-      "no",
-      " "
-    ],
-    "context_labels": [
-      "O",
-      "B-GROUP",
-      "O",
-      "O",
-      "B-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 11,
-    "prev_label": "B-EPISODE",
-    "label": "O",
-    "token": "[1080p AVC AAC]",
-    "row": 9,
-    "file_id": 9,
-    "filename": "[Airota][Sousou no Frieren][29][1080p AVC AAC][CHT]",
-    "context_tokens": [
-      "no",
-      " ",
-      "Frieren",
-      "]",
-      "[29]",
-      "[1080p AVC AAC]",
-      "[CHT]"
-    ],
-    "context_labels": [
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "O",
-      "B-EPISODE",
-      "O",
-      "B-SOURCE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 2,
-    "prev_label": "B-GROUP",
-    "label": "O",
-    "token": "]",
-    "row": 10,
-    "file_id": 10,
-    "filename": "[Airota][Sousou no Frieren][30][1080p AVC AAC][CHT]",
-    "context_tokens": [
-      "[",
-      "Airota",
-      "]",
-      "[",
-      "Sousou",
-      " ",
-      "no",
-      " "
-    ],
-    "context_labels": [
-      "O",
-      "B-GROUP",
-      "O",
-      "O",
-      "B-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 11,
-    "prev_label": "B-EPISODE",
-    "label": "O",
-    "token": "[1080p AVC AAC]",
-    "row": 10,
-    "file_id": 10,
-    "filename": "[Airota][Sousou no Frieren][30][1080p AVC AAC][CHT]",
-    "context_tokens": [
-      "no",
-      " ",
-      "Frieren",
-      "]",
-      "[30]",
-      "[1080p AVC AAC]",
-      "[CHT]"
-    ],
-    "context_labels": [
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "O",
-      "B-EPISODE",
-      "O",
-      "B-SOURCE"
-    ]
-  },
-  {
-    "type": "B_DIRECT_TO_O",
-    "index": 2,
-    "prev_label": "B-GROUP",
-    "label": "O",
-    "token": "]",
-    "row": 11,
-    "file_id": 11,
-    "filename": "[Airota][Sousou no Frieren][31][1080p AVC AAC][CHT]",
-    "context_tokens": [
-      "[",
-      "Airota",
-      "]",
-      "[",
-      "Sousou",
-      " ",
-      "no",
-      " "
-    ],
-    "context_labels": [
-      "O",
-      "B-GROUP",
-      "O",
-      "O",
-      "B-TITLE",
-      "I-TITLE",
-      "I-TITLE",
-      "I-TITLE"
-    ]
-  }
-]
-```
-
-## Tokenizer Split And Alignment
-
-### Dataset tokens vs selected tokenizer mismatches
-```json
-[
-  {
-    "file_id": 2,
-    "filename": "[LoliHouse] Maid-san wa Taberu Dake - 07 [WebRip 1080p HEVC-10bit AAC ASSx2]",
-    "common_prefix": 0,
-    "dataset_tokens": [
-      "[",
-      "LoliHouse",
-      "]",
-      " ",
-      "Maid",
-      "-",
-      "san",
-      " ",
-      "wa",
-      " ",
-      "Taberu",
-      " ",
-      "Dake",
-      " ",
-      "-",
-      " ",
-      "07",
-      " ",
-      "[WebRip 1080p HEVC-10bit AAC ASSx2]"
-    ],
-    "tokenizer_tokens": [
-      "[LoliHouse]",
-      " ",
-      "Maid",
-      "-",
-      "san",
-      " ",
-      "wa",
-      " ",
-      "Taberu",
-      " ",
-      "Dake",
-      " ",
-      "-",
-      " ",
-      "07",
-      " ",
-      "[WebRip 1080p HEVC-10bit AAC ASSx2]"
-    ],
-    "dataset_len": 19,
-    "tokenizer_len": 17
-  },
-  {
-    "file_id": 3,
-    "filename": "[ANi] 異世界悠閒農家 2 - 06 [1080P][Baha][WEB-DL][AAC AVC][CHT]",
-    "common_prefix": 0,
-    "dataset_tokens": [
-      "[",
-      "ANi",
-      "]",
-      " ",
-      "異",
-      "世",
-      "界",
-      "悠",
-      "閒",
-      "農",
-      "家",
-      " ",
-      "2",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "tokenizer_tokens": [
-      "[ANi]",
-      " ",
-      "異",
-      "��",
-      "界",
-      "悠",
-      "閒",
-      "農",
-      "家",
-      " ",
-      "2",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "dataset_len": 23,
-    "tokenizer_len": 21
-  },
-  {
-    "file_id": 4,
-    "filename": "[ANi] 木頭風紀委員和迷你裙 JK 的故事 - 06 [1080P][Baha][WEB-DL][AAC AVC][CHT]",
-    "common_prefix": 0,
-    "dataset_tokens": [
-      "[",
-      "ANi",
-      "]",
-      " ",
-      "木",
-      "頭",
-      "風",
-      "紀",
-      "委",
-      "員",
-      "和",
-      "迷",
-      "你",
-      "裙",
-      " ",
-      "JK",
-      " ",
-      "的",
-      "故",
-      "事",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "tokenizer_tokens": [
-      "[ANi]",
-      " ",
-      "木",
-      "頭",
-      "風",
-      "紀",
-      "委",
-      "員",
-      "和",
-      "迷",
-      "你",
-      "裙",
-      " ",
-      "JK",
-      " ",
-      "的",
-      "故",
-      "事",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "dataset_len": 30,
-    "tokenizer_len": 28
-  },
-  {
-    "file_id": 5,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][05][1080P][GB][MP4]",
-    "common_prefix": 0,
-    "dataset_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " ",
-      "-",
-      " ",
-      "Haru",
-      " ",
-      "no",
-      " ",
-      "Mai",
-      "]",
-      "[05]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "tokenizer_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[05]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "dataset_len": 20,
-    "tokenizer_len": 6
-  },
-  {
-    "file_id": 6,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][06][1080P][GB][MP4]",
-    "common_prefix": 0,
-    "dataset_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " ",
-      "-",
-      " ",
-      "Haru",
-      " ",
-      "no",
-      " ",
-      "Mai",
-      "]",
-      "[06]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "tokenizer_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[06]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "dataset_len": 20,
-    "tokenizer_len": 6
-  },
-  {
-    "file_id": 7,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][06][1080P][BIG5][MP4]",
-    "common_prefix": 0,
-    "dataset_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " ",
-      "-",
-      " ",
-      "Haru",
-      " ",
-      "no",
-      " ",
-      "Mai",
-      "]",
-      "[06]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "tokenizer_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[06]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "dataset_len": 20,
-    "tokenizer_len": 6
-  },
-  {
-    "file_id": 8,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][05][1080P][BIG5][MP4]",
-    "common_prefix": 0,
-    "dataset_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " ",
-      "-",
-      " ",
-      "Haru",
-      " ",
-      "no",
-      " ",
-      "Mai",
-      "]",
-      "[05]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "tokenizer_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[05]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "dataset_len": 20,
-    "tokenizer_len": 6
-  },
-  {
-    "file_id": 9,
-    "filename": "[Airota][Sousou no Frieren][29][1080p AVC AAC][CHT]",
-    "common_prefix": 0,
-    "dataset_tokens": [
-      "[",
-      "Airota",
-      "]",
-      "[",
-      "Sousou",
-      " ",
-      "no",
-      " ",
-      "Frieren",
-      "]",
-      "[29]",
-      "[1080p AVC AAC]",
-      "[CHT]"
-    ],
-    "tokenizer_tokens": [
-      "[Airota]",
-      "[Sousou no Frieren]",
-      "[29]",
-      "[1080p AVC AAC]",
-      "[CHT]"
-    ],
-    "dataset_len": 13,
-    "tokenizer_len": 5
-  },
-  {
-    "file_id": 10,
-    "filename": "[Airota][Sousou no Frieren][30][1080p AVC AAC][CHT]",
-    "common_prefix": 0,
-    "dataset_tokens": [
-      "[",
-      "Airota",
-      "]",
-      "[",
-      "Sousou",
-      " ",
-      "no",
-      " ",
-      "Frieren",
-      "]",
-      "[30]",
-      "[1080p AVC AAC]",
-      "[CHT]"
-    ],
-    "tokenizer_tokens": [
-      "[Airota]",
-      "[Sousou no Frieren]",
-      "[30]",
-      "[1080p AVC AAC]",
-      "[CHT]"
-    ],
-    "dataset_len": 13,
-    "tokenizer_len": 5
-  },
-  {
-    "file_id": 11,
-    "filename": "[Airota][Sousou no Frieren][31][1080p AVC AAC][CHT]",
-    "common_prefix": 0,
-    "dataset_tokens": [
-      "[",
-      "Airota",
-      "]",
-      "[",
-      "Sousou",
-      " ",
-      "no",
-      " ",
-      "Frieren",
-      "]",
-      "[31]",
-      "[1080p AVC AAC]",
-      "[CHT]"
-    ],
-    "tokenizer_tokens": [
-      "[Airota]",
-      "[Sousou no Frieren]",
-      "[31]",
-      "[1080p AVC AAC]",
-      "[CHT]"
-    ],
-    "dataset_len": 13,
-    "tokenizer_len": 5
-  }
-]
-```
-
-### Split examples
-```json
-[
-  {
-    "file_id": 1,
-    "filename": "Witch.Hat.Atelier.S01E07.1080p.NF.WEB-DL.JPN.AAC2.0.H.264.MSubs-ToonsHub",
-    "dataset_tokens": [
-      "Witch",
-      ".",
-      "Hat",
-      ".",
-      "Atelier",
-      ".",
-      "S01",
-      "E07",
-      ".",
-      "1080p",
-      ".",
-      "NF",
-      ".",
-      "WEB-DL",
-      ".",
-      "JP",
-      "N",
-      ".",
-      "AAC",
-      "2",
-      ".",
-      "0",
-      ".",
-      "H.264",
-      ".",
-      "MSubs",
-      "-",
-      "ToonsHub"
-    ],
-    "diagnosed_tokens": [
-      "Witch",
-      ".",
-      "Hat",
-      ".",
-      "Atelier",
-      ".",
-      "S01",
-      "E07",
-      ".",
-      "1080p",
-      ".",
-      "NF",
-      ".",
-      "WEB-DL",
-      ".",
-      "JP",
-      "N",
-      ".",
-      "AAC",
-      "2",
-      ".",
-      "0",
-      ".",
-      "H.264",
-      ".",
-      "MSubs",
-      "-",
-      "ToonsHub"
-    ],
-    "regex_tokens": [
-      "Witch",
-      ".",
-      "Hat",
-      ".",
-      "Atelier",
-      ".",
-      "S01",
-      "E07",
-      ".",
-      "1080p",
-      ".",
-      "NF",
-      ".",
-      "WEB-DL",
-      ".",
-      "JP",
-      "N",
-      ".",
-      "AAC",
-      "2",
-      ".",
-      "0",
-      ".",
-      "H.264",
-      ".",
-      "MSubs",
-      "-",
-      "ToonsHub"
-    ],
-    "char_tokens": [
-      "W",
-      "i",
-      "t",
-      "c",
-      "h",
-      ".",
-      "H",
-      "a",
-      "t",
-      ".",
-      "A",
-      "t",
-      "e",
-      "l",
-      "i",
-      "e",
-      "r",
-      ".",
-      "S",
-      "0",
-      "1",
-      "E",
-      "0",
-      "7",
-      ".",
-      "1",
-      "0",
-      "8",
-      "0",
-      "p",
-      ".",
-      "N",
-      "F",
-      ".",
-      "W",
-      "E",
-      "B",
-      "-",
-      "D",
-      "L",
-      ".",
-      "J",
-      "P",
-      "N",
-      ".",
-      "A",
-      "A",
-      "C",
-      "2",
-      ".",
-      "0",
-      ".",
-      "H",
-      ".",
-      "2",
-      "6",
-      "4",
-      ".",
-      "M",
-      "S",
-      "u",
-      "b",
-      "s",
-      "-",
-      "T",
-      "o",
-      "o",
-      "n",
-      "s",
-      "H",
-      "u",
-      "b"
-    ]
-  },
-  {
-    "file_id": 2,
-    "filename": "[LoliHouse] Maid-san wa Taberu Dake - 07 [WebRip 1080p HEVC-10bit AAC ASSx2]",
-    "dataset_tokens": [
-      "[",
-      "LoliHouse",
-      "]",
-      " ",
-      "Maid",
-      "-",
-      "san",
-      " ",
-      "wa",
-      " ",
-      "Taberu",
-      " ",
-      "Dake",
-      " ",
-      "-",
-      " ",
-      "07",
-      " ",
-      "[WebRip 1080p HEVC-10bit AAC ASSx2]"
-    ],
-    "diagnosed_tokens": [
-      "[LoliHouse]",
-      " ",
-      "Maid",
-      "-",
-      "san",
-      " ",
-      "wa",
-      " ",
-      "Taberu",
-      " ",
-      "Dake",
-      " ",
-      "-",
-      " ",
-      "07",
-      " ",
-      "[WebRip 1080p HEVC-10bit AAC ASSx2]"
-    ],
-    "regex_tokens": [
-      "[LoliHouse]",
-      " ",
-      "Maid",
-      "-",
-      "san",
-      " ",
-      "wa",
-      " ",
-      "Taberu",
-      " ",
-      "Dake",
-      " ",
-      "-",
-      " ",
-      "07",
-      " ",
-      "[WebRip 1080p HEVC-10bit AAC ASSx2]"
-    ],
-    "char_tokens": [
-      "[",
-      "L",
-      "o",
-      "l",
-      "i",
-      "H",
-      "o",
-      "u",
-      "s",
-      "e",
-      "]",
-      " ",
-      "M",
-      "a",
-      "i",
-      "d",
-      "-",
-      "s",
-      "a",
-      "n",
-      " ",
-      "w",
-      "a",
-      " ",
-      "T",
-      "a",
-      "b",
-      "e",
-      "r",
-      "u",
-      " ",
-      "D",
-      "a",
-      "k",
-      "e",
-      " ",
-      "-",
-      " ",
-      "0",
-      "7",
-      " ",
-      "[",
-      "W",
-      "e",
-      "b",
-      "R",
-      "i",
-      "p",
-      " ",
-      "1",
-      "0",
-      "8",
-      "0",
-      "p",
-      " ",
-      "H",
-      "E",
-      "V",
-      "C",
-      "-",
-      "1",
-      "0",
-      "b",
-      "i",
-      "t",
-      " ",
-      "A",
-      "A",
-      "C",
-      " ",
-      "A",
-      "S",
-      "S",
-      "x",
-      "2",
-      "]"
-    ]
-  },
-  {
-    "file_id": 3,
-    "filename": "[ANi] 異世界悠閒農家 2 - 06 [1080P][Baha][WEB-DL][AAC AVC][CHT]",
-    "dataset_tokens": [
-      "[",
-      "ANi",
-      "]",
-      " ",
-      "異",
-      "世",
-      "界",
-      "悠",
-      "閒",
-      "農",
-      "家",
-      " ",
-      "2",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "diagnosed_tokens": [
-      "[ANi]",
-      " ",
-      "異",
-      "世",
-      "界",
-      "悠",
-      "閒",
-      "農",
-      "家",
-      " ",
-      "2",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "regex_tokens": [
-      "[ANi]",
-      " ",
-      "異",
-      "世",
-      "界",
-      "悠",
-      "閒",
-      "農",
-      "家",
-      " ",
-      "2",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "char_tokens": [
-      "[",
-      "A",
-      "N",
-      "i",
-      "]",
-      " ",
-      "異",
-      "世",
-      "界",
-      "悠",
-      "閒",
-      "農",
-      "家",
-      " ",
-      "2",
-      " ",
-      "-",
-      " ",
-      "0",
-      "6",
-      " ",
-      "[",
-      "1",
-      "0",
-      "8",
-      "0",
-      "P",
-      "]",
-      "[",
-      "B",
-      "a",
-      "h",
-      "a",
-      "]",
-      "[",
-      "W",
-      "E",
-      "B",
-      "-",
-      "D",
-      "L",
-      "]",
-      "[",
-      "A",
-      "A",
-      "C",
-      " ",
-      "A",
-      "V",
-      "C",
-      "]",
-      "[",
-      "C",
-      "H",
-      "T",
-      "]"
-    ]
-  },
-  {
-    "file_id": 4,
-    "filename": "[ANi] 木頭風紀委員和迷你裙 JK 的故事 - 06 [1080P][Baha][WEB-DL][AAC AVC][CHT]",
-    "dataset_tokens": [
-      "[",
-      "ANi",
-      "]",
-      " ",
-      "木",
-      "頭",
-      "風",
-      "紀",
-      "委",
-      "員",
-      "和",
-      "迷",
-      "你",
-      "裙",
-      " ",
-      "JK",
-      " ",
-      "的",
-      "故",
-      "事",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "diagnosed_tokens": [
-      "[ANi]",
-      " ",
-      "木",
-      "頭",
-      "風",
-      "紀",
-      "委",
-      "員",
-      "和",
-      "迷",
-      "你",
-      "裙",
-      " ",
-      "JK",
-      " ",
-      "的",
-      "故",
-      "事",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "regex_tokens": [
-      "[ANi]",
-      " ",
-      "木",
-      "頭",
-      "風",
-      "紀",
-      "委",
-      "員",
-      "和",
-      "迷",
-      "你",
-      "裙",
-      " ",
-      "JK",
-      " ",
-      "的",
-      "故",
-      "事",
-      " ",
-      "-",
-      " ",
-      "06",
-      " ",
-      "[1080P]",
-      "[Baha]",
-      "[WEB-DL]",
-      "[AAC AVC]",
-      "[CHT]"
-    ],
-    "char_tokens": [
-      "[",
-      "A",
-      "N",
-      "i",
-      "]",
-      " ",
-      "木",
-      "頭",
-      "風",
-      "紀",
-      "委",
-      "員",
-      "和",
-      "迷",
-      "你",
-      "裙",
-      " ",
-      "J",
-      "K",
-      " ",
-      "的",
-      "故",
-      "事",
-      " ",
-      "-",
-      " ",
-      "0",
-      "6",
-      " ",
-      "[",
-      "1",
-      "0",
-      "8",
-      "0",
-      "P",
-      "]",
-      "[",
-      "B",
-      "a",
-      "h",
-      "a",
-      "]",
-      "[",
-      "W",
-      "E",
-      "B",
-      "-",
-      "D",
-      "L",
-      "]",
-      "[",
-      "A",
-      "A",
-      "C",
-      " ",
-      "A",
-      "V",
-      "C",
-      "]",
-      "[",
-      "C",
-      "H",
-      "T",
-      "]"
-    ]
-  },
-  {
-    "file_id": 5,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][05][1080P][GB][MP4]",
-    "dataset_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " ",
-      "-",
-      " ",
-      "Haru",
-      " ",
-      "no",
-      " ",
-      "Mai",
-      "]",
-      "[05]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "diagnosed_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[05]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "regex_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[05]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "char_tokens": [
-      "[",
-      "K",
-      "i",
-      "s",
-      "s",
-      "S",
-      "u",
-      "b",
-      "]",
-      "[",
-      "S",
-      "h",
-      "u",
-      "n",
-      "k",
-      "a",
-      "s",
-      "h",
-      "u",
-      "u",
-      "t",
-      "o",
-      "u",
-      " ",
-      "D",
-      "a",
-      "i",
-      "k",
-      "o",
-      "u",
-      "s",
-      "h",
-      "a",
-      " ",
-      "-",
-      " ",
-      "H",
-      "a",
-      "r",
-      "u",
-      " ",
-      "n",
-      "o",
-      " ",
-      "M",
-      "a",
-      "i",
-      "]",
-      "[",
-      "0",
-      "5",
-      "]",
-      "[",
-      "1",
-      "0",
-      "8",
-      "0",
-      "P",
-      "]",
-      "[",
-      "G",
-      "B",
-      "]",
-      "[",
-      "M",
-      "P",
-      "4",
-      "]"
-    ]
-  },
-  {
-    "file_id": 6,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][06][1080P][GB][MP4]",
-    "dataset_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " ",
-      "-",
-      " ",
-      "Haru",
-      " ",
-      "no",
-      " ",
-      "Mai",
-      "]",
-      "[06]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "diagnosed_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[06]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "regex_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[06]",
-      "[1080P]",
-      "[GB]",
-      "[MP4]"
-    ],
-    "char_tokens": [
-      "[",
-      "K",
-      "i",
-      "s",
-      "s",
-      "S",
-      "u",
-      "b",
-      "]",
-      "[",
-      "S",
-      "h",
-      "u",
-      "n",
-      "k",
-      "a",
-      "s",
-      "h",
-      "u",
-      "u",
-      "t",
-      "o",
-      "u",
-      " ",
-      "D",
-      "a",
-      "i",
-      "k",
-      "o",
-      "u",
-      "s",
-      "h",
-      "a",
-      " ",
-      "-",
-      " ",
-      "H",
-      "a",
-      "r",
-      "u",
-      " ",
-      "n",
-      "o",
-      " ",
-      "M",
-      "a",
-      "i",
-      "]",
-      "[",
-      "0",
-      "6",
-      "]",
-      "[",
-      "1",
-      "0",
-      "8",
-      "0",
-      "P",
-      "]",
-      "[",
-      "G",
-      "B",
-      "]",
-      "[",
-      "M",
-      "P",
-      "4",
-      "]"
-    ]
-  },
-  {
-    "file_id": 7,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][06][1080P][BIG5][MP4]",
-    "dataset_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " ",
-      "-",
-      " ",
-      "Haru",
-      " ",
-      "no",
-      " ",
-      "Mai",
-      "]",
-      "[06]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "diagnosed_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[06]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "regex_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[06]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "char_tokens": [
-      "[",
-      "K",
-      "i",
-      "s",
-      "s",
-      "S",
-      "u",
-      "b",
-      "]",
-      "[",
-      "S",
-      "h",
-      "u",
-      "n",
-      "k",
-      "a",
-      "s",
-      "h",
-      "u",
-      "u",
-      "t",
-      "o",
-      "u",
-      " ",
-      "D",
-      "a",
-      "i",
-      "k",
-      "o",
-      "u",
-      "s",
-      "h",
-      "a",
-      " ",
-      "-",
-      " ",
-      "H",
-      "a",
-      "r",
-      "u",
-      " ",
-      "n",
-      "o",
-      " ",
-      "M",
-      "a",
-      "i",
-      "]",
-      "[",
-      "0",
-      "6",
-      "]",
-      "[",
-      "1",
-      "0",
-      "8",
-      "0",
-      "P",
-      "]",
-      "[",
-      "B",
-      "I",
-      "G",
-      "5",
-      "]",
-      "[",
-      "M",
-      "P",
-      "4",
-      "]"
-    ]
-  },
-  {
-    "file_id": 8,
-    "filename": "[KissSub][Shunkashuutou Daikousha - Haru no Mai][05][1080P][BIG5][MP4]",
-    "dataset_tokens": [
-      "[",
-      "KissSub",
-      "]",
-      "[",
-      "Shunkashuutou",
-      " ",
-      "Daikousha",
-      " ",
-      "-",
-      " ",
-      "Haru",
-      " ",
-      "no",
-      " ",
-      "Mai",
-      "]",
-      "[05]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "diagnosed_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[05]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "regex_tokens": [
-      "[KissSub]",
-      "[Shunkashuutou Daikousha - Haru no Mai]",
-      "[05]",
-      "[1080P]",
-      "[BIG5]",
-      "[MP4]"
-    ],
-    "char_tokens": [
-      "[",
-      "K",
-      "i",
-      "s",
-      "s",
-      "S",
-      "u",
-      "b",
-      "]",
-      "[",
-      "S",
-      "h",
-      "u",
-      "n",
-      "k",
-      "a",
-      "s",
-      "h",
-      "u",
-      "u",
-      "t",
-      "o",
-      "u",
-      " ",
-      "D",
-      "a",
-      "i",
-      "k",
-      "o",
-      "u",
-      "s",
-      "h",
-      "a",
-      " ",
-      "-",
-      " ",
-      "H",
-      "a",
-      "r",
-      "u",
-      " ",
-      "n",
-      "o",
-      " ",
-      "M",
-      "a",
-      "i",
-      "]",
-      "[",
-      "0",
-      "5",
-      "]",
-      "[",
-      "1",
-      "0",
-      "8",
-      "0",
-      "P",
-      "]",
-      "[",
-      "B",
-      "I",
-      "G",
-      "5",
-      "]",
-      "[",
-      "M",
-      "P",
-      "4",
-      "]"
-    ]
-  }
-]
-```
-
-### Vocabulary coverage
-```json
-{
-  "total": 85312,
-  "unk": 5900,
-  "unk_rate": 0.06915791447861966,
-  "top_unk": [
-    [
-      "(BDRip 720p x264)",
-      66
-    ],
-    [
-      "Partie",
-      59
-    ],
-    [
-      "incantevole",
-      54
-    ],
-    [
-      "Muxed",
-      54
-    ],
-    [
-      "nonscordarmi",
-      54
-    ],
-    [
-      "NEET",
-      52
-    ],
-    [
-      "Dousei",
-      52
-    ],
-    [
-      "[krikoun68]",
-      52
-    ],
-    [
-      "[Blu-Ray - MUX - 960p - x264 - AC3 ITA-JAP - SUB ITA]",
-      51
-    ],
-    [
-      "CTR",
-      45
-    ],
-    [
-      "joseol",
-      45
-    ],
-    [
-      "e99",
-      45
-    ],
-    [
-      "(1440x1080 h264 AC3 AAC)",
-      45
-    ],
-    [
-      "VERS",
-      37
-    ],
-    [
-      "脙",
-      37
-    ],
-    [
-      "Shunkashuutou",
-      36
-    ],
-    [
-      "Daikousha",
-      36
-    ],
-    [
-      "houbatsu",
-      36
-    ],
-    [
-      "DEFINITIVA",
-      36
-    ],
-    [
-      "Crash",
-      35
-    ],
-    [
-      "Realm",
-      31
-    ],
-    [
-      "UHD",
-      31
-    ],
-    [
-      "[BDrip 1080P HEVC-10bit AAC]",
-      29
-    ],
-    [
-      "Choroi",
-      28
-    ],
-    [
-      "완",
-      28
-    ]
-  ]
-}
-```
-
-## Train Inference Tokenizer Comparison
-
-- Model dir: `checkpoints\dmhy-finetune\final`
-- Model tokenizer variant: `regex`
-- Dataset tokenizer variant: `regex`
-- Diagnostic tokenizer variant: `regex`
-- Model tokenizer vocab size: 3,000
-- Diagnostic tokenizer vocab size: 8,000
-
-If dataset and model tokenizer variants differ, validation loss can be low while real inference sees different token IDs and boundaries.
-
-## Model Confusion Analysis
-
-- Evaluated samples: 128
-- Entity precision: 0.9568
-- Entity recall: 0.9530
-- Entity F1: 0.9549
-
-### Boundary error classes
-- `B-boundary`: 26 (56.52%)
-- `entity-type`: 20 (43.48%)
-
-### Top token-label confusions
-| true | pred | count |
-| --- | --- | --- |
-| O | I-TITLE | 17 |
-| O | B-EPISODE | 6 |
-| B-SOURCE | O | 4 |
-| I-TITLE | O | 3 |
-| B-EPISODE | O | 3 |
-| B-SEASON | O | 2 |
-| B-RESOLUTION | B-SOURCE | 2 |
-| B-EPISODE | I-TITLE | 2 |
-| O | B-TITLE | 2 |
-| B-TITLE | I-TITLE | 2 |
-| O | B-SOURCE | 1 |
-| B-SEASON | I-TITLE | 1 |
-| O | B-SEASON | 1 |
-
-### Top entity-type confusions
-| true | pred | count |
-| --- | --- | --- |
-| O | TITLE | 19 |
-| O | EPISODE | 6 |
-| SOURCE | O | 4 |
-| TITLE | O | 3 |
-| EPISODE | O | 3 |
-| SEASON | O | 2 |
-| RESOLUTION | SOURCE | 2 |
-| EPISODE | TITLE | 2 |
-| O | SOURCE | 1 |
-| SEASON | TITLE | 1 |
-| O | SEASON | 1 |
-
-### Seqeval report
-```text
-              precision    recall  f1-score   support
-
-     EPISODE     0.9535    0.9609    0.9572       128
-       GROUP     1.0000    1.0000    1.0000        53
-  RESOLUTION     1.0000    0.9545    0.9767        44
-      SEASON     0.9630    0.8966    0.9286        29
-      SOURCE     0.9703    0.9608    0.9655       102
-     SPECIAL     1.0000    1.0000    1.0000         5
-       TITLE     0.9211    0.9333    0.9272       150
-
-   micro avg     0.9568    0.9530    0.9549       511
-   macro avg     0.9725    0.9580    0.9650       511
-weighted avg     0.9571    0.9530    0.9550       511
-
-```
-
-## Recommended Pipeline
-
-1. Use one tokenizer variant end to end and save it in the checkpoint metadata.
-2. Prefer char-level or a deterministic hybrid tokenizer for DMHY filenames; avoid generic subword tokenization for labels.
-3. For char-level runs, use `--tokenizer char --max-seq-length 128` with `vocab.char.json`.
-4. Add CRF decoding or constrained BIO decoding so illegal I-X transitions and impossible boundary jumps are blocked.
-5. Keep rule-assisted post-processing for high-confidence structural anchors: leading group bracket, ` - 07`, `S01E07`, source, and resolution.
-6. Track entity-level F1 and field exact-match on real filenames; do not accept low validation loss alone.