spencernorton commited on
Commit
7e5aa39
·
verified ·
1 Parent(s): 702de15

Release Armiger v3.3.0 (v3.0-gold-r13.3) model weights, updated charts, and concise model cards

Browse files
.gitattributes CHANGED
@@ -41,3 +41,4 @@ assets/armiger_vs_torrent_classifiers_6dims.png filter=lfs diff=lfs merge=lfs -t
41
  assets/armiger_categorical_breakdown.png filter=lfs diff=lfs merge=lfs -text
42
  assets/armiger_vs_llms_overall_and_6dims.png filter=lfs diff=lfs merge=lfs -text
43
  assets/armiger_vs_llms_cost_comparison.png filter=lfs diff=lfs merge=lfs -text
 
 
41
  assets/armiger_categorical_breakdown.png filter=lfs diff=lfs merge=lfs -text
42
  assets/armiger_vs_llms_overall_and_6dims.png filter=lfs diff=lfs merge=lfs -text
43
  assets/armiger_vs_llms_cost_comparison.png filter=lfs diff=lfs merge=lfs -text
44
+ assets/armiger_overview.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -21,28 +21,18 @@ tags:
21
  - cpu-inference
22
  ---
23
 
24
- # Armiger `v3.2.1` (Model Bundle `v3.0.0` / `v3.0-gold-r5`) — 15.7 MB INT8 ONNX CPU Classifier
25
 
26
- **Release:** [`v3.2.1`](https://github.com/spencernorton/armiger/releases/tag/v3.2.1) · **Model Version:** `3.0.0` (`v3.0-gold-r5`) · **Artifact:** `15.7 MB` (`armiger.int8.onnx` / `armiger.onnx`, per-channel asymmetric INT8 with FP32 embeddings) · **CPU latency:** `2.44 ms/item` (`326 items/s` batch-64, M4 Pro CPU) · **License:** Apache-2.0
27
 
28
- **Armiger (`NorviTech/armiger`)** is the CPU-only INT8 edition of the 9-head byte-level RoPE Transformer (`8,295,882` parameters). It needs only `numpy` + `onnxruntime` and keeps **`99.90%`** parity with the 32.5 MB FP32 model. Library and container release **`3.2.1`** uses model bundle **`3.0.0`**. For FP32 weights and all 6 plugins (`SwarmSurvivalNet`, `SwarmPriors`, `ZeroCueCatalog`, `AudioLanguageGuard`, `CalibrationAndDHT`, `ArmigerDHTDBSync`), use [`NorviTech/armiger-ultra`](https://huggingface.co/NorviTech/armiger-ultra).
29
 
30
- ## Neural Outputs & Benchmark Summary
31
 
32
- - **9 outputs, one CPU pass:** `category` (11-class), `adult`, `junk`/malware, `anime`, `english_subs`, spoken `lang`, `title_language`, `owner_scope`, and per-byte `span_bio` title/year/episode tags via a 7-tag Viterbi mask (`0` illegal transitions).
33
- - **INT8 accuracy (all 3 suites):**
34
- - **`Suite B LOTO N=1,025`:** **`96.85%`** 6-dimension composite (`95% CI [96.33%, 97.36%]` with plugins) vs. Gemini Flash (`94.56%`), OpenAI GPT Fast Tier (`93.49%`, 5-dim; Junk N/A (Unprompted)), Claude Haiku 4.5 (`92.88%`).
35
- - **`W5A-Test N=703` (Rule-3 Gold):** **`97.01%`** 11-class accuracy (`95% Wilson CI [95.47%, 98.04%]`); **`96.87%`** 6-core-head exact match.
36
- - **`R5-Test N=996` (7-key zero-leakage):** **`91.16%`** INT8 9-class accuracy (FP32 `91.27%`; `99.90%` parity), **`99.22%`** Title Language (`n=511`), **`99.09%`** English Subtitles (`n=110`), **`98.90%`** Owner Scope (`n=996`), **`90.32%`** BIO Balanced Byte Accuracy (`n=882`), **`85.84%`** all-heads exact match (`n=996`).
37
-
38
- ![Suite B LOTO Overall & 6-Dimension Comparison](https://huggingface.co/NorviTech/armiger-ultra/resolve/main/assets/fig_v3_vs_llms_1_overall_and_6dims_loto.png)
39
-
40
- ## Quickstart (Standalone CPU ONNX Runtime)
41
-
42
- ### Option A — Install the Python package (`v3.2.1`) with verified model download
43
 
44
  ```bash
45
- python -m pip install "armiger @ git+https://github.com/spencernorton/armiger.git@v3.2.1"
46
  python -m armiger download --directory ./models-cpu --variant cpu
47
  ```
48
 
@@ -62,31 +52,9 @@ res = model.predict({
62
  print(res["heads"]["category"], res["matching_identity"], res["mondrian_gates"]["passed_all"])
63
  ```
64
 
65
- ### Option B — Run directly from a checkout of this Hugging Face repository
66
-
67
- ```python
68
- from armiger_inference import ArmigerStandaloneONNX
69
-
70
- model = ArmigerStandaloneONNX() # auto-loads armiger.onnx, vocab.json, calibration.json
71
- res = model.predict({
72
- "name": "Tears.of.Steel.2012.1080p.BluRay.x264-OPEN",
73
- "size": 8_500_000_000,
74
- "files": [{"path": "Tears.of.Steel.2012.mkv", "size": 8_500_000_000}],
75
- })
76
- print(res["heads"]["category"], res["matching_identity"], res["mondrian_gates"]["passed_all"])
77
- ```
78
-
79
- ## Ready-Made Docker Setups & Integrations (`v3.2.1`)
80
 
81
- - **Full FP32 + 6-plugin suite:** [`NorviTech/armiger-ultra`](https://huggingface.co/NorviTech/armiger-ultra)
82
- - **Core package, containers & evaluation harness (`v3.2.1`):** [`spencernorton/armiger`](https://github.com/spencernorton/armiger) · **License:** Apache-2.0
83
- - **[Bitmagnet + Armiger Ultra](https://github.com/spencernorton/armiger/blob/main/examples/bitmagnet/README.md):** Two-line Docker Compose setup (`include: [oci://ghcr.io/spencernorton/armiger-bitmagnet-stack:3.2.1]`) · [Environment-Variable Reference](https://github.com/spencernorton/armiger/blob/main/docs/bitmagnet-configuration.md)
84
- - **[Rats Search + Armiger Ultra](https://github.com/spencernorton/armiger/blob/main/examples/rats-search/README.md):** Two-line Docker Compose setup (`include: [oci://ghcr.io/spencernorton/armiger-rats-search-stack:3.2.1]`)
85
- - **[Standalone Armiger Ultra HTTP Container](https://github.com/spencernorton/armiger/blob/main/docs/quickstart.md):** Prebuilt multi-arch image `ghcr.io/spencernorton/armiger:3.2.1` with `/v1/classify` and `/healthz`
86
- - **[Python API & Component Reference](https://github.com/spencernorton/armiger/blob/main/docs/reference.md)** · **[Evaluation & Limitations](https://github.com/spencernorton/armiger/blob/main/docs/evaluation.md)**
87
- - **Standalone sidecars & connectors (`v1.1.1`):** [`spencernorton/Armiger-Connectors`](https://github.com/spencernorton/Armiger-Connectors)
88
- - **Stock Bitmagnet (`bitmagnet-io/bitmagnet`):** [`bitmagnet/bitmagnet_connector.py`](https://github.com/spencernorton/Armiger-Connectors/blob/main/bitmagnet/bitmagnet_connector.py) — polls untagged torrents via GraphQL, runs vectorized Armiger inference, prunes `.exe`/`.scr`/`.lnk` malware and honeypot traps, writes `armiger-*` tags.
89
- - **Stock Rats Search (`librats/rats-search`):** [`rats-search/rats_search_connector.py`](https://github.com/spencernorton/Armiger-Connectors/blob/main/rats-search/rats_search_connector.py) — drop-in REST/SQLite sidecar classifier and safety filter.
90
- - **DH-Waypoint `processorapi/v1` plugin** ([`spencernorton/DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint)): [`dh-waypoint/`](https://github.com/spencernorton/Armiger-Connectors/tree/main/dh-waypoint) — admission (`/v1/admit`), batch (`/v1/classify`), 4-tier feedback (`/v1/feedback`).
91
- - **Torznab pre-grab proxy (`Prowlarr`/`Sonarr`/`Radarr`/`Lidarr`/`Readarr`):** [`arr-stack/armiger_torznab_proxy.py`](https://github.com/spencernorton/Armiger-Connectors/blob/main/arr-stack/armiger_torznab_proxy.py) — filters malware lures, dead swarms, and unrequested foreign-audio releases before grab.
92
- - **Post-import `*Arr`/qBittorrent webhook:** [`arr-stack/armiger_arr_webhook_receiver.py`](https://github.com/spencernorton/Armiger-Connectors/blob/main/arr-stack/armiger_arr_webhook_receiver.py) — `ffprobe` ground truth for local conformal calibration and opt-in anonymous BEP-44/46 updates.
 
21
  - cpu-inference
22
  ---
23
 
24
+ # Armiger `v3.3.0` — INT8 ONNX CPU Classifier
25
 
26
+ **Release:** [`v3.3.0`](https://github.com/spencernorton/armiger/releases/tag/v3.3.0) · **Artifact:** `17.6 MB` INT8 ONNX (`armiger.int8.onnx` / `armiger.onnx`) · **Parameters:** `8.05M` · **License:** Apache-2.0
27
 
28
+ **Armiger (`NorviTech/armiger`)** is the standalone CPU INT8 edition of the byte-level RoPE Transformer (`numpy` + `onnxruntime` only). For FP32 weights, pre-built `armiger-ultra` containers, and the 6-plugin pipeline, see [`NorviTech/armiger-ultra`](https://huggingface.co/NorviTech/armiger-ultra).
29
 
30
+ ![Armiger v3.3.0 Overview](assets/armiger_overview.png)
31
 
32
+ ## Quickstart
 
 
 
 
 
 
 
 
 
 
33
 
34
  ```bash
35
+ python -m pip install "armiger @ git+https://github.com/spencernorton/armiger.git@v3.3.0"
36
  python -m armiger download --directory ./models-cpu --variant cpu
37
  ```
38
 
 
52
  print(res["heads"]["category"], res["matching_identity"], res["mondrian_gates"]["passed_all"])
53
  ```
54
 
55
+ ## Links & Ecosystem
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
 
57
+ - **Full FP32 + 6-Plugin Suite:** [`NorviTech/armiger-ultra`](https://huggingface.co/NorviTech/armiger-ultra)
58
+ - **Core Package, Containers & Evaluation (`v3.3.0`):** [`spencernorton/armiger`](https://github.com/spencernorton/armiger) (`ghcr.io/spencernorton/armiger-ultra:3.3.0`)
59
+ - **Indexer Sidecars (Bitmagnet, Rats Search, `*Arr` Torznab):** [`spencernorton/Armiger-Connectors`](https://github.com/spencernorton/Armiger-Connectors)
60
+ - **DH-Waypoint DHT Crawler & Indexer:** [`spencernorton/DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint)
 
 
 
 
 
 
 
 
_r12_features_base.py ADDED
@@ -0,0 +1,932 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Standalone Armiger v2.1 / Armiger Ultra ONNX Featurizer & Inference Runtime (`armiger_inference.py`).
2
+
3
+ Implements exact feature parity with the Armiger v2.1 training pipeline (`sift/prepare.py` +
4
+ `enrich_side21_v2` 36-dim `stats` vector + `vocab.json` 4,096-extension lookup + CRC32
5
+ digit-folded path-word hashing) with zero external codebase dependencies beyond `numpy`
6
+ and `onnxruntime`.
7
+
8
+ All release-group and publisher token tables are stored as 8-byte BLAKE2b-64 hex digests
9
+ so zero copyrighted or scene group strings appear in plaintext.
10
+
11
+ R12 Session 2: derivative-group file-tree features
12
+ (`U_deriv_groups`, `E_episode_keys`, `P_multipart_idx`, `s_max`) are computed by
13
+ `compute_derivative_group_features` and appended to the side vector at
14
+ `side[24..27]`; negated subtitle clauses are stripped before the `side[13]`
15
+ subtitle cue; opaque top-level names gain a `|FILES: ...` primary-media suffix
16
+ in `name_bytes`. Legacy `side[0..23]` semantics are unchanged (only previously
17
+ dead `side[13]` becomes live), so `stats[:36]` remains the legacy R10/R11
18
+ prefix. Training (R12 Sessions 5 & 7) and inference must import the shared
19
+ `SIDE_DIM` / `STATS_DIM` constants below instead of hard-coding `24` / `36`.
20
+ """
21
+ from __future__ import annotations
22
+
23
+ import hashlib
24
+ import json
25
+ import math
26
+ import os
27
+ from pathlib import Path
28
+ import re
29
+ from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple
30
+ import zlib
31
+
32
+ try:
33
+ import numpy as np
34
+ except ImportError:
35
+ np = None # type: ignore[assignment]
36
+
37
+ SPAN_BIO_CLASSES: Tuple[str, ...] = (
38
+ "O",
39
+ "B-TITLE",
40
+ "I-TITLE",
41
+ "B-YEAR",
42
+ "I-YEAR",
43
+ "B-EP",
44
+ "I-EP",
45
+ )
46
+
47
+ # Side/stats dimensionality contract (R12 Session 2). `stats = base_stats + side`.
48
+ BASE_STATS_DIM: int = 12
49
+ LEGACY_SIDE_DIM: int = 24
50
+ SIDE_DIM: int = 28 # 24 legacy slots + 4 derivative-group features at side[24..27]
51
+ N_SIDE: int = SIDE_DIM
52
+ STATS_DIM: int = BASE_STATS_DIM + SIDE_DIM # 40 (legacy R10/R11 width was 36)
53
+ LEGACY_STATS_DIM: int = BASE_STATS_DIM + LEGACY_SIDE_DIM # 36
54
+ SIDE_FEATURE_NAMES: Tuple[str, ...] = (
55
+ "ebook_share", "pdf_share", "m4b_share", "comic_share", "console_share",
56
+ "app_share", "exec_share", "iso_share", "archive_share", "html_share",
57
+ "script_ext_present", "trap_flag", "small_video_flag", "subtitle_cue",
58
+ "sub_only_group", "anime_keyword", "dub_keyword", "game_group",
59
+ "software_group", "course_keyword", "book_comic_keyword",
60
+ "lang_english", "lang_non_english", "lang_unknown",
61
+ "deriv_u_log", "deriv_e_log", "deriv_p_multipart", "deriv_s_max",
62
+ )
63
+
64
+ NEURAL_HEAD_CLASSES: Dict[str, List[str]] = {
65
+ "category": [
66
+ "movie", "tv_show", "xxx", "music", "comic",
67
+ "game", "audiobook", "software", "ebook", "course", "unknown",
68
+ ],
69
+ "adult": ["non_adult", "adult"],
70
+ "junk": ["clean", "junk"],
71
+ "anime": ["non_anime", "anime"],
72
+ "english_subs": ["no_eng_subs", "has_eng_subs", "unknown"],
73
+ "lang": ["english", "non_english", "multi_or_unknown"],
74
+ "owner_scope": ["allowed_type", "excluded"],
75
+ "title_language": ["english_title", "non_english_title"],
76
+ "span_bio": list(SPAN_BIO_CLASSES),
77
+ }
78
+
79
+ _VOCAB_EXT_MAP: Optional[Dict[str, int]] = None
80
+ _PREP_KIND_EXTS = {
81
+ 0: frozenset("mkv mp4 avi ts m2ts mts vob wmv mpg mpeg mov flv rmvb rm m4v divx webm 3gp asf f4v ogv mxf m2v bik".split()),
82
+ 1: frozenset("mp3 flac m4a m4b aac ogg opus wav wma ape alac aif aiff aifc ac3 dts mka mpc wv shn mid midi dff dsf".split()),
83
+ 2: frozenset("zip rar 7z gz tgz bz2 xz zst lz lzma tar cab iso img dmg mdf mds nrg wim par par2 z".split()),
84
+ 3: frozenset("exe msi apk ipa dll bat cmd sh vbs ps1 com scr jar deb rpm appimage run lnk".split()),
85
+ 4: frozenset("jpg jpeg png gif webp bmp tif tiff avif jxl heic heif psd ai svg ico tga cr2 nef arw dng raw eps".split()),
86
+ 5: frozenset("pdf epub mobi azw azw3 djvu djv fb2 txt rtf doc docx odt xls xlsx ppt pptx csv chm cbz cbr cb7 html htm mht mhtml md tex lit".split()),
87
+ 6: frozenset("srt ass ssa sub idx vtt sup smi sbv ttml dfxp usf scc stl mks".split()),
88
+ }
89
+ _PREP_EXT_TO_KIND = {e: k for k, es in _PREP_KIND_EXTS.items() for e in es}
90
+ _PREP_SPLIT_ARC_RE = re.compile(r"[rsz]\d{2,3}|\d{3}")
91
+ _PREP_WORD_RE = re.compile(r"[^\W_]+")
92
+ _PREP_DIGIT_RE = re.compile(r"\d")
93
+ _PREP_SAMPLE_RE = re.compile(r"(?<![a-z])samples?(?![a-z])", re.IGNORECASE)
94
+
95
+ # BLAKE2b-64 digest sets for release-tag disambiguation (zero plaintext group names)
96
+ _SUB_ONLY_GRP_H = frozenset(['00d47045a1f1cd70', '0380c309e464115a', '0665f92d474878ff', '0f8f3809c6074d35', '1bc080c5d14b4eae', '1bf62e9b91086379', '22325081668ed28b', '26503f6826942362', '33b21e80d5dbfa62', '3e370e64b7539495', '4297edae7e25ba02', '4491be5b41a66832', '44e08227cfda9421', '4b956de9c462ee70', '519230ab4d0911f7', '581a8006ea8bd801', '65dfe6849ae4bb0d', '66ced9df20d8f558', '6a437c446b5778d6', '6cff013c3e752ac8', '6d3b6a62beedc878', '7bc52c5f863149cc', '867c636203d0524d', 'a7637cb5c531d8e9', 'b61d75b73d8e0c59', 'bed3322685febfa5', 'c603a601b7cb9d4c', 'd4eaba2bc7ef10d6', 'e20a8d50ddf1aa2e', 'e3fd326207da261c', 'fc94bcdf9fcb0164'])
97
+ _DUAL_DUB_GRP_H = frozenset(['16bdcdbbbd493d9e', '17addcc669201a7b', '1cb486cd76cea8c6', '1cc2f6b1aa24d6f7', '236a71bfa2d47ca3', '24eee113ee4bddb0', '341556a463a4784b', '3658912d4f05f762', '6149de24f7784f27', '627dcb38527b8e4c', '6f09d409143a679c', '727c5e1f0f05ad51', '8b7c320e1df4bf3e', '9e64e0c8c213bf00', 'ac3675ff37aaf11e', 'b2c26515563660f5', 'bd672057ff15b985', 'd9621931df7eb664', 'e20db1736162ba0c', 'e2ee075c6b5545ad', 'eb446df9e0b5f645', 'f7a175e4cf1907fb', 'fe0fe2b14bb48506'])
98
+ _GAME_GRP_H = frozenset(['03f22fe2a782da12', '0ae36a7939ac0be1', '1217fb0deda20335', '1c8f3166691fa2c6', '306372c75c8ceaa2', '32ba70bb6b5ae022', '4509f87fe97d6ed9', '4afe068be7dcf016', '6f7a14ce24f000a0', '7403ff2453c8a173', '7de6d237db23b5fa', '9641051915abdbb6', 'a491e84e39e2064e', 'a65a1816debe0915', 'aa89d2a9d5d18f01', 'aaac60aa2d75341b', 'bb98906accdd6e05', 'c200372b38fb6d26', 'c430ddffd526c993', 'cc81415c4d797870', 'e37a0da20d8d24ac', 'ed5d3b0f00026714', 'ee7b45399bcf968a', 'f0982ea7c2028c50', 'f1737c198253be9d', 'f766dbb74e38db04', 'fbe41c3bbd33ac19'])
99
+ _SW_GRP_H = frozenset(['09455d377a262608', '0e30392b37df5495', '103064b4fea08f1c', '11a120607b793e4c', '158ba6de7392b368', '193a01c7e1bd0f62', '2199b8b3ff553486', '240843438078b817', '2508ee34b17c2225', '2a4eb6e114be0fab', '352836dac99f04b7', '391bd8a58914224a', '4185301675b9d80c', '4472c2744066912a', '4c4684b615a127b2', '522a470219d3a4f1', '5a6bb90382b6c1f6', '5ca7613c608a9ea5', '63f15e4687bfc135', '66a7555e70f0872e', '73d877e885d34562', '77c343b865d05e31', '780409423e866d09', '7e0a43f55a6714fd', '97409999bccb98f4', '996bd865c828c8f9', '9b60cdfff0e2ab1c', 'a330b94802e4b6c6', 'a9c5a9029bc6a596', 'b48e9326bf6316ef', 'c595d125bfd5ebe9', 'c8575232dac0347d', 'd64fa92611ac0eb2', 'e182e241e26cb119', 'e88087f130a6e7d6'])
100
+ _BOOK_COMIC_GRP_H = frozenset(['0cad54552908fa65', '338c3fad71b18311', '5913147687b3c1e5', '712f82b4b85131a0', '7f5fc23af41380b2', '8d8b63a8bb00b7e4', '913bd80742a62e95', 'c00722c39b756d92', 'ecfebbed5476af20', 'ee2807f40b38f779', 'f89c7ccfb7b1338c', 'fdffac028227e18e', 'ffb2c2ead4c7ba7e'])
101
+ _BENIGN_README_EXE_H = frozenset(['062b96e72d602b6e', '9a9f15179701c6ae', 'ddf05591355cf7e0'])
102
+
103
+ _TOKEN_SCAN_RE = re.compile(r"[A-Za-z0-9]+(?:-[A-Za-z0-9]+)*")
104
+ _SIDE_ANIME_KW_RE = re.compile(
105
+ r"(?i)\b(?:anime|ova|ona|donghua|aeni|bdrip[ ._-]*1080p[ ._-]*flac|dual[ ._-]*audio[ ._-]*\[?(?:jpn|jap))\b"
106
+ )
107
+ _SIDE_VID_CLAIM_RE = re.compile(
108
+ r"(?i)\b(?:1080p|720p|2160p|4k|bluray|blu-ray|bdrip|brrip|web-dl|webdl|webrip|hdtv|dvdrip|hdrip|camrip|hdcam|telesync|x264|x265|hevc|avc|h264|h265|s\d{1,2}e\d{1,3}|season[ ._-]*\d+|fc2-?ppv)\b|高清首发|更新至第"
109
+ )
110
+ _SIDE_DECEPTIVE_SUB_STRIP_RE = re.compile(
111
+ r"(?i)\b(?:eng(?:lish)?|japanese|jap|jpn|spanish|esp|french|fre|german|ger|italian|ita|russian|rus)[ ._-]+(?:hard|soft)?sub(?:s|bed|titled|titles)?\b|"
112
+ r"\b(?:hard|soft)?sub(?:s|bed|titled|titles)?[ ._-]+(?:eng|english|esp|ja|jpn|fr|de|it|ru)\b|"
113
+ r"\b(?:m[ ._-]?subs?|multi[ ._-]?subs?|esubs?|vostfr|hardsubs?)\b"
114
+ )
115
+ _SIDE_ANIME_DUB_KW_RE = re.compile(
116
+ r"(?i)\b(?:dual[ ._-]?audio|dubbed|eng(?:lish)?[ ._-]?dub(?:bed)?|multi[ ._-]?audio)\b"
117
+ )
118
+ _SIDE_SW_APP_TITLE_RE = re.compile(
119
+ r"(?i)\b(?:v?\d+\.\d+\.\d+|repack|portable|cracked|keygen|patch|patcher|activat(?:ed|or)|"
120
+ r"multilingual|win(?:32|64)|x64|x86|macos|macosx|android|apk|pc[ ._-]?game|nintendo[ ._-]?switch|ps4|ps5|"
121
+ r"emulator|driver|firmware|setup|installer|downloader|studio|vst3?|"
122
+ r"pro[ ._-]+v?\d|ce[ ._-]+v?\d)\b"
123
+ )
124
+ _SIDE_COURSE_RE = re.compile(
125
+ r"(?i)\b(?:complete[ ._-]+course|video[ ._-]+course|masterclass|bootcamp|tutorial[ ._-]+series|certification[ ._-]+course|hands-on[ ._-]+training)\b"
126
+ )
127
+ _SIDE_BOOK_COMIC_KW_RE = re.compile(
128
+ r"(?i)\b(?:audiobooks?|audible|m4b|unabridged|narrated[ ._-]?by|ebook|e-book|epub|mobi|azw3|calibre)\b"
129
+ )
130
+ _SIDE_VPN_TRAP_RE = re.compile(
131
+ r"(?i)免费安装下载vpn|打不开网页|codec[ ._-]?(?:pack|setup|install)|player[ ._-]?setup|instructions?[ ._-]?to[ ._-]?watch"
132
+ )
133
+ _SIDE_CJK_RE = re.compile(r"[\u4e00-\u9fff]")
134
+ _SIDE_EP_ZIP_RE = re.compile(
135
+ r"(?i)(?:^\[[^\[\]]{2,30}\].*?(?:\s-\s*\d{1,3}\b|\bS\d{1,2}E\d{1,3}\b).*?\b(?:1080p|720p|2160p|480p)\b|"
136
+ r"\bS\d{1,2}E\d{1,3}\b.*?\b(?:1080p|720p|2160p|480p|WEB-DL|WEBRip|BluRay|BDRip|HDTV|x264|x265|HEVC)\b|"
137
+ r"\b(?:1080p|720p|2160p|480p|576p|4k)\b.*?\b(?:WEB-DL|WEBRip|BluRay|BDRip|BRRip|HDRip|HDTV|HDCAM|CAMRip|DVDRip|x264|x265|H\.?264|H\.?265|HEVC|AV1|XviD|DivX)\b|"
138
+ r"\b(?:WEB-DL|WEBRip|BDRip|BRRip|HDRip|HDTV|HDCAM|CAMRip|DVDRip)\b(?:.*?\b(?:x264|x265|H\.?264|H\.?265|HEVC|AV1|XviD|DivX))?)"
139
+ )
140
+ _SIDE_BENIGN_ZIP_RE = re.compile(
141
+ r"(?i)\b(?:batch|complete|season[ ._-]*\d+|s\d{1,2}[ ._-]*complete|e?\d{1,3}[ ._-]*[~-][ ._-]*e?\d{1,3}|시즌\s*\d+|완결|全集|合集|"
142
+ r"ost|soundtrack|artbook|scans?|wallpapers?|fonts?|subs?|subtitles?)\b"
143
+ )
144
+
145
+
146
+ # --- R12 Session 2: derivative-group file-tree features ---------------------
147
+ # Brief-mandated video set (.mkv .mp4 .avi .ogv .mpg .mpeg .m4v .ts .wmv .mov
148
+ # .webm .vob .iso) plus same-family containers seen in the gold corpus
149
+ # (.m2ts/.mts/.divx/.flv/.rmvb/.3gp/.asf) so primary videos are never missed.
150
+ _DERIV_VIDEO_EXTS = frozenset(
151
+ "mkv mp4 avi ogv mpg mpeg m4v ts wmv mov webm vob iso m2ts mts divx flv rmvb 3gp asf".split()
152
+ )
153
+ _DERIV_SAMPLE_RE = re.compile(r"(?i)\bsample\b")
154
+ # IA/transcode rendition suffixes stripped repeatedly from a video stem.
155
+ _DERIV_SUFFIX_RE = re.compile(
156
+ r"(?i)(?:\.(?:ia|512kb|h264|h\.264|x264|hd|sd)|_(?:archive|thumbs|files|512kb|h264|ia))+$"
157
+ )
158
+ _EP_SXE_RE = re.compile(r"(?i)(?<![a-z0-9])s(\d{1,2})[\s._\-]*e(\d{1,3})(?![0-9])")
159
+ _EP_NXN_RE = re.compile(r"(?<![0-9a-zA-Z])(\d{1,2})x(\d{2})(?![0-9a-zA-Z])", re.IGNORECASE)
160
+ _EP_WORD_RE = re.compile(r"(?i)\b(?:episode)[\s._\-]*(\d{1,3})(?![0-9])")
161
+ _EP_SHORT_RE = re.compile(r"(?i)(?<![a-z0-9])ep\.?\s*(\d{1,3})(?![0-9])")
162
+ _EP_BARE_E_RE = re.compile(r"(?i)(?<![a-z0-9])e(\d{1,3})(?![a-z0-9])")
163
+ # Anime fansub dash numbering (`[Group] Show - 01v2 [CRC].mkv`), train-validated in WS-1.
164
+ _EP_DASH_NUM_RE = re.compile(r"\s-\s(\d{1,3})(?:v\d)?(?![0-9a-zA-Z])")
165
+ _SEASON_HINT_RE = re.compile(
166
+ r"(?i)(?:(?<![a-z0-9])s(\d{1,2})(?![0-9])|\bseason[\s._\-]*(\d{1,2})(?![0-9]))"
167
+ )
168
+ _MULTIPART_TOKEN_RE = re.compile(
169
+ r"(?i)\b(?:cd|disc|disk|pt|part|side)[\s._\-]*([1-4])\b"
170
+ )
171
+ _VTS_STEM_RE = re.compile(r"(?i)^vts_\d+_\d+$")
172
+ _PILOT_RE = re.compile(r"(?i)\b(?:unaired[\s._\-]*)?pilot\b")
173
+ _YEAR_TOKEN_RE = re.compile(r"(?<![0-9])(?:19[2-9]\d|20[0-2]\d)(?![0-9])")
174
+ # Negated subtitle clauses are removed before the side[13] subtitle cue so
175
+ # "no english subs" / "without eng subs" / "non-english subtitles" cannot fire it.
176
+ _NEGATED_SUB_RE = re.compile(
177
+ r"(?i)\b(?:no|without|non|none|pas\s+de|kein[e]?|sin)[\s._\-]+"
178
+ r"(?:[-\w]+[\s._\-]+){0,2}?(?:eng(?:lish)?[\s._\-]+)?"
179
+ r"(?:sub(?:title)?s?|srt|ass|ssa|cc)\b"
180
+ r"|\bsub(?:title)?s?[\s:_-]*(?:no|none|n/a|nil|false)\b"
181
+ )
182
+ _SUB_CUE_RE = re.compile(
183
+ r"(?i)(?:\b(?:eng(?:lish)?|en|hard|soft|multi)[\s._\-]*"
184
+ r"(?:sub(?:title)?s?|srt|ass|ssa|cc)\b"
185
+ r"|\b(?:subtitles?|subs|subbed|esubs?|closed[\s._\-]*captions?)\b"
186
+ r"|\.(?:srt|ass|ssa)\b)"
187
+ )
188
+ _MEDIA_SUMMARY_EXTS = frozenset(
189
+ set(_DERIV_VIDEO_EXTS)
190
+ | set("mp3 flac m4b m4a ogg opus epub mobi azw azw3 pdf cbz cbr cb7 "
191
+ "exe msi apk dmg nsp xci xcz nsz".split())
192
+ )
193
+ _DERIV_LOG_NORM: float = math.log1p(64.0)
194
+
195
+
196
+ def _normalize_file_entries(files: Any) -> List[Tuple[str, int]]:
197
+ """Normalize torrent `files` (dicts / (path, size) tuples / strings) to `(path, size)` pairs."""
198
+ out: List[Tuple[str, int]] = []
199
+ for f in files or []:
200
+ if isinstance(f, Mapping):
201
+ p = str(f.get("path") or f.get("p") or f.get("name") or "")
202
+ try:
203
+ sz = int(f.get("size") or f.get("s") or 0)
204
+ except (TypeError, ValueError, OverflowError):
205
+ sz = 0
206
+ elif isinstance(f, (tuple, list)) and len(f) >= 1:
207
+ p = str(f[0] or "")
208
+ try:
209
+ sz = int(f[1]) if len(f) > 1 else 0
210
+ except (TypeError, ValueError, OverflowError):
211
+ sz = 0
212
+ else:
213
+ p, sz = str(f or ""), 0
214
+ if p and "/.____padding_file/" not in p and not p.startswith(".____padding_file"):
215
+ out.append((p, max(sz, 0)))
216
+ return out
217
+
218
+
219
+ def _deriv_stem_key(parent: str, stem: str) -> Tuple[str, str]:
220
+ """Collapse transcode renditions of one work to a single derivative-group key."""
221
+ s = stem
222
+ for _ in range(4):
223
+ stripped = _DERIV_SUFFIX_RE.sub("", s)
224
+ if stripped == s:
225
+ break
226
+ s = stripped
227
+ norm_s = re.sub(r"[\W_]+", " ", s.lower(), flags=re.UNICODE).strip() or s.lower().strip()
228
+ norm_p = re.sub(r"[\W_]+", " ", parent.lower(), flags=re.UNICODE).strip()
229
+ return norm_p, norm_s
230
+
231
+
232
+ def _episode_keys_from_text(text: str, default_season: Optional[int] = None) -> set:
233
+ """Distinct episode keys in one path/name string (multipart CD/Part tokens never count)."""
234
+ keys: set = set()
235
+ if not text:
236
+ return keys
237
+ seasons = {
238
+ int(g) for m in _SEASON_HINT_RE.finditer(text)
239
+ for g in m.groups() if g is not None
240
+ }
241
+ for m_sxe in _EP_SXE_RE.finditer(text):
242
+ seasons.add(int(m_sxe.group(1)))
243
+ for m_nxn in _EP_NXN_RE.finditer(text):
244
+ seasons.add(int(m_nxn.group(1)))
245
+ season = seasons.pop() if len(seasons) == 1 else (default_season if not seasons else None)
246
+
247
+ def add_bare(num: int) -> None:
248
+ keys.add(f"s{season:02d}e{num:03d}" if season is not None else f"ep{num:04d}")
249
+
250
+ for m in _EP_SXE_RE.finditer(text):
251
+ keys.add(f"s{int(m.group(1)):02d}e{int(m.group(2)):03d}")
252
+ for m in _EP_NXN_RE.finditer(text):
253
+ keys.add(f"s{int(m.group(1)):02d}e{int(m.group(2)):03d}")
254
+ for m in _EP_WORD_RE.finditer(text):
255
+ add_bare(int(m.group(1)))
256
+ for m in _EP_SHORT_RE.finditer(text):
257
+ add_bare(int(m.group(1)))
258
+ # Bare `E01` stems (brief's `Ep?` with optional `p`), e.g. `Show.S01/E01.mkv`.
259
+ # The lookbehind already excludes the `E` inside `S01E01` (digit before it).
260
+ for m in _EP_BARE_E_RE.finditer(text):
261
+ add_bare(int(m.group(1)))
262
+ for m in _EP_DASH_NUM_RE.finditer(text):
263
+ add_bare(int(m.group(1)))
264
+ if not keys and _PILOT_RE.search(text):
265
+ keys.add("pilot")
266
+ return keys
267
+
268
+
269
+ def compute_derivative_group_features(
270
+ files: Any,
271
+ raw_name: str = "",
272
+ ) -> Tuple[float, float, float, float]:
273
+ """Compute `(U_deriv_groups, E_episode_keys, P_multipart_idx, s_max)` for a torrent file tree.
274
+
275
+ - `U`: unique normalized video derivative stems; IA transcodes of one stem
276
+ (`.mp4`/`.ogv`/`.ia.mp4`/`.512kb` ...) collapse to one group and all
277
+ `VIDEO_TS/VTS_*` files collapse to one group.
278
+ - `E`: distinct episode keys (`SxxExx`, `NxNN`, `Ep N`, `Episode N`, bare
279
+ `ENN` under a season folder). Pure multipart splits (`CD1/CD2`,
280
+ `Part 1/2` of one film) contribute `0`; a two-part TV pilot carrying an
281
+ explicit `SxxExx`/`Pilot`/season marker counts its parts (`E >= 2`).
282
+ - `P`: `1.0` for a multi-disc/part movie split or `VIDEO_TS` with `E <= 1`.
283
+ - `s_max`: largest deduplicated primary stem share, using the largest file
284
+ per derivative stem so IA renditions cannot depress it. Sample files
285
+ (`sample` token, or `< 15 MB` when total video exceeds `100 MB`) are ignored.
286
+ """
287
+ entries = _normalize_file_entries(files)
288
+ videos: List[Tuple[str, str, str, int]] = [] # (path, parent, stem, size)
289
+ for path, sz in entries:
290
+ p_norm = path.replace("\\", "/")
291
+ parent, _, base = p_norm.rpartition("/")
292
+ if "." not in base:
293
+ continue
294
+ stem, ext = base.rsplit(".", 1)
295
+ if ext.lower() in _DERIV_VIDEO_EXTS:
296
+ videos.append((p_norm, parent, stem, sz))
297
+
298
+ total_video = sum(sz for _, _, _, sz in videos)
299
+ kept = [
300
+ v for v in videos
301
+ if not _DERIV_SAMPLE_RE.search(v[0].rsplit("/", 1)[-1])
302
+ and not (0 < v[3] < 15_000_000 and total_video > 100_000_000)
303
+ ]
304
+ if kept:
305
+ videos = kept
306
+
307
+ rep_bytes: Dict[Tuple[str, str], int] = {}
308
+ ep_keys: set = set()
309
+ part_indices: set = set()
310
+ has_video_ts = False
311
+ tv_marker = bool(_EP_SXE_RE.search(raw_name or "") or _PILOT_RE.search(raw_name or "")
312
+ or _SEASON_HINT_RE.search(raw_name or ""))
313
+ title_seasons = {
314
+ int(g) for m in _SEASON_HINT_RE.finditer(raw_name or "")
315
+ for g in m.groups() if g is not None
316
+ } | {int(m.group(1)) for m in _EP_SXE_RE.finditer(raw_name or "")} | {int(m.group(1)) for m in _EP_NXN_RE.finditer(raw_name or "")}
317
+ title_default_season = next(iter(title_seasons)) if len(title_seasons) == 1 else None
318
+ ep_keys.update(_episode_keys_from_text(raw_name or "", default_season=title_default_season))
319
+
320
+ for p_norm, parent, stem, sz in videos:
321
+ ep_keys.update(_episode_keys_from_text(p_norm, default_season=title_default_season))
322
+ part_indices.update(int(m.group(1)) for m in _MULTIPART_TOKEN_RE.finditer(p_norm))
323
+ if _EP_SXE_RE.search(p_norm) or _PILOT_RE.search(p_norm) or _SEASON_HINT_RE.search(p_norm):
324
+ tv_marker = True
325
+ if "/video_ts/" in f"/{p_norm.lower()}" or _VTS_STEM_RE.match(stem):
326
+ has_video_ts = True
327
+ key = ("__video_ts__", "")
328
+ else:
329
+ key = _deriv_stem_key(parent, stem)
330
+ rep_bytes[key] = max(rep_bytes.get(key, 0), sz)
331
+
332
+ e_count = len(ep_keys)
333
+ # Two-part TV pilot: explicit TV marker + distinct parts of one episode.
334
+ if tv_marker and len(part_indices) >= 2 and e_count <= 1:
335
+ e_count = len(part_indices)
336
+
337
+ u_count = len(rep_bytes)
338
+ p_idx = 1.0 if (has_video_ts or len(part_indices) >= 2) and e_count <= 1 else 0.0
339
+ rep_total = sum(rep_bytes.values())
340
+ if u_count == 0:
341
+ s_max = 0.0
342
+ elif rep_total > 0:
343
+ s_max = max(rep_bytes.values()) / rep_total
344
+ else: # sizes unknown: equal-share assumption over derivative groups
345
+ s_max = 1.0 / u_count
346
+ return float(u_count), float(e_count), float(p_idx), float(s_max)
347
+
348
+
349
+ def primary_media_file_summary(files: Any, top_n: int = 3) -> str:
350
+ """Top-`top_n` largest primary media relative paths, ` ; `-joined (largest first)."""
351
+ media = [
352
+ (p, sz) for p, sz in _normalize_file_entries(files)
353
+ if "." in p.rsplit("/", 1)[-1]
354
+ and p.rsplit("/", 1)[-1].rsplit(".", 1)[-1].lower() in _MEDIA_SUMMARY_EXTS
355
+ ]
356
+ media.sort(key=lambda x: -x[1])
357
+ return " ; ".join(p for p, _ in media[:top_n])
358
+
359
+
360
+ def build_enriched_name_text(name: str, files: Any, max_bytes: int = 256) -> str:
361
+ """Append |FILES: <top-3 media basenames> when name is opaque or hides multi-episode files."""
362
+ summary = primary_media_file_summary(files, top_n=3)
363
+ if not summary:
364
+ return name
365
+ _, e_count, _, _ = compute_derivative_group_features(files, name)
366
+ name_has_ep = bool(_episode_keys_from_text(name))
367
+ is_opaque_slug = not bool(re.search(r"[A-Za-z]{3,}", name))
368
+ hidden_ep_pack = (e_count >= 2 and not name_has_ep)
369
+ if not (is_opaque_slug or hidden_ep_pack):
370
+ return name
371
+ basenames = " ; ".join(
372
+ p.rsplit("/", 1)[-1] for p in summary.split(" ; ")
373
+ )
374
+ return f"{name} |FILES: {basenames}"
375
+
376
+
377
+ def _h64(s: str) -> str:
378
+ return hashlib.blake2b(s.strip().lower().encode("utf-8"), digest_size=8).hexdigest()
379
+
380
+
381
+ def _extract_token_hashes(text: str) -> frozenset[str]:
382
+ toks = [m.group(0).lower() for m in _TOKEN_SCAN_RE.finditer(text)]
383
+ out = {_h64(t) for t in toks}
384
+ for i in range(len(toks) - 1):
385
+ out.add(_h64(f"{toks[i]} {toks[i + 1]}"))
386
+ return frozenset(out)
387
+
388
+
389
+ _VOCAB_CACHE_BY_PATH: Dict[str, Dict[str, int]] = {}
390
+
391
+
392
+ def _get_vocab_ext_map(vocab_path: Optional[Path] = None) -> Dict[str, int]:
393
+ global _VOCAB_EXT_MAP
394
+ if vocab_path is not None:
395
+ vp = Path(vocab_path)
396
+ if vp.exists():
397
+ key = str(vp.resolve())
398
+ if key not in _VOCAB_CACHE_BY_PATH:
399
+ data = json.loads(vp.read_text(encoding="utf-8"))
400
+ _VOCAB_CACHE_BY_PATH[key] = {e: i + 1 for i, e in enumerate(data.get("exts", []))}
401
+ _VOCAB_EXT_MAP = _VOCAB_CACHE_BY_PATH[key]
402
+ return _VOCAB_EXT_MAP
403
+ if vocab_path is None and _VOCAB_EXT_MAP is not None and len(_VOCAB_EXT_MAP) > 64:
404
+ return _VOCAB_EXT_MAP
405
+ vpath = Path(__file__).resolve().with_name("vocab.json")
406
+ if not vpath.exists() and os.environ.get("ARMIGER_MODEL_DIR"):
407
+ vpath = Path(os.environ["ARMIGER_MODEL_DIR"]) / "vocab.json"
408
+ if vpath.exists():
409
+ key = str(vpath.resolve())
410
+ if key not in _VOCAB_CACHE_BY_PATH:
411
+ data = json.loads(vpath.read_text(encoding="utf-8"))
412
+ _VOCAB_CACHE_BY_PATH[key] = {e: i + 1 for i, e in enumerate(data.get("exts", []))}
413
+ _VOCAB_EXT_MAP = _VOCAB_CACHE_BY_PATH[key]
414
+ else:
415
+ _VOCAB_EXT_MAP = {
416
+ "mp3": 2, "mp4": 3, "mkv": 4, "avi": 7, "srt": 10, "zip": 12,
417
+ "rar": 17, "exe": 23, "flac": 24, "png": 25, "dll": 31, "m4b": 53,
418
+ "epub": 81, "cbz": 96, "lnk": 150, "scr": 310,
419
+ }
420
+ return _VOCAB_EXT_MAP
421
+
422
+
423
+ def _hash_ext(ext: str) -> int:
424
+ if not ext:
425
+ return 4096
426
+ return _get_vocab_ext_map().get(ext.lower(), 4096)
427
+
428
+
429
+ def _hash_word(w: str) -> int:
430
+ if not w:
431
+ return 0
432
+ folded = _PREP_DIGIT_RE.sub("0", w.lower())
433
+ return (zlib.crc32(folded.encode("utf-8")) % 32767) + 1
434
+
435
+
436
+ def _log2_size_bucket(sz: int) -> int:
437
+ return min(47, max(int(sz), 1).bit_length() - 1)
438
+
439
+
440
+ def _prep_split_path(path: str) -> Tuple[str, str, str, int]:
441
+ p_norm = path.replace("\\", "/")
442
+ parent, _, base = p_norm.rpartition("/")
443
+ i = base.rfind(".")
444
+ ext = base[i + 1:].lower() if i > 0 else ""
445
+ if len(ext) <= 10 and ext.isascii() and ext.isalnum():
446
+ stem = base[:i]
447
+ else:
448
+ ext, stem = "", base
449
+ return parent.rpartition("/")[2], stem, ext, p_norm.count("/")
450
+
451
+
452
+ def _prep_path_words(parent: str, stem: str) -> List[int]:
453
+ p_words = [_hash_word(w) for w in _PREP_WORD_RE.findall(parent.lower())][-2:] if parent else []
454
+ s_words = [_hash_word(w) for w in _PREP_WORD_RE.findall(stem.lower())][-(4 - len(p_words)):]
455
+ res = p_words + s_words
456
+ while len(res) < 4:
457
+ res.append(0)
458
+ return res[:4]
459
+
460
+
461
+ def encode_torrent_features(
462
+ row: Mapping[str, Any],
463
+ max_name_len: int = 256,
464
+ max_files: int = 64,
465
+ ) -> Dict[str, Any]:
466
+ """Encode a torrent metadata dictionary into the exact ONNX tensor inputs for Armiger v2.1."""
467
+ name = str(row.get("name") or row.get("title") or "")
468
+
469
+ raw_files = row.get("files") or []
470
+ try:
471
+ total_size = max(0, int(row.get("size") or 0))
472
+ except (TypeError, ValueError, OverflowError):
473
+ total_size = 0
474
+ all_items: List[Tuple[str, int]] = _normalize_file_entries(raw_files)
475
+ all_items.sort(key=lambda x: -x[1])
476
+ items: List[Tuple[str, int]] = all_items[:max_files]
477
+ if not items and (row.get("ext") or row.get("extension") or "." in name):
478
+ items.append((name, total_size))
479
+
480
+ # R12: opaque/folder names expose their primary media filenames to the
481
+ # byte/char/word encoders; derivative-group features use the full tree.
482
+ name_text = build_enriched_name_text(name, all_items or items, max_bytes=max_name_len)
483
+ b_name = name_text.encode("utf-8", "replace")
484
+ if len(b_name) > max_name_len:
485
+ b_name = b_name[:max_name_len].decode("utf-8", "ignore").encode("utf-8")
486
+ raw_bytes = list(b_name) if b_name else [32]
487
+ deriv_features = compute_derivative_group_features(all_items or items, name)
488
+
489
+ ext_map = _get_vocab_ext_map()
490
+ exts: List[int] = []
491
+ sizes: List[int] = []
492
+ depths: List[int] = []
493
+ words_list: List[List[int]] = []
494
+ f_exts_str: List[str] = []
495
+
496
+ kbytes = [0.0] * 8
497
+ kcount = [0] * 8
498
+ has_sample = False
499
+
500
+ for path, sz in items:
501
+ parent, stem, ext, depth = _prep_split_path(path)
502
+ f_exts_str.append(ext)
503
+ exts.append(ext_map.get(ext, 4096))
504
+ sizes.append(_log2_size_bucket(sz))
505
+ depths.append(min(depth, 7))
506
+ words_list.append(_prep_path_words(parent, stem))
507
+ k = _PREP_EXT_TO_KIND.get(ext)
508
+ if k is None:
509
+ k = 2 if _PREP_SPLIT_ARC_RE.fullmatch(ext) else 7
510
+ kbytes[k] += max(sz, 0)
511
+ kcount[k] += 1
512
+ has_sample = has_sample or (_PREP_SAMPLE_RE.search(path) is not None)
513
+
514
+ tot_k = sum(kbytes)
515
+ weights = kbytes if tot_k > 0 else [float(c) for c in kcount]
516
+ wsum = sum(weights)
517
+ try:
518
+ raw_fc = max(0, int(row.get("file_count") or 0))
519
+ except (TypeError, ValueError, OverflowError):
520
+ raw_fc = 0
521
+ count = max(raw_fc, len(all_items), len(items))
522
+ biggest = max((max(s, 0) for _, s in items), default=0)
523
+ max_share = (biggest / tot_k if tot_k > 0 else (1.0 / len(items) if items else 0.0))
524
+
525
+ base_stats = [
526
+ math.log1p(max(count, 0)),
527
+ math.log1p(max(total_size if total_size > 0 else int(tot_k), 0)),
528
+ ]
529
+ base_stats += [w / wsum if wsum > 0 else 0.0 for w in weights]
530
+ base_stats += [max_share, float(has_sample)]
531
+
532
+ side = [0.0] * SIDE_DIM
533
+ n_f = float(max(1, len(items)))
534
+
535
+ m_grp = re.match(r"^\[([^\[\]]{2,30})\]", name)
536
+ grp_lc = m_grp.group(1).strip().lower() if m_grp else ""
537
+ grp_h = _h64(grp_lc) if grp_lc else ""
538
+ title_tok_h = _extract_token_hashes(name)
539
+
540
+ has_sub_grp = (grp_h in _SUB_ONLY_GRP_H) or bool(title_tok_h & _SUB_ONLY_GRP_H)
541
+ has_dub_grp = (grp_h in _DUAL_DUB_GRP_H) or bool(title_tok_h & _DUAL_DUB_GRP_H)
542
+ has_game_grp = bool(title_tok_h & _GAME_GRP_H)
543
+ has_sw_grp = bool(title_tok_h & _SW_GRP_H)
544
+ has_book_grp = bool(title_tok_h & _BOOK_COMIC_GRP_H)
545
+
546
+ if items:
547
+ ebk_set = {"epub", "mobi", "azw3", "azw", "djvu", "fb2", "lit", "iba"}
548
+ pdf_set = {"pdf"}
549
+ m4b_set = {"m4b", "aa", "aax"}
550
+ com_set = {"cbz", "cbr", "cb7", "cbt"}
551
+ con_set = {"nsp", "xci", "xcz", "nsz", "wbfs", "rvz", "gcz", "cia", "3ds", "vpk", "ps4", "fpkg"}
552
+ app_set = {"dmg", "pkg", "apk", "ipa", "deb", "rpm", "appimage"}
553
+ exe_dll_set = {"exe", "msi", "bat", "cmd", "sh", "com", "dll"}
554
+ iso_set = {"iso", "bin", "cue", "mdf", "mds", "nrg", "img"}
555
+ arc_set = {"rar", "zip", "7z", "tar", "gz", "xz", "bz2", "001"}
556
+ htm_set = {"html", "htm", "url", "mht", "torrent"}
557
+ scr_set = {"scr", "lnk", "vbs", "hta", "pif", "ps1", "wsf", "reg"}
558
+ vid_set = {"mkv", "mp4", "avi", "ts", "m2ts", "wmv", "mov", "webm", "flv", "mpg", "mpeg", "m4v", "vob", "ogv", "rmvb", "3gp", "asf", "divx"}
559
+ bd_game_benign = {"vcdiff", "apk", "bdmv", "clpi", "mpls", "ifo", "bup", "vob", "npa", "vpk", "pck", "umd", "pak", "xp3", "rgss3a", "wolf", "nsp", "xci", "wbfs"}
560
+
561
+ side[0] = sum(1 for e in f_exts_str if e in ebk_set) / n_f
562
+ side[1] = sum(1 for e in f_exts_str if e in pdf_set) / n_f
563
+ side[2] = sum(1 for e in f_exts_str if e in m4b_set) / n_f
564
+ side[3] = sum(1 for e in f_exts_str if e in com_set) / n_f
565
+ side[4] = sum(1 for e in f_exts_str if e in con_set) / n_f
566
+ side[5] = sum(1 for e in f_exts_str if e in app_set) / n_f
567
+
568
+ vc = bool(_SIDE_VID_CLAIM_RE.search(name) or _SIDE_ANIME_KW_RE.search(name) or has_sub_grp or has_dub_grp)
569
+ has_any_vid_inf = any(e in vid_set for e in f_exts_str)
570
+ has_renpy_structure = (
571
+ not vc
572
+ and not has_any_vid_inf
573
+ and any(
574
+ p_i.replace("\\", "/").lower().endswith((".rpa", ".rpyb")) and int(sz_i or 0) >= 65536
575
+ for p_i, sz_i in items
576
+ )
577
+ )
578
+ renpy_say_re = re.compile(r"(?i)(?:^|/)lib/(?:py\d+-)?windows-[^/]+/say\.vbs$")
579
+
580
+ n_real_exec_dll = 0
581
+ has_real_exe = False
582
+ has_dll = False
583
+ has_scr_trap = False
584
+ has_oversized_lnk = False
585
+ has_root_launcher = False
586
+ has_unconditional_trap_ext = False
587
+ has_any_scr_ext = False
588
+ vid_depths: List[int] = []
589
+
590
+ for (p_i, _), e_str, s_b, d_i, w_i in zip(items, f_exts_str, sizes, depths, words_list):
591
+ p_posix = p_i.replace("\\", "/")
592
+ if has_renpy_structure and renpy_say_re.search(p_posix):
593
+ continue
594
+ if e_str in scr_set:
595
+ has_any_scr_ext = True
596
+ if e_str in {"scr", "vbs", "vbe", "hta", "pif", "wsf", "wsh", "cpl"}:
597
+ has_unconditional_trap_ext = True
598
+ p_tok_h = _extract_token_hashes(p_i)
599
+ is_benign_stub = (e_str == "exe") and (s_b < 15 or 16570 in w_i or bool(p_tok_h & _BENIGN_README_EXE_H))
600
+ if e_str in exe_dll_set and not is_benign_stub:
601
+ n_real_exec_dll += 1
602
+ if e_str == "exe" and not is_benign_stub and s_b >= 15:
603
+ has_real_exe = True
604
+ if d_i == 0:
605
+ has_root_launcher = True
606
+ elif e_str == "dll":
607
+ has_dll = True
608
+ if e_str in {"scr", "lnk", "hta", "vbs", "pif", "ps1", "wsf", "cmd", "com", "one", "xll", "xlam"} and s_b >= 12:
609
+ has_scr_trap = True
610
+ if d_i == 0:
611
+ has_root_launcher = True
612
+ if e_str == "lnk" and s_b >= 15:
613
+ has_oversized_lnk = True
614
+ if e_str in vid_set:
615
+ vid_depths.append(d_i)
616
+
617
+ side[6] = n_real_exec_dll / n_f
618
+ side[7] = sum(1 for e in f_exts_str if e in iso_set) / n_f
619
+ side[8] = sum(1 for e in f_exts_str if e in arc_set) / n_f
620
+ side[9] = sum(1 for e in f_exts_str if e in htm_set) / n_f
621
+ side[10] = 1.0 if has_any_scr_ext else 0.0
622
+
623
+ is_sw_or_game = bool(
624
+ has_game_grp
625
+ or has_sw_grp
626
+ or (not vc and _SIDE_SW_APP_TITLE_RE.search(name))
627
+ or any(e in bd_game_benign for e in f_exts_str)
628
+ )
629
+ vid_share = base_stats[2]
630
+ flagged = has_unconditional_trap_ext
631
+ if not is_sw_or_game:
632
+ if vc and has_real_exe and has_dll:
633
+ flagged = True
634
+ elif vc and len(vid_depths) >= 1 and (has_real_exe or has_root_launcher or has_scr_trap):
635
+ flagged = True
636
+ elif (vc or len(vid_depths) > 0) and has_oversized_lnk:
637
+ flagged = True
638
+ elif vc and vid_share == 0.0 and (has_real_exe or has_scr_trap):
639
+ flagged = True
640
+ elif len(items) <= 2 and any(e in ("zip", "zipx", "rar", "7z") for e in f_exts_str) and vid_share == 0.0:
641
+ if _SIDE_EP_ZIP_RE.search(name) and not _SIDE_BENIGN_ZIP_RE.search(name):
642
+ flagged = True
643
+ elif len(items) <= 2 and any(e in ("iso", "img", "vhd", "vhdx", "udf") for e in f_exts_str) and vid_share == 0.0:
644
+ name_stem = re.sub(r"\.(?:iso|img|vhd|vhdx|udf|dmg)$", "", name.strip(), flags=re.IGNORECASE)
645
+ if _SIDE_EP_ZIP_RE.search(name_stem) and not re.search(r"(?i)\b(?:dvd|bd25|bd50|remux|untouched|pal|ntsc|iso)\b", name_stem):
646
+ flagged = True
647
+
648
+ if flagged or _SIDE_VPN_TRAP_RE.search(name) or (vc and vid_share == 0.0 and (side[6] > 0 or side[9] > 0 or side[10] > 0)):
649
+ side[11] = 1.0
650
+ if vc and vid_share > 0 and 0 < base_stats[1] < 14.22:
651
+ side[12] = 1.0
652
+
653
+ # R12: strip negated subtitle clauses ("no english subs", ...) before the
654
+ # side[13] subtitle cue; previously side[13] was never populated at all.
655
+ if _SUB_CUE_RE.search(_NEGATED_SUB_RE.sub(" ", name)):
656
+ side[13] = 1.0
657
+
658
+ ag = bool(_SIDE_ANIME_KW_RE.search(name)) or has_sub_grp or has_dub_grp
659
+ if ag:
660
+ side[15] = 1.0
661
+ clean_audio_name = _SIDE_DECEPTIVE_SUB_STRIP_RE.sub(" ", name)
662
+ clean_tok_h = _extract_token_hashes(clean_audio_name)
663
+ ad = bool(_SIDE_ANIME_DUB_KW_RE.search(clean_audio_name)) or (grp_h in _DUAL_DUB_GRP_H) or bool(clean_tok_h & _DUAL_DUB_GRP_H)
664
+ if ad:
665
+ side[16] = 1.0
666
+ if (grp_h in _SUB_ONLY_GRP_H) and not ad:
667
+ side[14] = 1.0
668
+ if has_game_grp:
669
+ side[17] = 1.0
670
+ if has_sw_grp and not _SIDE_CJK_RE.search(name):
671
+ side[18] = 1.0
672
+ if _SIDE_COURSE_RE.search(name) and not ag:
673
+ side[19] = 1.0
674
+ if _SIDE_BOOK_COMIC_KW_RE.search(name) or has_book_grp:
675
+ side[20] = 1.0
676
+
677
+ lang_str = (row.get("language") or "").strip().lower()
678
+ if lang_str in ("ja", "ru", "fr", "es", "de", "it", "pt", "zh", "ko", "hi", "pl", "nl", "tr", "ar") or _SIDE_CJK_RE.search(name) or ((grp_h in _SUB_ONLY_GRP_H) and not ad):
679
+ side[22] = 1.0
680
+ elif lang_str in ("en", "eng", "english") or ad:
681
+ side[21] = 1.0
682
+ else:
683
+ side[23] = 1.0
684
+
685
+ # R12 derivative-group features (raw counts stay in `deriv_features`;
686
+ # the network sees bounded log-normalized U/E, binary P, raw s_max).
687
+ _u, _e, _p, _smax = deriv_features
688
+ side[24] = min(1.0, math.log1p(_u) / _DERIV_LOG_NORM)
689
+ side[25] = min(1.0, math.log1p(_e) / _DERIV_LOG_NORM)
690
+ side[26] = float(_p)
691
+ side[27] = float(_smax)
692
+
693
+ return {
694
+ "name_bytes": raw_bytes,
695
+ "ext": exts if exts else [0],
696
+ "size": sizes if sizes else [0],
697
+ "depth": depths if depths else [0],
698
+ "words": words_list if words_list else [[0, 0, 0, 0]],
699
+ "stats": base_stats + side,
700
+ "deriv_features": deriv_features,
701
+ "name_text": name_text,
702
+ }
703
+
704
+
705
+ def build_onnx_batch_inputs(
706
+ rows: Sequence[Mapping[str, Any]],
707
+ max_name_len: int = 256,
708
+ max_files: int = 64,
709
+ stats_dim: Optional[int] = None,
710
+ ) -> Dict[str, np.ndarray]:
711
+ """Collate a sequence of torrent dicts into padded NumPy arrays for ONNX Runtime.
712
+
713
+ `stats_dim` defaults to `STATS_DIM` (40). Pass a legacy checkpoint's width
714
+ (e.g. `36`) to emit the legacy `stats` prefix during the R12 transition;
715
+ `ArmigerStandaloneONNX.predict_batch` does this automatically from the
716
+ loaded session's `stats` input shape.
717
+ """
718
+ encoded = [encode_torrent_features(r, max_name_len=max_name_len, max_files=max_files) for r in rows]
719
+ bsz = len(encoded)
720
+ max_l = max(len(e["name_bytes"]) for e in encoded)
721
+ max_f = max(len(e["ext"]) for e in encoded)
722
+ width = STATS_DIM if stats_dim is None else int(stats_dim)
723
+
724
+ name_b = np.zeros((bsz, max_l), dtype=np.int64)
725
+ name_l = np.zeros((bsz,), dtype=np.int64)
726
+ ext_t = np.zeros((bsz, max_f), dtype=np.int32)
727
+ size_t = np.zeros((bsz, max_f), dtype=np.int64)
728
+ depth_t = np.zeros((bsz, max_f), dtype=np.int64)
729
+ words_t = np.zeros((bsz, max_f, 4), dtype=np.int32)
730
+ stats_t = np.zeros((bsz, width), dtype=np.float32)
731
+
732
+ for i, e in enumerate(encoded):
733
+ lb = len(e["name_bytes"])
734
+ lf = len(e["ext"])
735
+ name_b[i, :lb] = e["name_bytes"]
736
+ name_l[i] = lb
737
+ ext_t[i, :lf] = e["ext"]
738
+ size_t[i, :lf] = e["size"]
739
+ depth_t[i, :lf] = e["depth"]
740
+ words_t[i, :lf, :] = e["words"]
741
+ stats_t[i, : min(width, len(e["stats"]))] = e["stats"][:width]
742
+
743
+ return {
744
+ "name_bytes": name_b,
745
+ "name_len": name_l,
746
+ "ext": ext_t,
747
+ "size": size_t,
748
+ "depth": depth_t,
749
+ "words": words_t,
750
+ "stats": stats_t,
751
+ }
752
+
753
+
754
+ def _softmax(logits: np.ndarray) -> np.ndarray:
755
+ shifted = logits - np.max(logits, axis=-1, keepdims=True)
756
+ exps = np.exp(shifted)
757
+ return exps / np.sum(exps, axis=-1, keepdims=True)
758
+
759
+
760
+ _DEFAULT_MONDRIAN_THRESHOLDS: Dict[str, float] = {"adult": 0.5451, "junk": 0.1010, "owner_scope": 0.4770}
761
+
762
+
763
+ def _mondrian_value(v: Any) -> Optional[float]:
764
+ return float(v) if isinstance(v, (int, float)) and not isinstance(v, bool) and 0.0 < float(v) < 1.0 else None
765
+
766
+
767
+ def load_mondrian_thresholds(calibration_path: str | Path | None) -> Tuple[Dict[str, float], Dict[str, Dict[str, float]]]:
768
+ """(global gate thresholds, per-gate strata) from `calibration.json`; defaults only where nothing is fitted.
769
+
770
+ Same precedence as `calibration_and_dht.ArmigerCalibrator`: `mondrian_thresholds[gate]` (a dict replaces the
771
+ legacy block entirely), else legacy `mondrian_conformal_thresholds[gate]["threshold" | "threshold_*"]`;
772
+ strata from `mondrian_stratified_thresholds[gate]["strata"][<category>]` (fitted per predicted category).
773
+ """
774
+ thresholds = dict(_DEFAULT_MONDRIAN_THRESHOLDS)
775
+ strata: Dict[str, Dict[str, float]] = {}
776
+ try:
777
+ cfg = json.loads(Path(calibration_path).read_text(encoding="utf-8")) if calibration_path else {}
778
+ except (OSError, ValueError):
779
+ cfg = {}
780
+ if not isinstance(cfg, dict):
781
+ return thresholds, strata
782
+ flat = cfg.get("mondrian_thresholds")
783
+ if not isinstance(flat, dict):
784
+ legacy = cfg.get("mondrian_conformal_thresholds")
785
+ flat = {}
786
+ for gate, block in (legacy.items() if isinstance(legacy, dict) else ()):
787
+ if isinstance(block, dict):
788
+ keys = ["threshold"] + sorted(k for k in block if str(k).startswith("threshold_"))
789
+ flat[gate] = next((block[k] for k in keys if _mondrian_value(block.get(k)) is not None), None)
790
+ else:
791
+ flat[gate] = block
792
+ for gate in thresholds:
793
+ v = _mondrian_value(flat.get(gate, flat.get("scope") if gate == "owner_scope" else None))
794
+ if v is not None:
795
+ thresholds[gate] = v
796
+ raw_strata = cfg.get("mondrian_stratified_thresholds")
797
+ for gate, block in (raw_strata.items() if isinstance(raw_strata, dict) else ()):
798
+ per = block.get("strata") if isinstance(block, dict) else None
799
+ if isinstance(per, dict):
800
+ g_key = "owner_scope" if gate == "scope" else str(gate)
801
+ strata[g_key] = {str(k): t for k, v in per.items() if (t := _mondrian_value(v)) is not None}
802
+ return thresholds, strata
803
+
804
+
805
+ class ArmigerStandaloneONNX:
806
+ """Self-contained ONNX Runtime classifier for `NorviTech/armiger` and `NorviTech/armiger-ultra`."""
807
+
808
+ def __init__(
809
+ self,
810
+ onnx_path: Optional[str | Path] = None,
811
+ vocab_path: Optional[str | Path] = None,
812
+ calibration_path: Optional[str | Path] = None,
813
+ ) -> None:
814
+ import onnxruntime as ort
815
+
816
+ base = Path(__file__).resolve().parent
817
+ if onnx_path is None:
818
+ for cand in ("armiger.onnx", "armiger.int8.onnx", "armiger_ultra.fp32.onnx"):
819
+ if (base / cand).exists():
820
+ onnx_path = base / cand
821
+ break
822
+ if onnx_path is None:
823
+ raise FileNotFoundError("Could not find armiger.onnx or armiger_ultra.fp32.onnx")
824
+ self.onnx_path = Path(onnx_path)
825
+ default_vocab = base / "vocab.json"
826
+ if not default_vocab.exists() and self.onnx_path.with_name("vocab.json").exists():
827
+ default_vocab = self.onnx_path.with_name("vocab.json")
828
+ self.vocab_path = Path(vocab_path) if vocab_path else default_vocab
829
+ _get_vocab_ext_map(self.vocab_path)
830
+ default_cal = base / "calibration.json"
831
+ if not default_cal.exists() and self.onnx_path.with_name("calibration.json").exists():
832
+ default_cal = self.onnx_path.with_name("calibration.json")
833
+ self.calibration_path = Path(calibration_path) if calibration_path else default_cal
834
+ self.mondrian_thresholds, self.mondrian_strata = load_mondrian_thresholds(self.calibration_path)
835
+ self.session = ort.InferenceSession(str(self.onnx_path), providers=["CPUExecutionProvider"])
836
+ self.head_names = [o.name for o in self.session.get_outputs()]
837
+
838
+ @staticmethod
839
+ def _apply_v3_post_neural_guard(row: Mapping[str, Any], item_res: Dict[str, Any]) -> Dict[str, Any]:
840
+ return _apply_v3_post_neural_guard(row, item_res)
841
+
842
+ def mondrian_threshold(self, gate: str, stratum: Optional[str] = None) -> float:
843
+ """Stratified Mondrian threshold for `gate` (per predicted category when fitted, else global)."""
844
+ per = self.mondrian_strata.get(gate, {})
845
+ if stratum is not None and str(stratum) in per:
846
+ return per[str(stratum)]
847
+ return self.mondrian_thresholds[gate]
848
+
849
+ def _session_stats_dim(self) -> int:
850
+ """`stats` width expected by the loaded checkpoint (legacy 36 or R12 40)."""
851
+ try:
852
+ for inp in self.session.get_inputs():
853
+ if inp.name == "stats" and len(inp.shape) == 2 and isinstance(inp.shape[1], int):
854
+ return int(inp.shape[1])
855
+ except Exception:
856
+ pass
857
+ return STATS_DIM
858
+
859
+ def predict_batch(self, rows: Sequence[Mapping[str, Any]]) -> List[Dict[str, Any]]:
860
+ feed = build_onnx_batch_inputs(rows, stats_dim=self._session_stats_dim())
861
+ raw_outs = self.session.run(self.head_names, feed)
862
+ logits_by_head = dict(zip(self.head_names, raw_outs))
863
+ results: List[Dict[str, Any]] = []
864
+ for i in range(len(rows)):
865
+ item_res: Dict[str, Any] = {"heads": {}, "probabilities": {}}
866
+ for h, arr in logits_by_head.items():
867
+ vec = arr[i].astype(np.float64)
868
+ if vec.ndim == 2:
869
+ # 3D batch tensor (e.g. span_bio [B, 256, 7]) -> 2D per-item [256, 7]
870
+ shifted = vec - np.max(vec, axis=-1, keepdims=True)
871
+ exps = np.exp(shifted)
872
+ probs_2d = exps / np.maximum(1e-12, np.sum(exps, axis=-1, keepdims=True))
873
+ raw_nm = str(rows[i].get("name") or rows[i].get("title") or "")
874
+ try:
875
+ try:
876
+ from byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans
877
+ except ImportError:
878
+ from byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans
879
+ raw_b = raw_nm.encode("utf-8", errors="replace")[: vec.shape[0]]
880
+ tag_ids = [int(x) for x in decode_bio_logits_viterbi(vec, raw_b).tolist()]
881
+ except Exception:
882
+ tag_ids = [int(x) for x in np.argmax(probs_2d, axis=-1).tolist()]
883
+ classes = NEURAL_HEAD_CLASSES.get(
884
+ h,
885
+ list(SPAN_BIO_CLASSES),
886
+ )
887
+ tag_labels = [classes[idx_c] if idx_c < len(classes) else str(idx_c) for idx_c in tag_ids]
888
+ item_res["heads"][h] = tag_labels
889
+ mean_probs = np.mean(probs_2d, axis=0)
890
+ item_res["probabilities"][h] = {
891
+ classes[k]: round(float(mean_probs[k]), 6) for k in range(min(len(classes), len(mean_probs)))
892
+ }
893
+ try:
894
+ item_res["span_bio"] = decode_bio_tags_to_spans(raw_nm, tag_ids)
895
+ except Exception:
896
+ item_res["span_bio"] = {"tag_ids": tag_ids, "tags": tag_labels}
897
+ continue
898
+ probs = _softmax(vec)
899
+ classes = NEURAL_HEAD_CLASSES.get(h, [str(k) for k in range(len(probs))])
900
+ idx_c = int(np.argmax(probs))
901
+ item_res["heads"][h] = classes[idx_c] if idx_c < len(classes) else str(idx_c)
902
+ item_res["probabilities"][h] = {
903
+ classes[k]: round(float(probs[k]), 6) for k in range(min(len(classes), len(probs)))
904
+ }
905
+ self._apply_v3_post_neural_guard(rows[i], item_res)
906
+ p_junk = item_res["probabilities"].get("junk", {}).get("junk", 0.0)
907
+ p_adult = item_res["probabilities"].get("adult", {}).get("adult", 0.0)
908
+ # Calibrated excluded probability is kept as-is: the conformal gate (not the argmax label) decides.
909
+ p_excl = item_res["probabilities"].get("owner_scope", {}).get("excluded", 0.0)
910
+ stratum = item_res["heads"].get("category")
911
+ thr = {g: self.mondrian_threshold(g, stratum) for g in ("adult", "junk", "owner_scope")}
912
+ item_res["p_junk"] = p_junk
913
+ item_res["mondrian_gates"] = {
914
+ "adult_flagged": bool(p_adult >= thr["adult"]),
915
+ "junk_flagged": bool(p_junk >= thr["junk"]),
916
+ "scope_excluded": bool(p_excl >= thr["owner_scope"]),
917
+ "passed_all": bool(p_adult < thr["adult"] and p_junk < thr["junk"] and p_excl < thr["owner_scope"]),
918
+ "thresholds": thr,
919
+ "owner_scope_stratum": stratum,
920
+ }
921
+ results.append(item_res)
922
+ return results
923
+
924
+ def predict(self, row: Mapping[str, Any]) -> Dict[str, Any]:
925
+ return self.predict_batch([row])[0]
926
+
927
+
928
+ ArmigerONNXClassifier = ArmigerStandaloneONNX
929
+
930
+
931
+ def _apply_v3_post_neural_guard(row, result):
932
+ return result
armiger.int8.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a0dab886be6252739d4c89ce01d65d8d2135204beaec6b00159d038cf420f329
3
- size 15721824
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c0e96936038988c7465b7fe041c4dc3c92df5a8cb4f56c2666271cd4d3e79420
3
+ size 17599636
armiger.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a0dab886be6252739d4c89ce01d65d8d2135204beaec6b00159d038cf420f329
3
- size 15721824
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c0e96936038988c7465b7fe041c4dc3c92df5a8cb4f56c2666271cd4d3e79420
3
+ size 17599636
armiger_dht_rsi.py CHANGED
@@ -45,14 +45,14 @@ from cryptography.hazmat.primitives.kdf.hkdf import HKDF
45
 
46
 
47
  import sys
48
- _THIS_DIR = pathlib.Path(__file__).resolve().parent
49
- if str(_THIS_DIR) not in sys.path:
50
- sys.path.insert(0, str(_THIS_DIR))
51
 
52
  from dht_rsi_models_and_privacy import (
 
53
  AnonymityGuard,
54
  AnonymityViolationError,
55
  AnonymizedTelemetryRecord,
 
 
56
  BlindedTitleStem,
57
  BundleFormat,
58
  CanaryItem,
@@ -62,6 +62,9 @@ from dht_rsi_models_and_privacy import (
62
  ContinuousAggregationResult,
63
  ContinuousDeltaSubmission,
64
  DHTRSIMode,
 
 
 
65
  EncryptedTelemetryBundle,
66
  MediaCategory,
67
  PeerAttestation,
@@ -73,6 +76,7 @@ from dht_rsi_models_and_privacy import (
73
  VerifiedObservation,
74
  ZeroCueCatalogDelta,
75
  _bep44_canonical_sign_buffer,
 
76
  pure_laplace_noise,
77
  resolve_opt_in_sharing,
78
  )
@@ -91,6 +95,7 @@ from dht_rsi_crypto_and_adapters import (
91
  _OptInAdapterBase,
92
  )
93
  from dht_rsi_verification import (
 
94
  AuthoritativeASNVerifier,
95
  PhysicalGroundTruthVerifier,
96
  SafeFormatValidator,
@@ -136,9 +141,13 @@ class ArmigerDHTRSIEngine:
136
  dp_sigma: float = 0.05,
137
  max_seen_delta_digests: int = 4096,
138
  max_tracked_senders: int = 1024,
 
139
  delta_freshness_window_seconds: int = 14 * 86400,
140
  max_telemetry_queue_size: int = 1024,
141
  max_item_encryption_retries: int = 3,
 
 
 
142
  ) -> None:
143
  self.opt_in_sharing: bool = resolve_opt_in_sharing(opt_in_sharing)
144
  if mode is None:
@@ -151,9 +160,13 @@ class ArmigerDHTRSIEngine:
151
 
152
  self.mode = DHTRSIMode(resolved_mode)
153
  self.current_seq = int(current_seq)
 
154
  self._telemetry_seq = int(current_seq)
155
  self.trusted_signers = trusted_signers or {}
156
  self.trusted_pq_signers = trusted_pq_signers or {}
 
 
 
157
  self.require_pq = bool(require_pq)
158
  self.threshold_m = int(threshold_m)
159
  self.min_distinct_asns = int(min_distinct_asns)
@@ -169,7 +182,12 @@ class ArmigerDHTRSIEngine:
169
  self.max_item_encryption_retries = max(int(max_item_encryption_retries), 1)
170
  self._seen_delta_digests: Dict[str, Tuple[int, Tuple[str, bytes]]] = {}
171
  self._inbound_sender_seq: Dict[Tuple[str, bytes], Tuple[int, int]] = {}
172
- self._evicted_sender_seq_floor: int = int(current_seq)
 
 
 
 
 
173
 
174
  self.verifier = PhysicalGroundTruthVerifier()
175
  self.anonymizer = TelemetryAnonymizer(
@@ -188,6 +206,8 @@ class ArmigerDHTRSIEngine:
188
  self._outbound_bundles_shared: int = 0
189
  self._outbound_packets_blocked_opt_out: int = 0
190
  self._outbound_bytes_shared: int = 0
 
 
191
  self._telemetry_encryption_errors: int = 0
192
  self._telemetry_dropped_backlog: int = 0
193
  self._last_telemetry_error: str = ""
@@ -259,6 +279,8 @@ class ArmigerDHTRSIEngine:
259
  "outbound_bundles_shared": int(self._outbound_bundles_shared),
260
  "outbound_packets_blocked_opt_out": int(self._outbound_packets_blocked_opt_out),
261
  "outbound_bytes_shared": int(self._outbound_bytes_shared),
 
 
262
  "ephemeral_identity_enabled": bool(self.anonymity_guard.ephemeral_identity),
263
  "ephemeral_pubkey_hex": self.anonymity_guard.current_pubkey_hex() if self.opt_in_sharing else None,
264
  "ephemeral_rotations": int(self.anonymity_guard.rotation_count),
@@ -371,28 +393,82 @@ class ArmigerDHTRSIEngine:
371
  for sk in expired_senders:
372
  self._inbound_sender_seq.pop(sk, None)
373
 
 
 
 
 
 
 
374
  def ingest_signed_delta_bundle(
375
  self,
376
  signed_bundle: Mapping[str, Any],
 
 
377
  ) -> Tuple[bool, str, Optional[Dict[str, Any]]]:
378
  """Verify Ed25519 signature and AnonymityGuard schema of an inbound delta bundle (works in pull_only mode)."""
379
  if self.mode == DHTRSIMode.OFF:
380
  return False, "plugin_mode_off", None
381
 
 
 
 
382
  try:
383
  v_raw = signed_bundle["v"]
384
- v_bytes = v_raw if isinstance(v_raw, (bytes, bytearray)) else str(v_raw).encode("utf-8")
385
- seq = int(signed_bundle["seq"])
 
 
 
 
 
 
 
 
 
 
 
 
 
 
386
  salt = signed_bundle.get("salt", b"armiger-rsi-delta")
 
 
387
  if isinstance(salt, str):
388
- salt = salt.encode("utf-8")
389
- salt_bytes = bytes(salt)
 
 
 
 
 
 
 
390
  if len(salt_bytes) > 64:
391
  return False, f"bep44_salt_exceeds_64_bytes:{len(salt_bytes)}", None
392
- pub_raw = signed_bundle.get("k") or signed_bundle.get("pubkey_hex")
393
- pub_bytes = bytes.fromhex(pub_raw) if isinstance(pub_raw, str) else bytes(pub_raw)
394
- sig_raw = signed_bundle.get("sig") or signed_bundle.get("sig_hex")
395
- sig_bytes = bytes.fromhex(sig_raw) if isinstance(sig_raw, str) else bytes(sig_raw)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
396
  except Exception as exc:
397
  return False, f"malformed_bundle_envelope:{exc}", None
398
 
@@ -406,10 +482,10 @@ class ArmigerDHTRSIEngine:
406
  self._prune_inbound_delta_tracking(now_ts)
407
 
408
  sender_key = (pub_bytes.hex(), salt_bytes)
409
- prev_entry = self._inbound_sender_seq.get(sender_key)
410
  min_seq = max(
411
  self.current_seq,
412
- prev_entry[0] if prev_entry is not None else self._evicted_sender_seq_floor,
413
  )
414
  if seq <= min_seq:
415
  return False, f"stale_or_replayed_delta_seq:{seq}<={min_seq}", None
@@ -427,25 +503,33 @@ class ArmigerDHTRSIEngine:
427
  if epoch_val <= 0 or abs(now_ts - epoch_val) > self.delta_freshness_window_seconds:
428
  return False, f"stale_or_future_delta_epoch:{epoch_val}", None
429
 
430
- # Any prior digest for this same sender_key is now superseded by the higher monotonic seq
 
 
 
 
 
 
 
 
 
431
  superseded_digests = [
432
  d for d, (_, sk) in self._seen_delta_digests.items() if sk == sender_key
433
  ]
434
- for d in superseded_digests:
435
- self._seen_delta_digests.pop(d, None)
436
-
437
- while len(self._seen_delta_digests) >= self.max_seen_delta_digests:
438
- oldest_d = next(iter(self._seen_delta_digests))
439
- self._seen_delta_digests.pop(oldest_d, None)
440
- self._seen_delta_digests[bundle_digest] = (now_ts, sender_key)
 
 
 
 
 
 
441
 
442
- if sender_key in self._inbound_sender_seq:
443
- self._inbound_sender_seq.pop(sender_key, None)
444
- while len(self._inbound_sender_seq) >= self.max_tracked_senders:
445
- oldest_sk = next(iter(self._inbound_sender_seq))
446
- evicted_seq, _ = self._inbound_sender_seq.pop(oldest_sk)
447
- self._evicted_sender_seq_floor = max(self._evicted_sender_seq_floor, evicted_seq)
448
- self._inbound_sender_seq[sender_key] = (seq, now_ts)
449
  scale = float(parsed.get("scale", 1.0 / 127.0))
450
  w_q8 = parsed.get("delta_w_q8", [])
451
  b_q8 = parsed.get("delta_b_q8", [])
@@ -459,14 +543,48 @@ class ArmigerDHTRSIEngine:
459
  "reliability_bins": parsed.get("reliability_bins", []),
460
  "sender_ephemeral_pubkey_hex": pub_bytes.hex(),
461
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
462
  return True, "verified_and_ingested", decoded
463
 
464
  def ingest_signed_bundle(
465
  self,
466
  signed_bundle: Mapping[str, Any],
 
 
467
  ) -> Tuple[bool, str, Optional[Dict[str, Any]]]:
468
  """Alias for `ingest_signed_delta_bundle`."""
469
- return self.ingest_signed_delta_bundle(signed_bundle)
470
 
471
  async def publish_to_dht(
472
  self,
@@ -608,6 +726,14 @@ class ArmigerDHTRSIEngine:
608
  require_pq: Optional[bool] = None,
609
  require_authoritative_asn: Optional[bool] = None,
610
  asn_verifier: Optional[AuthoritativeASNVerifier] = None,
 
 
 
 
 
 
 
 
611
  ) -> Tuple[bool, str, Optional[CanaryRegressionReport]]:
612
  """Verify signed DHT update manifest, check ASN quorum, validate safe formats, and run 500-canary gate."""
613
  if self.mode == DHTRSIMode.OFF:
@@ -624,6 +750,27 @@ class ArmigerDHTRSIEngine:
624
  for wots_pk_hex in WOTSPlusSHAKE256._ED25519_TO_WOTS_PUBS.get(norm_ed, ()):
625
  eff_pq_signers[wots_pk_hex] = identity
626
  require_pq_flag = self.require_pq if require_pq is None else bool(require_pq)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
627
  ok, reason = ThresholdSignatureVerifier.verify(
628
  manifest=manifest,
629
  trusted_signers=self.trusted_signers,
@@ -631,6 +778,9 @@ class ArmigerDHTRSIEngine:
631
  current_seq=self.current_seq,
632
  trusted_pq_signers=eff_pq_signers,
633
  require_pq=require_pq_flag,
 
 
 
634
  )
635
  if not ok:
636
  return False, f"manifest_threshold_failed:{reason}", None
@@ -640,14 +790,58 @@ class ArmigerDHTRSIEngine:
640
  if actual_sha.lower() != manifest.bundle_sha256.lower():
641
  return False, "bundle_sha256_digest_mismatch", None
642
 
643
- # 3. Verify K-of-N distinct-ASN Sybil quorum
644
- q_ok, q_reason, asns = SybilQuorumVerifier.verify_quorum(
645
- attestations=attestations,
646
- target_hash=manifest.bundle_sha256,
647
- min_distinct_asns=max(self.min_distinct_asns, manifest.min_asn_quorum),
648
- asn_verifier=asn_verifier,
649
- require_authoritative_asn=require_authoritative_asn,
650
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
651
  if not q_ok:
652
  return False, f"sybil_quorum_failed:{q_reason}", None
653
 
@@ -664,6 +858,280 @@ class ArmigerDHTRSIEngine:
664
  if not canary_report.passed:
665
  return False, f"golden_canary_gate_rejected:{canary_report.reason}", canary_report
666
 
667
- # All checks passed! Advance monotonic sequence
668
  self.current_seq = manifest.seq
 
669
  return True, "update_staged_and_verified", canary_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
45
 
46
 
47
  import sys
 
 
 
48
 
49
  from dht_rsi_models_and_privacy import (
50
+ AcyclicReleaseEnvelope,
51
  AnonymityGuard,
52
  AnonymityViolationError,
53
  AnonymizedTelemetryRecord,
54
+ AuthenticatedConformalBaseline,
55
+ BenchmarkAttestationPredicate,
56
  BlindedTitleStem,
57
  BundleFormat,
58
  CanaryItem,
 
62
  ContinuousAggregationResult,
63
  ContinuousDeltaSubmission,
64
  DHTRSIMode,
65
+ DSSE_TYPE_ATTESTATION_V2,
66
+ DSSE_TYPE_BASELINE_V1,
67
+ DSSE_TYPE_RELEASE_V2,
68
  EncryptedTelemetryBundle,
69
  MediaCategory,
70
  PeerAttestation,
 
76
  VerifiedObservation,
77
  ZeroCueCatalogDelta,
78
  _bep44_canonical_sign_buffer,
79
+ dsse_pae,
80
  pure_laplace_noise,
81
  resolve_opt_in_sharing,
82
  )
 
95
  _OptInAdapterBase,
96
  )
97
  from dht_rsi_verification import (
98
+ AcyclicProvenanceVerifier,
99
  AuthoritativeASNVerifier,
100
  PhysicalGroundTruthVerifier,
101
  SafeFormatValidator,
 
141
  dp_sigma: float = 0.05,
142
  max_seen_delta_digests: int = 4096,
143
  max_tracked_senders: int = 1024,
144
+ max_sender_tombstones: Optional[int] = None,
145
  delta_freshness_window_seconds: int = 14 * 86400,
146
  max_telemetry_queue_size: int = 1024,
147
  max_item_encryption_retries: int = 3,
148
+ trusted_evaluator_signers: Optional[Dict[str, str]] = None,
149
+ trusted_contributor_signers: Optional[Dict[str, str]] = None,
150
+ require_acyclic_envelope: bool = True,
151
  ) -> None:
152
  self.opt_in_sharing: bool = resolve_opt_in_sharing(opt_in_sharing)
153
  if mode is None:
 
160
 
161
  self.mode = DHTRSIMode(resolved_mode)
162
  self.current_seq = int(current_seq)
163
+ self.current_bundle_sha256: Optional[str] = None
164
  self._telemetry_seq = int(current_seq)
165
  self.trusted_signers = trusted_signers or {}
166
  self.trusted_pq_signers = trusted_pq_signers or {}
167
+ self.trusted_evaluator_signers = dict(trusted_evaluator_signers or {})
168
+ self.trusted_contributor_signers = dict(trusted_contributor_signers or {})
169
+ self.require_acyclic_envelope = bool(require_acyclic_envelope)
170
  self.require_pq = bool(require_pq)
171
  self.threshold_m = int(threshold_m)
172
  self.min_distinct_asns = int(min_distinct_asns)
 
182
  self.max_item_encryption_retries = max(int(max_item_encryption_retries), 1)
183
  self._seen_delta_digests: Dict[str, Tuple[int, Tuple[str, bytes]]] = {}
184
  self._inbound_sender_seq: Dict[Tuple[str, bytes], Tuple[int, int]] = {}
185
+ self._evicted_sender_tombstones: Dict[Tuple[str, bytes], Tuple[int, int]] = {}
186
+ self.max_sender_tombstones: int = (
187
+ max(int(max_sender_tombstones), 1)
188
+ if max_sender_tombstones is not None
189
+ else max(self.max_tracked_senders * 8, 256)
190
+ )
191
 
192
  self.verifier = PhysicalGroundTruthVerifier()
193
  self.anonymizer = TelemetryAnonymizer(
 
206
  self._outbound_bundles_shared: int = 0
207
  self._outbound_packets_blocked_opt_out: int = 0
208
  self._outbound_bytes_shared: int = 0
209
+ self._outbound_udp_packets_sent: int = 0
210
+ self._outbound_udp_puts_succeeded: int = 0
211
  self._telemetry_encryption_errors: int = 0
212
  self._telemetry_dropped_backlog: int = 0
213
  self._last_telemetry_error: str = ""
 
279
  "outbound_bundles_shared": int(self._outbound_bundles_shared),
280
  "outbound_packets_blocked_opt_out": int(self._outbound_packets_blocked_opt_out),
281
  "outbound_bytes_shared": int(self._outbound_bytes_shared),
282
+ "outbound_udp_packets_sent": int(getattr(self, "_outbound_udp_packets_sent", 0)),
283
+ "outbound_udp_puts_succeeded": int(getattr(self, "_outbound_udp_puts_succeeded", 0)),
284
  "ephemeral_identity_enabled": bool(self.anonymity_guard.ephemeral_identity),
285
  "ephemeral_pubkey_hex": self.anonymity_guard.current_pubkey_hex() if self.opt_in_sharing else None,
286
  "ephemeral_rotations": int(self.anonymity_guard.rotation_count),
 
393
  for sk in expired_senders:
394
  self._inbound_sender_seq.pop(sk, None)
395
 
396
+ expired_tombstones = [
397
+ sk for sk, (_, seen_ts) in self._evicted_sender_tombstones.items() if seen_ts < cutoff
398
+ ]
399
+ for sk in expired_tombstones:
400
+ self._evicted_sender_tombstones.pop(sk, None)
401
+
402
  def ingest_signed_delta_bundle(
403
  self,
404
  signed_bundle: Mapping[str, Any],
405
+ *,
406
+ commit: bool = True,
407
  ) -> Tuple[bool, str, Optional[Dict[str, Any]]]:
408
  """Verify Ed25519 signature and AnonymityGuard schema of an inbound delta bundle (works in pull_only mode)."""
409
  if self.mode == DHTRSIMode.OFF:
410
  return False, "plugin_mode_off", None
411
 
412
+ if not isinstance(signed_bundle, Mapping):
413
+ return False, "malformed_bundle_envelope:not_mapping", None
414
+
415
  try:
416
  v_raw = signed_bundle["v"]
417
+ if isinstance(v_raw, str):
418
+ if len(v_raw) > 2 * 1024 * 1024:
419
+ return False, "malformed_bundle_envelope:v_too_large", None
420
+ v_bytes = v_raw.encode("utf-8")
421
+ elif isinstance(v_raw, (bytes, bytearray, memoryview)):
422
+ if len(v_raw) > 2 * 1024 * 1024:
423
+ return False, "malformed_bundle_envelope:v_too_large", None
424
+ v_bytes = bytes(v_raw)
425
+ else:
426
+ return False, f"malformed_bundle_envelope:invalid_v_type:{type(v_raw).__name__}", None
427
+
428
+ raw_seq = signed_bundle["seq"]
429
+ if isinstance(raw_seq, bool) or not isinstance(raw_seq, int) or raw_seq < 1 or raw_seq > (2**63 - 1):
430
+ return False, f"malformed_bundle_envelope:invalid_seq:{raw_seq!r}", None
431
+ seq = int(raw_seq)
432
+
433
  salt = signed_bundle.get("salt", b"armiger-rsi-delta")
434
+ if salt is None:
435
+ salt = b"armiger-rsi-delta"
436
  if isinstance(salt, str):
437
+ if len(salt) > 256:
438
+ return False, f"bep44_salt_exceeds_64_bytes:{len(salt)}", None
439
+ salt_bytes = salt.encode("utf-8")
440
+ elif isinstance(salt, (bytes, bytearray, memoryview)):
441
+ if len(salt) > 64:
442
+ return False, f"bep44_salt_exceeds_64_bytes:{len(salt)}", None
443
+ salt_bytes = bytes(salt)
444
+ else:
445
+ return False, f"malformed_bundle_envelope:invalid_salt_type:{type(salt).__name__}", None
446
  if len(salt_bytes) > 64:
447
  return False, f"bep44_salt_exceeds_64_bytes:{len(salt_bytes)}", None
448
+
449
+ pub_raw = signed_bundle["k"] if signed_bundle.get("k") is not None else signed_bundle.get("pubkey_hex")
450
+ if isinstance(pub_raw, str):
451
+ if len(pub_raw.strip()) != 64:
452
+ return False, "malformed_bundle_envelope:invalid_pubkey_hex_length", None
453
+ pub_bytes = bytes.fromhex(pub_raw.strip())
454
+ elif isinstance(pub_raw, (bytes, bytearray, memoryview)):
455
+ if len(pub_raw) != 32:
456
+ return False, f"malformed_bundle_envelope:invalid_pubkey_length:{len(pub_raw)}", None
457
+ pub_bytes = bytes(pub_raw)
458
+ else:
459
+ return False, f"malformed_bundle_envelope:invalid_pubkey_type:{type(pub_raw).__name__}", None
460
+
461
+ sig_raw = signed_bundle["sig"] if signed_bundle.get("sig") is not None else signed_bundle.get("sig_hex")
462
+ if isinstance(sig_raw, str):
463
+ if len(sig_raw.strip()) != 128:
464
+ return False, "malformed_bundle_envelope:invalid_sig_hex_length", None
465
+ sig_bytes = bytes.fromhex(sig_raw.strip())
466
+ elif isinstance(sig_raw, (bytes, bytearray, memoryview)):
467
+ if len(sig_raw) != 64:
468
+ return False, f"malformed_bundle_envelope:invalid_sig_length:{len(sig_raw)}", None
469
+ sig_bytes = bytes(sig_raw)
470
+ else:
471
+ return False, f"malformed_bundle_envelope:invalid_sig_type:{type(sig_raw).__name__}", None
472
  except Exception as exc:
473
  return False, f"malformed_bundle_envelope:{exc}", None
474
 
 
482
  self._prune_inbound_delta_tracking(now_ts)
483
 
484
  sender_key = (pub_bytes.hex(), salt_bytes)
485
+ prev_entry = self._inbound_sender_seq.get(sender_key) or self._evicted_sender_tombstones.get(sender_key)
486
  min_seq = max(
487
  self.current_seq,
488
+ prev_entry[0] if prev_entry is not None else 0,
489
  )
490
  if seq <= min_seq:
491
  return False, f"stale_or_replayed_delta_seq:{seq}<={min_seq}", None
 
503
  if epoch_val <= 0 or abs(now_ts - epoch_val) > self.delta_freshness_window_seconds:
504
  return False, f"stale_or_future_delta_epoch:{epoch_val}", None
505
 
506
+ # Never evict an unexpired sender high-water mark: if tracking capacity (active + tombstones)
507
+ # is full of unexpired senders, reject previously unseen sender identities fail-closed.
508
+ is_known_sender = sender_key in self._inbound_sender_seq or sender_key in self._evicted_sender_tombstones
509
+ if (
510
+ not is_known_sender
511
+ and len(self._inbound_sender_seq) >= self.max_tracked_senders
512
+ and len(self._evicted_sender_tombstones) >= self.max_sender_tombstones
513
+ ):
514
+ return False, "replay_tracking_capacity_exhausted", None
515
+
516
  superseded_digests = [
517
  d for d, (_, sk) in self._seen_delta_digests.items() if sk == sender_key
518
  ]
519
+ projected_digest_count = len(self._seen_delta_digests) - len(superseded_digests)
520
+ if projected_digest_count >= self.max_seen_delta_digests:
521
+ evictable_d = next(
522
+ (
523
+ d
524
+ for d, (_, sk) in self._seen_delta_digests.items()
525
+ if d not in superseded_digests
526
+ and (sk in self._inbound_sender_seq or sk in self._evicted_sender_tombstones or sk == sender_key)
527
+ ),
528
+ None,
529
+ )
530
+ if evictable_d is None:
531
+ return False, "replay_digest_capacity_exhausted", None
532
 
 
 
 
 
 
 
 
533
  scale = float(parsed.get("scale", 1.0 / 127.0))
534
  w_q8 = parsed.get("delta_w_q8", [])
535
  b_q8 = parsed.get("delta_b_q8", [])
 
543
  "reliability_bins": parsed.get("reliability_bins", []),
544
  "sender_ephemeral_pubkey_hex": pub_bytes.hex(),
545
  }
546
+ if not commit:
547
+ return True, "verified_candidate", decoded
548
+
549
+ for d in superseded_digests:
550
+ self._seen_delta_digests.pop(d, None)
551
+
552
+ while len(self._seen_delta_digests) >= self.max_seen_delta_digests:
553
+ evictable_d = next(
554
+ (
555
+ d
556
+ for d, (_, sk) in self._seen_delta_digests.items()
557
+ if sk in self._inbound_sender_seq or sk in self._evicted_sender_tombstones or sk == sender_key
558
+ ),
559
+ None,
560
+ )
561
+ if evictable_d is None:
562
+ return False, "replay_digest_capacity_exhausted", None
563
+ self._seen_delta_digests.pop(evictable_d, None)
564
+
565
+ eff_retention_ts = max(now_ts, epoch_val)
566
+ self._seen_delta_digests[bundle_digest] = (eff_retention_ts, sender_key)
567
+
568
+ self._evicted_sender_tombstones.pop(sender_key, None)
569
+ if sender_key in self._inbound_sender_seq:
570
+ self._inbound_sender_seq.pop(sender_key, None)
571
+ while len(self._inbound_sender_seq) >= self.max_tracked_senders:
572
+ if len(self._evicted_sender_tombstones) >= self.max_sender_tombstones:
573
+ return False, "replay_tracking_capacity_exhausted", None
574
+ oldest_sk = next(iter(self._inbound_sender_seq))
575
+ evicted_seq, evicted_ts = self._inbound_sender_seq.pop(oldest_sk)
576
+ self._evicted_sender_tombstones[oldest_sk] = (evicted_seq, evicted_ts)
577
+ self._inbound_sender_seq[sender_key] = (seq, eff_retention_ts)
578
  return True, "verified_and_ingested", decoded
579
 
580
  def ingest_signed_bundle(
581
  self,
582
  signed_bundle: Mapping[str, Any],
583
+ *,
584
+ commit: bool = True,
585
  ) -> Tuple[bool, str, Optional[Dict[str, Any]]]:
586
  """Alias for `ingest_signed_delta_bundle`."""
587
+ return self.ingest_signed_delta_bundle(signed_bundle, commit=commit)
588
 
589
  async def publish_to_dht(
590
  self,
 
726
  require_pq: Optional[bool] = None,
727
  require_authoritative_asn: Optional[bool] = None,
728
  asn_verifier: Optional[AuthoritativeASNVerifier] = None,
729
+ bootstrap_checkpoint_seq: Optional[int] = None,
730
+ authenticated_baseline: Optional[AuthenticatedConformalBaseline] = None,
731
+ expected_parent_bundle_sha256: Optional[str] = None,
732
+ expected_canary_suite_sha256: Optional[str] = None,
733
+ release_envelope: Optional[AcyclicReleaseEnvelope] = None,
734
+ trusted_evaluator_signers: Optional[Dict[str, str]] = None,
735
+ trusted_contributor_signers: Optional[Dict[str, str]] = None,
736
+ require_acyclic_envelope: Optional[bool] = None,
737
  ) -> Tuple[bool, str, Optional[CanaryRegressionReport]]:
738
  """Verify signed DHT update manifest, check ASN quorum, validate safe formats, and run 500-canary gate."""
739
  if self.mode == DHTRSIMode.OFF:
 
750
  for wots_pk_hex in WOTSPlusSHAKE256._ED25519_TO_WOTS_PUBS.get(norm_ed, ()):
751
  eff_pq_signers[wots_pk_hex] = identity
752
  require_pq_flag = self.require_pq if require_pq is None else bool(require_pq)
753
+ q_ok, q_reason, asns = SybilQuorumVerifier.verify_quorum(
754
+ attestations=attestations,
755
+ target_hash=manifest.bundle_sha256,
756
+ min_distinct_asns=max(self.min_distinct_asns, manifest.min_asn_quorum),
757
+ asn_verifier=asn_verifier,
758
+ require_authoritative_asn=require_authoritative_asn,
759
+ )
760
+ if authenticated_baseline is not None:
761
+ b_ok, b_reason = authenticated_baseline.verify_signature(self.trusted_signers)
762
+ if not b_ok:
763
+ return False, f"authenticated_baseline_failed:{b_reason}", None
764
+ eff_parent_sha = (expected_parent_bundle_sha256 or self.current_bundle_sha256 or "").strip().lower()
765
+ if not eff_parent_sha:
766
+ return False, "authenticated_baseline_missing_expected_parent_bundle_sha256", None
767
+ if authenticated_baseline.parent_bundle_sha256.strip().lower() != eff_parent_sha:
768
+ return False, "authenticated_baseline_parent_bundle_mismatch", None
769
+ eff_suite_sha = (expected_canary_suite_sha256 or self.canary_gate.suite_sha256()).strip().lower()
770
+ if authenticated_baseline.canary_suite_sha256.strip().lower() != eff_suite_sha:
771
+ return False, "authenticated_baseline_canary_suite_mismatch", None
772
+ baseline_metrics = authenticated_baseline.to_metrics_dict()
773
+
774
  ok, reason = ThresholdSignatureVerifier.verify(
775
  manifest=manifest,
776
  trusted_signers=self.trusted_signers,
 
778
  current_seq=self.current_seq,
779
  trusted_pq_signers=eff_pq_signers,
780
  require_pq=require_pq_flag,
781
+ asn_verifier=asn_verifier,
782
+ verified_asn_quorum=asns if q_ok else None,
783
+ bootstrap_checkpoint_seq=bootstrap_checkpoint_seq,
784
  )
785
  if not ok:
786
  return False, f"manifest_threshold_failed:{reason}", None
 
790
  if actual_sha.lower() != manifest.bundle_sha256.lower():
791
  return False, "bundle_sha256_digest_mismatch", None
792
 
793
+ # 2b. Enforce acyclic provenance envelope (disjoint contributors / evaluators / release signers)
794
+ eff_req_env = (
795
+ self.require_acyclic_envelope
796
+ if require_acyclic_envelope is None
797
+ else bool(require_acyclic_envelope)
 
 
798
  )
799
+ if eff_req_env or release_envelope is not None:
800
+ if release_envelope is None:
801
+ return False, "missing_acyclic_release_envelope", None
802
+ if (
803
+ release_envelope.manifest.canonical_payload() != manifest.canonical_payload()
804
+ or dict(release_envelope.manifest.signatures) != dict(manifest.signatures)
805
+ or dict(release_envelope.manifest.pq_signatures) != dict(manifest.pq_signatures)
806
+ ):
807
+ return False, "release_envelope_manifest_mismatch", None
808
+ eff_parent_sha = (expected_parent_bundle_sha256 or self.current_bundle_sha256 or "").strip().lower()
809
+ if not eff_parent_sha:
810
+ return False, "release_envelope_missing_expected_parent_bundle_sha256", None
811
+ if release_envelope.parent_bundle_sha256.strip().lower() != eff_parent_sha:
812
+ return False, "release_envelope_parent_bundle_mismatch", None
813
+ eff_suite_sha = (expected_canary_suite_sha256 or self.canary_gate.suite_sha256()).strip().lower()
814
+ if release_envelope.canary_suite_sha256.strip().lower() != eff_suite_sha:
815
+ return False, "release_envelope_canary_suite_mismatch", None
816
+ eff_eval_signers = (
817
+ dict(trusted_evaluator_signers)
818
+ if trusted_evaluator_signers is not None
819
+ else dict(self.trusted_evaluator_signers)
820
+ )
821
+ eff_contrib_signers = (
822
+ dict(trusted_contributor_signers)
823
+ if trusted_contributor_signers is not None
824
+ else dict(self.trusted_contributor_signers)
825
+ )
826
+ env_ok, env_reason = AcyclicProvenanceVerifier.verify_envelope(
827
+ release_envelope,
828
+ trusted_release_signers=self.trusted_signers,
829
+ trusted_evaluator_signers=eff_eval_signers,
830
+ threshold_m=self.threshold_m,
831
+ min_evaluator_asns=max(self.min_distinct_asns, manifest.min_asn_quorum),
832
+ current_seq=self.current_seq,
833
+ bootstrap_checkpoint_seq=bootstrap_checkpoint_seq,
834
+ require_pq=require_pq_flag,
835
+ asn_verifier=asn_verifier,
836
+ trusted_release_pq_signers=eff_pq_signers,
837
+ trusted_contributor_signers=eff_contrib_signers,
838
+ )
839
+ if not env_ok:
840
+ return False, f"acyclic_provenance_failed:{env_reason}", None
841
+ if authenticated_baseline is None and release_envelope.authenticated_baseline is not None:
842
+ baseline_metrics = release_envelope.authenticated_baseline.to_metrics_dict()
843
+
844
+ # 3. Verify K-of-N distinct-ASN Sybil quorum
845
  if not q_ok:
846
  return False, f"sybil_quorum_failed:{q_reason}", None
847
 
 
858
  if not canary_report.passed:
859
  return False, f"golden_canary_gate_rejected:{canary_report.reason}", canary_report
860
 
861
+ # All checks passed! Advance monotonic sequence and active bundle digest
862
  self.current_seq = manifest.seq
863
+ self.current_bundle_sha256 = actual_sha.lower()
864
  return True, "update_staged_and_verified", canary_report
865
+
866
+ def dispatch_contribution_udp(
867
+ self,
868
+ signed_bundle: Optional[Mapping[str, Any]] = None,
869
+ *,
870
+ delta_w: Optional[Sequence[float]] = None,
871
+ delta_b: Optional[Sequence[float]] = None,
872
+ reliability_bins: Optional[Sequence[Any]] = None,
873
+ n_samples: int = 25,
874
+ epoch: Optional[Union[int, float]] = None,
875
+ seq: Optional[int] = None,
876
+ model_tier: str = "ultra",
877
+ salt: bytes = b"armiger-rsi-delta",
878
+ rotate_key: Optional[bool] = None,
879
+ dht_nodes: Optional[Sequence[Tuple[str, int]]] = None,
880
+ collector_nodes: Optional[Sequence[Tuple[str, int]]] = None,
881
+ discover_collectors: bool = True,
882
+ release_pubkey_hex: Optional[str] = None,
883
+ release_salt: Union[bytes, str, None] = None,
884
+ timeout: float = 2.5,
885
+ max_dht_puts: int = 8,
886
+ ) -> Dict[str, Any]:
887
+ """Publish a signed `< 1,000`-byte BEP-44 delta item over outbound UDP to Mainline DHT
888
+ (`target = SHA-1(ephemeral_pubkey || "armiger-rsi-delta")`) and directly to collector
889
+ DHT nodes (`:6881/udp`) discovered via the release pointer.
890
+
891
+ Strictly sends 0 packets and opens 0 sockets when `opt_in_sharing=False`.
892
+ """
893
+ if not self.opt_in_sharing or self.mode != DHTRSIMode.ANONYMOUS_CONTRIBUTE:
894
+ self._outbound_packets_blocked_opt_out += 1
895
+ return {
896
+ "dispatched": False,
897
+ "reason": "opt_in_sharing_disabled",
898
+ "packets_sent": 0,
899
+ "mainline_puts_ok": 0,
900
+ "collector_puts_ok": 0,
901
+ }
902
+
903
+ if signed_bundle is None:
904
+ bundle = self.export_signed_delta_bundle(
905
+ delta_w=delta_w,
906
+ delta_b=delta_b,
907
+ reliability_bins=reliability_bins,
908
+ n_samples=n_samples,
909
+ epoch=epoch,
910
+ seq=seq,
911
+ model_tier=model_tier,
912
+ salt=salt,
913
+ rotate_key=rotate_key,
914
+ )
915
+ if bundle is None:
916
+ return {
917
+ "dispatched": False,
918
+ "reason": "opt_in_sharing_disabled",
919
+ "packets_sent": 0,
920
+ "mainline_puts_ok": 0,
921
+ "collector_puts_ok": 0,
922
+ }
923
+ else:
924
+ bundle = dict(signed_bundle)
925
+
926
+ v_raw = bundle["v"]
927
+ v_bytes = v_raw.encode("utf-8") if isinstance(v_raw, str) else bytes(v_raw)
928
+ seq_int = int(bundle["seq"])
929
+ salt_raw = bundle.get("salt", b"armiger-rsi-delta")
930
+ salt_bytes = salt_raw.encode("utf-8") if isinstance(salt_raw, str) else bytes(salt_raw)
931
+ k_raw = bundle.get("k") if bundle.get("k") is not None else bundle.get("pubkey_hex")
932
+ pub_bytes = bytes.fromhex(k_raw.strip()) if isinstance(k_raw, str) else bytes(k_raw)
933
+ sig_raw = bundle.get("sig") if bundle.get("sig") is not None else bundle.get("sig_hex")
934
+ sig_bytes = bytes.fromhex(sig_raw.strip()) if isinstance(sig_raw, str) else bytes(sig_raw)
935
+
936
+ # Pre-transmission Canary Firewall & Ed25519 signature verification
937
+ self.anonymity_guard.inspect_wire_payload(v_bytes)
938
+ if len(v_bytes) > 1000:
939
+ raise AnonymityViolationError(f"bep44_value_exceeds_1000_bytes:{len(v_bytes)}")
940
+ vk = ed25519.Ed25519PublicKey.from_public_bytes(pub_bytes)
941
+ vk.verify(sig_bytes, _bep44_canonical_sign_buffer(v=v_bytes, seq=seq_int, salt=salt_bytes))
942
+
943
+ import socket as _socket
944
+ try:
945
+ from verify_dht import (
946
+ DEFAULT_RELEASE_PUBKEY_HEX,
947
+ DEFAULT_RELEASE_SALT,
948
+ _decode_compact_nodes_with_id,
949
+ _xor_distance,
950
+ bdecode_krpc,
951
+ bencode,
952
+ generate_bep42_node_id,
953
+ parse_bootstrap_nodes,
954
+ verify_dht_release,
955
+ )
956
+ except ImportError:
957
+ from verify_dht import (
958
+ DEFAULT_RELEASE_PUBKEY_HEX,
959
+ DEFAULT_RELEASE_SALT,
960
+ _decode_compact_nodes_with_id,
961
+ _xor_distance,
962
+ bdecode_krpc,
963
+ bencode,
964
+ generate_bep42_node_id,
965
+ parse_bootstrap_nodes,
966
+ verify_dht_release,
967
+ )
968
+
969
+ target_20 = hashlib.sha1(pub_bytes + salt_bytes).digest()
970
+ eff_dht_nodes: List[Tuple[str, int]] = (
971
+ list(dht_nodes) if dht_nodes is not None else parse_bootstrap_nodes()
972
+ )
973
+ eff_collectors: List[Tuple[str, int]] = list(collector_nodes or ())
974
+ env_collectors = os.environ.get("ARMIGER_DHT_COLLECTOR_NODES", "").strip()
975
+ if env_collectors:
976
+ for c_addr in parse_bootstrap_nodes(env_collectors):
977
+ if c_addr not in eff_collectors:
978
+ eff_collectors.append(c_addr)
979
+
980
+ if discover_collectors and eff_dht_nodes:
981
+ try:
982
+ ptr_rec = verify_dht_release(
983
+ pubkey_hex=release_pubkey_hex or DEFAULT_RELEASE_PUBKEY_HEX,
984
+ salt=release_salt or DEFAULT_RELEASE_SALT,
985
+ dht_nodes=eff_dht_nodes,
986
+ pointer_only=True,
987
+ timeout=timeout,
988
+ max_dht_queries=16,
989
+ )
990
+ for c_item in ptr_rec.get("collector_nodes", []) or []:
991
+ if isinstance(c_item, (list, tuple)) and len(c_item) == 2:
992
+ c_tup = (str(c_item[0]), int(c_item[1]))
993
+ if c_tup not in eff_collectors:
994
+ eff_collectors.append(c_tup)
995
+ except Exception:
996
+ pass
997
+
998
+ packets_sent = 0
999
+ mainline_puts_ok = 0
1000
+ collector_puts_ok = 0
1001
+ errors: List[str] = []
1002
+ put_attempted_addrs: Set[Tuple[str, int]] = set()
1003
+
1004
+ def _single_socket_get_and_put(addr: Tuple[str, int]) -> Tuple[bool, List[Tuple[bytes, Tuple[str, int]]]]:
1005
+ nonlocal packets_sent
1006
+ host_str, port_int = str(addr[0]).strip(), int(addr[1])
1007
+ try:
1008
+ resolved_ip = _socket.gethostbyname(host_str)
1009
+ except Exception as exc:
1010
+ errors.append(f"{host_str}:{port_int}:resolve:{exc}")
1011
+ return False, []
1012
+ norm_addr = (resolved_ip, port_int)
1013
+ if norm_addr in put_attempted_addrs:
1014
+ return False, []
1015
+ sock = _socket.socket(_socket.AF_INET, _socket.SOCK_DGRAM)
1016
+ refs: List[Tuple[bytes, Tuple[str, int]]] = []
1017
+ try:
1018
+ sock.settimeout(max(0.2, float(timeout)))
1019
+ sock.connect(norm_addr)
1020
+ local_ip = str(sock.getsockname()[0]) or "127.0.0.1"
1021
+ node_id = generate_bep42_node_id(local_ip)
1022
+ q_get = {
1023
+ b"t": b"cg01",
1024
+ b"y": b"q",
1025
+ b"q": b"get",
1026
+ b"a": {b"id": node_id, b"target": target_20},
1027
+ }
1028
+ sock.send(bencode(q_get))
1029
+ packets_sent += 1
1030
+ self._outbound_udp_packets_sent += 1
1031
+ raw_get = sock.recv(8192)
1032
+ msg_get = bdecode_krpc(raw_get)
1033
+ if not isinstance(msg_get, dict) or msg_get.get(b"y") != b"r":
1034
+ return False, refs
1035
+ r_body = msg_get.get(b"r")
1036
+ if not isinstance(r_body, dict):
1037
+ return False, refs
1038
+ refs.extend(_decode_compact_nodes_with_id(r_body.get(b"nodes"), None))
1039
+ for entry in r_body.get(b"nodes_list", []) or []:
1040
+ if (
1041
+ isinstance(entry, list)
1042
+ and len(entry) == 3
1043
+ and isinstance(entry[0], (bytes, bytearray))
1044
+ and len(entry[0]) == 20
1045
+ and isinstance(entry[1], (bytes, bytearray))
1046
+ ):
1047
+ refs.append((bytes(entry[0]), ( bytes(entry[1]).decode("ascii", errors="ignore"), int(entry[2]) )))
1048
+ tok = r_body.get(b"token")
1049
+ if not isinstance(tok, (bytes, bytearray)) or not tok:
1050
+ return False, refs
1051
+ put_attempted_addrs.add(norm_addr)
1052
+ q_put = {
1053
+ b"t": b"cp01",
1054
+ b"y": b"q",
1055
+ b"q": b"put",
1056
+ b"a": {
1057
+ b"id": node_id,
1058
+ b"k": pub_bytes,
1059
+ b"salt": salt_bytes,
1060
+ b"seq": int(seq_int),
1061
+ b"sig": sig_bytes,
1062
+ b"token": bytes(tok),
1063
+ b"v": v_bytes,
1064
+ },
1065
+ }
1066
+ sock.send(bencode(q_put))
1067
+ packets_sent += 1
1068
+ self._outbound_udp_packets_sent += 1
1069
+ raw_put = sock.recv(8192)
1070
+ msg_put = bdecode_krpc(raw_put)
1071
+ if isinstance(msg_put, dict) and msg_put.get(b"y") == b"r":
1072
+ self._outbound_udp_puts_succeeded += 1
1073
+ return True, refs
1074
+ err_val = msg_put.get(b"e_detail") or msg_put.get(b"e") if isinstance(msg_put, dict) else "invalid_resp"
1075
+ errors.append(f"{resolved_ip}:{port_int}:put_rejected:{err_val}")
1076
+ return False, refs
1077
+ except Exception as exc:
1078
+ errors.append(f"{resolved_ip}:{port_int}:{exc}")
1079
+ return False, refs
1080
+ finally:
1081
+ sock.close()
1082
+
1083
+ resolved_collector_set: Set[Tuple[str, int]] = set()
1084
+ for c_host, c_port in eff_collectors:
1085
+ try:
1086
+ c_ip = _socket.gethostbyname(str(c_host).strip())
1087
+ resolved_collector_set.add((c_ip, int(c_port)))
1088
+ except Exception:
1089
+ continue
1090
+
1091
+ # 1. Direct BEP-44 put to discovered/configured collector DHT nodes (:6881/udp)
1092
+ for c_addr in eff_collectors:
1093
+ ok_put, _ = _single_socket_get_and_put(c_addr)
1094
+ if ok_put:
1095
+ collector_puts_ok += 1
1096
+
1097
+ # 2. Iterative BEP-44 lookup + put to Mainline DHT nodes
1098
+ frontier: List[Tuple[int, Tuple[str, int]]] = [
1099
+ (_xor_distance(None, target_20), n_addr) for n_addr in eff_dht_nodes
1100
+ ]
1101
+ visited_walk: Set[Tuple[str, int]] = set()
1102
+ while frontier and len(visited_walk) < 16 and mainline_puts_ok < max_dht_puts:
1103
+ frontier.sort(key=lambda item: item[0])
1104
+ _, cur_addr = frontier.pop(0)
1105
+ try:
1106
+ cur_ip = _socket.gethostbyname(str(cur_addr[0]).strip())
1107
+ norm_cur = (cur_ip, int(cur_addr[1]))
1108
+ except Exception:
1109
+ continue
1110
+ if norm_cur in visited_walk:
1111
+ continue
1112
+ visited_walk.add(norm_cur)
1113
+ if norm_cur in put_attempted_addrs:
1114
+ if norm_cur in resolved_collector_set and collector_puts_ok > 0:
1115
+ mainline_puts_ok += 1
1116
+ continue
1117
+ ok_put, refs = _single_socket_get_and_put(norm_cur)
1118
+ if ok_put:
1119
+ mainline_puts_ok += 1
1120
+ if norm_cur in resolved_collector_set:
1121
+ collector_puts_ok += 1
1122
+ for ref_nid, ref_addr in refs:
1123
+ if ref_addr not in visited_walk:
1124
+ frontier.append((_xor_distance(ref_nid, target_20), ref_addr))
1125
+
1126
+ return {
1127
+ "dispatched": (mainline_puts_ok + collector_puts_ok) > 0,
1128
+ "target_hex": target_20.hex(),
1129
+ "pubkey_hex": pub_bytes.hex(),
1130
+ "seq": seq_int,
1131
+ "payload_bytes": len(v_bytes),
1132
+ "packets_sent": packets_sent,
1133
+ "mainline_puts_ok": mainline_puts_ok,
1134
+ "collector_puts_ok": collector_puts_ok,
1135
+ "collector_nodes": [list(c) for c in eff_collectors],
1136
+ "errors": errors,
1137
+ }
armiger_inference.py CHANGED
@@ -130,7 +130,7 @@ def _extract_token_hashes(text: str) -> frozenset[str]:
130
 
131
  def _get_vocab_ext_map(vocab_path: Optional[Path] = None) -> Dict[str, int]:
132
  global _VOCAB_EXT_MAP
133
- if _VOCAB_EXT_MAP is not None and len(_VOCAB_EXT_MAP) > 64:
134
  return _VOCAB_EXT_MAP
135
  vpath = vocab_path if (vocab_path and vocab_path.exists()) else Path(__file__).resolve().with_name("vocab.json")
136
  if not vpath.exists() and os.environ.get("ARMIGER_MODEL_DIR"):
@@ -469,7 +469,11 @@ class ArmigerStandaloneONNX:
469
  ) -> None:
470
  import onnxruntime as ort
471
 
472
- base = Path(__file__).resolve().parent
 
 
 
 
473
  if onnx_path is None:
474
  for cand in ("armiger.onnx", "armiger.int8.onnx", "armiger_ultra.fp32.onnx"):
475
  if (base / cand).exists():
@@ -485,14 +489,28 @@ class ArmigerStandaloneONNX:
485
  _get_vocab_ext_map(self.vocab_path)
486
  self.calibration_path = Path(calibration_path) if calibration_path else (base / "calibration.json")
487
  self.session = ort.InferenceSession(str(self.onnx_path), providers=["CPUExecutionProvider"])
488
- self.head_names = [o.name for o in self.session.get_outputs()]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
489
 
490
  @staticmethod
491
  def _apply_v3_post_neural_guard(row: Mapping[str, Any], item_res: Dict[str, Any]) -> Dict[str, Any]:
492
  return _apply_v3_post_neural_guard(row, item_res)
493
 
494
  def predict_batch(self, rows: Sequence[Mapping[str, Any]]) -> List[Dict[str, Any]]:
495
- feed = build_onnx_batch_inputs(rows)
496
  raw_outs = self.session.run(self.head_names, feed)
497
  logits_by_head = dict(zip(self.head_names, raw_outs))
498
  results: List[Dict[str, Any]] = []
@@ -507,9 +525,9 @@ class ArmigerStandaloneONNX:
507
  probs_2d = exps / np.maximum(1e-12, np.sum(exps, axis=-1, keepdims=True))
508
  raw_nm = str(rows[i].get("name") or rows[i].get("title") or "")
509
  try:
510
- from armiger.byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans
511
- except ImportError:
512
  from byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans
 
 
513
  raw_b = raw_nm.encode("utf-8", errors="replace")[: vec.shape[0]]
514
  tag_ids = [int(x) for x in decode_bio_logits_viterbi(vec, raw_b).tolist()]
515
  classes = NEURAL_HEAD_CLASSES.get(
 
130
 
131
  def _get_vocab_ext_map(vocab_path: Optional[Path] = None) -> Dict[str, int]:
132
  global _VOCAB_EXT_MAP
133
+ if vocab_path is None and _VOCAB_EXT_MAP is not None and len(_VOCAB_EXT_MAP) > 64:
134
  return _VOCAB_EXT_MAP
135
  vpath = vocab_path if (vocab_path and vocab_path.exists()) else Path(__file__).resolve().with_name("vocab.json")
136
  if not vpath.exists() and os.environ.get("ARMIGER_MODEL_DIR"):
 
469
  ) -> None:
470
  import onnxruntime as ort
471
 
472
+ try:
473
+ from artifacts import model_directory
474
+ base = model_directory()
475
+ except ImportError:
476
+ base = Path(__file__).resolve().parent
477
  if onnx_path is None:
478
  for cand in ("armiger.onnx", "armiger.int8.onnx", "armiger_ultra.fp32.onnx"):
479
  if (base / cand).exists():
 
489
  _get_vocab_ext_map(self.vocab_path)
490
  self.calibration_path = Path(calibration_path) if calibration_path else (base / "calibration.json")
491
  self.session = ort.InferenceSession(str(self.onnx_path), providers=["CPUExecutionProvider"])
492
+ self.head_names = [o.name for o in self.session.get_outputs() if o.name != "bio64"]
493
+ self._batch_fn = build_onnx_batch_inputs
494
+ try:
495
+ for inp in self.session.get_inputs():
496
+ if getattr(inp, "name", "") == "stats":
497
+ sh = getattr(inp, "shape", None)
498
+ if isinstance(sh, (list, tuple)) and len(sh) >= 2 and type(sh[1]) is int and sh[1] == 41:
499
+ try:
500
+ from r13_features import build_onnx_batch_inputs as _r13_batch_fn
501
+ except ImportError:
502
+ from r13_features import build_onnx_batch_inputs as _r13_batch_fn # type: ignore
503
+ self._batch_fn = _r13_batch_fn
504
+ break
505
+ except Exception:
506
+ pass
507
 
508
  @staticmethod
509
  def _apply_v3_post_neural_guard(row: Mapping[str, Any], item_res: Dict[str, Any]) -> Dict[str, Any]:
510
  return _apply_v3_post_neural_guard(row, item_res)
511
 
512
  def predict_batch(self, rows: Sequence[Mapping[str, Any]]) -> List[Dict[str, Any]]:
513
+ feed = self._batch_fn(rows)
514
  raw_outs = self.session.run(self.head_names, feed)
515
  logits_by_head = dict(zip(self.head_names, raw_outs))
516
  results: List[Dict[str, Any]] = []
 
525
  probs_2d = exps / np.maximum(1e-12, np.sum(exps, axis=-1, keepdims=True))
526
  raw_nm = str(rows[i].get("name") or rows[i].get("title") or "")
527
  try:
 
 
528
  from byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans
529
+ except ImportError:
530
+ from byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans # type: ignore
531
  raw_b = raw_nm.encode("utf-8", errors="replace")[: vec.shape[0]]
532
  tag_ids = [int(x) for x in decode_bio_logits_viterbi(vec, raw_b).tolist()]
533
  classes = NEURAL_HEAD_CLASSES.get(
assets/armiger_overview.png ADDED

Git LFS Details

  • SHA256: 460f06a773229ce4f9648c586dd19fd0fd7bc05e746866f25a1fab43bf390ca1
  • Pointer size: 131 Bytes
  • Size of remote file: 299 kB
calibration.json CHANGED
@@ -1,67 +1,84 @@
1
  {
2
- "temperature_scaling_stop_fit": {
3
- "category": {
4
- "temperature_T": 0.65,
5
- "cal_temp_scaled": {
6
- "n": 995,
7
- "fitted_on": "snap_wave13_r5_20261021T115351Z_bfb9dfb67b41_val_guarded"
8
- }
9
- },
10
- "lang": {
11
- "temperature_T": 0.7,
12
- "cal_temp_scaled": {
13
- "n": 368,
14
- "fitted_on": "snap_wave13_r5_20261021T115351Z_bfb9dfb67b41_val_guarded"
15
- }
16
  },
17
- "owner_scope": {
18
- "temperature_T": 0.55,
19
- "cal_temp_scaled": {
20
- "n": 995,
21
- "fitted_on": "snap_wave13_r5_20261021T115351Z_bfb9dfb67b41_val_guarded"
22
- }
 
 
 
23
  },
24
- "title_language": {
25
- "temperature_T": 0.95,
26
- "cal_temp_scaled": {
27
- "n": 566,
28
- "fitted_on": "snap_wave13_r5_20261021T115351Z_bfb9dfb67b41_val_guarded"
29
- }
 
 
30
  },
31
  "english_subs": {
32
- "temperature_T": 0.5,
33
- "cal_temp_scaled": {
34
- "n": 137,
35
- "fitted_on": "snap_wave13_r5_20261021T115351Z_bfb9dfb67b41_val_guarded"
36
- }
 
 
37
  },
38
- "anime": {
39
- "temperature_T": 0.9,
40
- "cal_temp_scaled": {
41
- "n": 243,
42
- "fitted_on": "snap_wave13_r5_20261021T115351Z_bfb9dfb67b41_val_guarded"
43
- }
44
  },
45
- "adult": {
46
- "temperature_T": 0.65,
47
- "cal_temp_scaled": {
48
- "n": 864,
49
- "fitted_on": "snapshot_wave5a_val"
50
- }
 
 
51
  },
52
- "junk": {
53
- "temperature_T": 0.65,
54
- "cal_temp_scaled": {
55
- "n": 864,
56
- "fitted_on": "snapshot_wave5a_val"
57
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
58
  }
59
  },
60
  "menon_logit_adjustment_stop_fit": {
61
- "D80000": {
62
  "tau": 0.0
63
  },
64
- "D100000": {
65
  "tau": 0.0
66
  },
67
  "E100000_TC_v3": {
@@ -83,106 +100,2676 @@
83
  "tau": 0.0
84
  }
85
  },
 
 
 
 
 
 
 
 
 
 
86
  "mondrian_conformal_thresholds": {
87
  "owner_scope": {
88
  "alpha": 0.02,
89
- "threshold_r5clean": 0.52486
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
90
  }
91
  },
92
- "calibration_version": "v3.0-gold-mit3-final-guarded",
93
  "temperature_audit": {
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
94
  "category": {
95
- "n": 995,
96
- "nll_at_T1": 0.284156,
97
- "nll_at_best_T": 0.250881,
98
- "best_T_unclamped": 0.65,
99
- "shipped_T_guarded": 0.65,
100
- "ece5_at_T1_pct": 6.6104,
101
- "ece5_guarded_pct": 2.1656,
102
- "acc_pct": 91.4573
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
103
  },
104
  "lang": {
105
- "n": 368,
106
- "nll_at_T1": 0.308272,
107
- "nll_at_best_T": 0.289053,
108
- "best_T_unclamped": 0.7,
109
- "shipped_T_guarded": 0.7,
110
- "ece5_at_T1_pct": 6.9697,
111
- "ece5_guarded_pct": 3.2334,
112
- "acc_pct": 89.9457
 
 
 
 
 
113
  },
114
  "owner_scope": {
115
- "n": 995,
116
- "nll_at_T1": 0.042003,
117
- "nll_at_best_T": 0.027084,
118
- "best_T_unclamped": 0.55,
119
- "shipped_T_guarded": 0.55,
120
- "ece5_at_T1_pct": 2.1499,
121
- "ece5_guarded_pct": 0.202,
122
- "acc_pct": 98.995
 
 
 
 
 
123
  },
124
  "title_language": {
125
- "n": 566,
126
- "nll_at_T1": 0.137678,
127
- "nll_at_best_T": 0.13748,
128
- "best_T_unclamped": 0.95,
129
- "shipped_T_guarded": 0.95,
130
- "ece5_at_T1_pct": 1.0472,
131
- "ece5_guarded_pct": 0.7014,
132
- "acc_pct": 95.4064
 
 
 
 
 
 
 
 
 
 
 
 
 
 
133
  },
134
- "english_subs": {
135
- "n": 137,
136
- "nll_at_T1": 0.189409,
137
- "nll_at_best_T": 0.132055,
138
- "best_T_unclamped": 0.5,
139
- "shipped_T_guarded": 0.5,
140
- "ece5_at_T1_pct": 9.9213,
141
- "ece5_guarded_pct": 2.5349,
142
- "acc_pct": 96.3504
143
  },
144
- "anime": {
145
- "n": 243,
146
- "nll_at_T1": 0.391146,
147
- "nll_at_best_T": 0.389958,
148
- "best_T_unclamped": 0.9,
149
- "shipped_T_guarded": 0.9,
150
- "ece5_at_T1_pct": 3.8909,
151
- "ece5_guarded_pct": 2.9506,
152
- "acc_pct": 84.7737
153
  },
154
- "adult": {
155
- "inherited": true,
156
- "reason": "no Gold-val label support; retained wave5a fit"
 
 
 
157
  },
158
- "junk": {
159
- "inherited": true,
160
- "reason": "no Gold-val label support; retained wave5a fit"
161
- }
162
- },
163
- "mondrian_thresholds": {
164
- "adult": 0.5451,
165
- "junk": 0.101,
166
- "owner_scope": 0.477
167
- },
168
- "mondrian_audit": {
169
  "owner_scope": {
170
- "alpha": 0.02,
171
- "n_neg": 256,
172
- "n_pos": 739,
173
- "val_fpr_at_thr": 0.015625,
174
- "val_recall_at_thr": 0.994587,
175
- "method": "split-conformal FPR quantile ceil((n+1)(1-alpha)) on val negatives, temperature-scaled probs"
 
 
 
 
 
 
176
  }
177
- },
178
- "fitted_snapshot_id": "snap_wave13_r5_20261021T115351Z_bfb9dfb67b41",
179
- "calibrated_onnx_fp32_sha256": "6459a20d7bf45d46d59efbcb50ffb8a519bacda14ae36403dcc510eceaa38f44",
180
- "mondrian_fitted_calibration_split": {
181
- "adult": 0.799935,
182
- "junk": 0.259358,
183
- "owner_scope": 0.944382,
184
- "method": "split-conformal FPR quantile on packed calibration split (512), temperature-scaled; NOT shipped as gates (recall cost on confirmation) - see mondrian_refit_report"
185
- },
186
- "mondrian_provenance": "Inherited production gates retained for v3.0.0 (higher adult/owner recall on confirmation splits than calibration-fitted FPR gates; fitted alternates recorded for operators).",
187
- "calibrated_onnx_int8_sha256": "a0dab886be6252739d4c89ce01d65d8d2135204beaec6b00159d038cf420f329"
188
  }
 
1
  {
2
+ "calibrated_onnx_fp32_sha256": "6f94aef676e5bec4968378437769cfdc768fefcc6a21854b8319b33456ebd2ec",
3
+ "calibration_version": "v3.0-gold-r13_3-val-only",
4
+ "feature_stats_dim": 41,
5
+ "fitted_snapshot_id": "snap_r13_20261024T140340Z_064c8e6ccb1f",
6
+ "fp32_vs_int8_val_parity": {
7
+ "adult": {
8
+ "fp32_int8_agreement_pct": 100.0,
9
+ "meets_99_5pct_gate": true
 
 
 
 
 
 
10
  },
11
+ "all_heads_min_agreement_pct": 99.57,
12
+ "anime": {
13
+ "bal_acc_drift_pct": 0.11,
14
+ "fp32_int8_agreement_pct": 99.93,
15
+ "fp32_val_bal_acc": 77.67,
16
+ "fp32_val_macro_f1": 75.02,
17
+ "int8_val_bal_acc": 77.78,
18
+ "int8_val_macro_f1": 75.18,
19
+ "meets_99_5pct_gate": true
20
  },
21
+ "category": {
22
+ "bal_acc_drift_pct": 0.34,
23
+ "fp32_int8_agreement_pct": 99.57,
24
+ "fp32_val_bal_acc": 85.15,
25
+ "fp32_val_macro_f1": 84.97,
26
+ "int8_val_bal_acc": 84.81,
27
+ "int8_val_macro_f1": 84.75,
28
+ "meets_99_5pct_gate": true
29
  },
30
  "english_subs": {
31
+ "bal_acc_drift_pct": 0.0,
32
+ "fp32_int8_agreement_pct": 99.57,
33
+ "fp32_val_bal_acc": 97.65,
34
+ "fp32_val_macro_f1": 97.57,
35
+ "int8_val_bal_acc": 97.65,
36
+ "int8_val_macro_f1": 97.57,
37
+ "meets_99_5pct_gate": true
38
  },
39
+ "junk": {
40
+ "fp32_int8_agreement_pct": 100.0,
41
+ "meets_99_5pct_gate": true
 
 
 
42
  },
43
+ "lang": {
44
+ "bal_acc_drift_pct": 0.0,
45
+ "fp32_int8_agreement_pct": 99.79,
46
+ "fp32_val_bal_acc": 73.68,
47
+ "fp32_val_macro_f1": 74.78,
48
+ "int8_val_bal_acc": 73.68,
49
+ "int8_val_macro_f1": 74.78,
50
+ "meets_99_5pct_gate": true
51
  },
52
+ "meets_99_5pct_gate": true,
53
+ "owner_scope": {
54
+ "bal_acc_drift_pct": 0.0,
55
+ "fp32_int8_agreement_pct": 100.0,
56
+ "fp32_val_bal_acc": 96.32,
57
+ "fp32_val_macro_f1": 96.32,
58
+ "int8_val_bal_acc": 96.32,
59
+ "int8_val_macro_f1": 96.32,
60
+ "meets_99_5pct_gate": true
61
+ },
62
+ "span_bio": {
63
+ "fp32_int8_agreement_pct": 99.7,
64
+ "fp32_val_byte_acc": 67.93,
65
+ "int8_val_byte_acc": 67.95
66
+ },
67
+ "title_language": {
68
+ "bal_acc_drift_pct": 0.0,
69
+ "fp32_int8_agreement_pct": 99.86,
70
+ "fp32_val_bal_acc": 83.33,
71
+ "fp32_val_macro_f1": 89.5,
72
+ "int8_val_bal_acc": 83.33,
73
+ "int8_val_macro_f1": 89.5,
74
+ "meets_99_5pct_gate": true
75
  }
76
  },
77
  "menon_logit_adjustment_stop_fit": {
78
+ "D100000": {
79
  "tau": 0.0
80
  },
81
+ "D80000": {
82
  "tau": 0.0
83
  },
84
  "E100000_TC_v3": {
 
100
  "tau": 0.0
101
  }
102
  },
103
+ "mondrian_audit": {
104
+ "owner_scope": {
105
+ "alpha": 0.02,
106
+ "method": "split-conformal FPR quantile ceil((n+1)(1-alpha)) on val negatives, 2-fold temperature-scaled probs",
107
+ "n_neg": 625,
108
+ "n_pos": 786,
109
+ "val_fpr_at_thr": 0.0176,
110
+ "val_recall_at_thr": 0.975827
111
+ }
112
+ },
113
  "mondrian_conformal_thresholds": {
114
  "owner_scope": {
115
  "alpha": 0.02,
116
+ "threshold": 0.280814
117
+ }
118
+ },
119
+ "mondrian_thresholds": {
120
+ "owner_scope": 0.280814
121
+ },
122
+ "owner_scope_r12": {
123
+ "counts": {
124
+ "audiobook": 61,
125
+ "comic": 149,
126
+ "course": 22,
127
+ "ebook": 75,
128
+ "game": 157,
129
+ "movie": 377,
130
+ "music": 184,
131
+ "software": 118,
132
+ "tv_show": 267,
133
+ "unknown": 0,
134
+ "xxx": 1
135
+ },
136
+ "excluded_categories": [
137
+ "music",
138
+ "comic",
139
+ "game",
140
+ "audiobook",
141
+ "software",
142
+ "ebook",
143
+ "course"
144
+ ],
145
+ "feature_stats_dim": 41,
146
+ "fitting": "two-fold group cross-fit audit; full validation refit",
147
+ "folds": [
148
+ {
149
+ "balanced_error": 0.03196061087255235,
150
+ "counts": {
151
+ "audiobook": 32,
152
+ "comic": 82,
153
+ "course": 13,
154
+ "ebook": 41,
155
+ "game": 79,
156
+ "movie": 172,
157
+ "music": 87,
158
+ "software": 65,
159
+ "tv_show": 133,
160
+ "unknown": 0,
161
+ "xxx": 1
162
+ },
163
+ "heldout_fold": 0,
164
+ "holdout_groups": [
165
+ "r10-comp-00005",
166
+ "r10-comp-00049",
167
+ "r10-comp-00059",
168
+ "r10-comp-00067",
169
+ "r10-comp-00084",
170
+ "r10-comp-00092",
171
+ "r10-comp-00122",
172
+ "r10-comp-00176",
173
+ "r10-comp-00204",
174
+ "r10-comp-00221",
175
+ "r10-comp-00236",
176
+ "r10-comp-00238",
177
+ "r10-comp-00240",
178
+ "r10-comp-00269",
179
+ "r10-comp-00311",
180
+ "r10-comp-00343",
181
+ "r10-comp-00371",
182
+ "r10-comp-00379",
183
+ "r10-comp-00384",
184
+ "r10-comp-00393",
185
+ "r10-comp-00450",
186
+ "r10-comp-00469",
187
+ "r10-comp-00542",
188
+ "r10-comp-00553",
189
+ "r10-comp-00558",
190
+ "r10-comp-00560",
191
+ "r10-comp-00563",
192
+ "r10-comp-00610",
193
+ "r10-comp-00614",
194
+ "r10-comp-00619",
195
+ "r10-comp-00626",
196
+ "r10-comp-00646",
197
+ "r10-comp-00675",
198
+ "r10-comp-00705",
199
+ "r10-comp-00732",
200
+ "r10-comp-00749",
201
+ "r10-comp-00768",
202
+ "r10-comp-00769",
203
+ "r10-comp-00777",
204
+ "r10-comp-00817",
205
+ "r10-comp-00848",
206
+ "r10-comp-00874",
207
+ "r10-comp-00892",
208
+ "r10-comp-01011",
209
+ "r10-comp-01014",
210
+ "r10-comp-01039",
211
+ "r10-comp-01047",
212
+ "r10-comp-01136",
213
+ "r10-comp-01277",
214
+ "r10-comp-01309",
215
+ "r10-comp-01320",
216
+ "r10-comp-01341",
217
+ "r10-comp-01353",
218
+ "r10-comp-01356",
219
+ "r10-comp-01405",
220
+ "r10-comp-01441",
221
+ "r10-comp-01445",
222
+ "r10-comp-01447",
223
+ "r10-comp-01485",
224
+ "r10-comp-01566",
225
+ "r10-comp-01622",
226
+ "r10-comp-01626",
227
+ "r10-comp-01647",
228
+ "r10-comp-01693",
229
+ "r10-comp-01725",
230
+ "r10-comp-01751",
231
+ "r10-comp-01760",
232
+ "r10-comp-01795",
233
+ "r10-comp-01808",
234
+ "r10-comp-01816",
235
+ "r10-comp-01826",
236
+ "r10-comp-01827",
237
+ "r10-comp-01860",
238
+ "r10-comp-01893",
239
+ "r10-comp-01907",
240
+ "r10-comp-01941",
241
+ "r10-comp-01972",
242
+ "r10-comp-01981",
243
+ "r10-comp-01999",
244
+ "r10-comp-02018",
245
+ "r10-comp-02137",
246
+ "r10-comp-02146",
247
+ "r10-comp-02206",
248
+ "r10-comp-02236",
249
+ "r10-comp-02302",
250
+ "r10-comp-02305",
251
+ "r10-comp-02341",
252
+ "r10-comp-02361",
253
+ "r10-comp-02373",
254
+ "r10-comp-02396",
255
+ "r10-comp-02421",
256
+ "r10-comp-02448",
257
+ "r10-comp-02497",
258
+ "r10-comp-02548",
259
+ "r10-comp-02553",
260
+ "r10-comp-02591",
261
+ "r10-comp-02593",
262
+ "r10-comp-02600",
263
+ "r10-comp-02622",
264
+ "r10-comp-02640",
265
+ "r10-comp-02654",
266
+ "r10-comp-02698",
267
+ "r10-comp-02764",
268
+ "r10-comp-02824",
269
+ "r10-comp-02839",
270
+ "r10-comp-02930",
271
+ "r10-comp-02983",
272
+ "r10-comp-02990",
273
+ "r10-comp-03005",
274
+ "r10-comp-03040",
275
+ "r10-comp-03133",
276
+ "r10-comp-03188",
277
+ "r10-comp-03229",
278
+ "r10-comp-03239",
279
+ "r10-comp-03260",
280
+ "r10-comp-03265",
281
+ "r10-comp-03274",
282
+ "r10-comp-03282",
283
+ "r10-comp-03298",
284
+ "r10-comp-03300",
285
+ "r10-comp-03317",
286
+ "r10-comp-03384",
287
+ "r10-comp-03436",
288
+ "r10-comp-03535",
289
+ "r10-comp-03565",
290
+ "r10-comp-03586",
291
+ "r10-comp-03590",
292
+ "r10-comp-03597",
293
+ "r10-comp-03603",
294
+ "r10-comp-03633",
295
+ "r10-comp-03648",
296
+ "r10-comp-03649",
297
+ "r10-comp-03681",
298
+ "r10-comp-03711",
299
+ "r10-comp-03823",
300
+ "r10-comp-03838",
301
+ "r10-comp-03846",
302
+ "r10-comp-03847",
303
+ "r10-comp-03857",
304
+ "r10-comp-03859",
305
+ "r10-comp-03877",
306
+ "r10-comp-03892",
307
+ "r10-comp-03894",
308
+ "r10-comp-03916",
309
+ "r10-comp-03924",
310
+ "r10-comp-03942",
311
+ "r10-comp-03957",
312
+ "r10-comp-03963",
313
+ "r10-comp-04000",
314
+ "r10-comp-04029",
315
+ "r10-comp-04049",
316
+ "r10-comp-04114",
317
+ "r10-comp-04119",
318
+ "r10-comp-04139",
319
+ "r10-comp-04197",
320
+ "r10-comp-04238",
321
+ "r10-comp-04251",
322
+ "r10-comp-04270",
323
+ "r10-comp-04271",
324
+ "r10-comp-04278",
325
+ "r10-comp-04286",
326
+ "r10-comp-04297",
327
+ "r10-comp-04322",
328
+ "r10-comp-04340",
329
+ "r10-comp-04342",
330
+ "r10-comp-04349",
331
+ "r10-comp-04365",
332
+ "r10-comp-04385",
333
+ "r10-comp-04393",
334
+ "r10-comp-04438",
335
+ "r10-comp-04450",
336
+ "r10-comp-04494",
337
+ "r10-comp-04505",
338
+ "r10-comp-04534",
339
+ "r10-comp-04544",
340
+ "r10-comp-04555",
341
+ "r10-comp-04594",
342
+ "r10-comp-04596",
343
+ "r10-comp-04605",
344
+ "r10-comp-04637",
345
+ "r10-comp-04656",
346
+ "r10-comp-04657",
347
+ "r10-comp-04660",
348
+ "r10-comp-04671",
349
+ "r10-comp-04701",
350
+ "r10-comp-04717",
351
+ "r10-comp-04725",
352
+ "r10-comp-04740",
353
+ "r10-comp-04742",
354
+ "r10-comp-04753",
355
+ "r10-comp-04755",
356
+ "r10-comp-04762",
357
+ "r10-comp-04772",
358
+ "r10-comp-04776",
359
+ "r10-comp-04783",
360
+ "r10-comp-04796",
361
+ "r10-comp-04812",
362
+ "r10-comp-04827",
363
+ "r10-comp-04845",
364
+ "r10-comp-04871",
365
+ "r10-comp-04876",
366
+ "r10-comp-04879",
367
+ "r10-comp-04887",
368
+ "r10-comp-04898",
369
+ "r10-comp-04913",
370
+ "r10-comp-04919",
371
+ "r10-comp-04956",
372
+ "r10-comp-04966",
373
+ "r10-comp-04986",
374
+ "r10-comp-04990",
375
+ "r10-comp-05023",
376
+ "r10-comp-05027",
377
+ "r10-comp-05029",
378
+ "r10-comp-05038",
379
+ "r10-comp-05061",
380
+ "r10-comp-05070",
381
+ "r10-comp-05080",
382
+ "r10-comp-05081",
383
+ "r10-comp-05083",
384
+ "r10-comp-05099",
385
+ "r10-comp-05106",
386
+ "r10-comp-05111",
387
+ "r10-comp-05151",
388
+ "r10-comp-05157",
389
+ "r10-comp-05162",
390
+ "r10-comp-05168",
391
+ "r10-comp-05178",
392
+ "r10-comp-05218",
393
+ "r10-comp-05224",
394
+ "r10-comp-05234",
395
+ "r10-comp-05276",
396
+ "r10-comp-05309",
397
+ "r10-comp-05310",
398
+ "r10-comp-05316",
399
+ "r10-comp-05337",
400
+ "r10-comp-05349",
401
+ "r10-comp-05350",
402
+ "r10-comp-05352",
403
+ "r10-comp-05356",
404
+ "r10-comp-05366",
405
+ "r10-comp-05369",
406
+ "r10-comp-05370",
407
+ "r10-comp-05381",
408
+ "r10-comp-05383",
409
+ "r10-comp-05384",
410
+ "r10-comp-05389",
411
+ "r10-comp-05390",
412
+ "r10-comp-05427",
413
+ "r10-comp-05440",
414
+ "r10-comp-05452",
415
+ "r10-comp-05465",
416
+ "r10-comp-05466",
417
+ "r10-comp-05481",
418
+ "r10-comp-05486",
419
+ "r10-comp-05490",
420
+ "r10-comp-05539",
421
+ "r10-comp-05542",
422
+ "r10-comp-05545",
423
+ "r10-comp-05547",
424
+ "r10-comp-05553",
425
+ "r10-comp-05606",
426
+ "r10-comp-05621",
427
+ "r10-comp-05630",
428
+ "r10-comp-05653",
429
+ "r10-comp-05668",
430
+ "r10-comp-05682",
431
+ "r10-comp-05685",
432
+ "r10-comp-05688",
433
+ "r10-comp-05691",
434
+ "r10-comp-05711",
435
+ "r10-comp-05723",
436
+ "r10-comp-05725",
437
+ "r10-comp-05737",
438
+ "r10-comp-05744",
439
+ "r10-comp-05751",
440
+ "r10-comp-05755",
441
+ "r10-comp-05783",
442
+ "r10-comp-05789",
443
+ "r10-comp-05802",
444
+ "r10-comp-05849",
445
+ "r10-comp-05877",
446
+ "r10-comp-05880",
447
+ "r10-comp-05892",
448
+ "r10-comp-05908",
449
+ "r10-comp-05928",
450
+ "r10-comp-05957",
451
+ "r10-comp-05958",
452
+ "r10-comp-05980",
453
+ "r10-comp-06003",
454
+ "r10-comp-06019",
455
+ "r10-comp-06024",
456
+ "r10-comp-06032",
457
+ "r10-comp-06051",
458
+ "r10-comp-06057",
459
+ "r10-comp-06079",
460
+ "r10-comp-06089",
461
+ "r10-comp-06109",
462
+ "r10-comp-06139",
463
+ "r10-comp-06140",
464
+ "r10-comp-06151",
465
+ "r10-comp-06194",
466
+ "r10-comp-06196",
467
+ "r10-comp-06216",
468
+ "r10-comp-06230",
469
+ "r10-comp-06231",
470
+ "r10-comp-06286",
471
+ "r10-comp-06293",
472
+ "r10-comp-06306",
473
+ "r10-comp-06330",
474
+ "r10-comp-06354",
475
+ "r10-comp-06366",
476
+ "r10-comp-06391",
477
+ "r10-comp-06412",
478
+ "r10-comp-06415",
479
+ "r10-comp-06420",
480
+ "r10-comp-06423",
481
+ "r10-comp-06451",
482
+ "r10-comp-06457",
483
+ "r10-comp-06467",
484
+ "r10-comp-06498",
485
+ "r10-comp-06505",
486
+ "r10-comp-06523",
487
+ "r10-comp-06524",
488
+ "r10-comp-06545",
489
+ "r10-comp-06579",
490
+ "r10-comp-06590",
491
+ "r10-comp-06595",
492
+ "r10-comp-06638",
493
+ "r10-comp-06654",
494
+ "r10-comp-06676",
495
+ "r10-comp-06691",
496
+ "r10-comp-06702",
497
+ "r10-comp-06711",
498
+ "r10-comp-06721",
499
+ "r10-comp-06744",
500
+ "r10-comp-06779",
501
+ "r10-comp-06785",
502
+ "r10-comp-06802",
503
+ "r10-comp-06815",
504
+ "r10-comp-06843",
505
+ "r10-comp-06863",
506
+ "r10-comp-06874",
507
+ "r10-comp-06885",
508
+ "r10-comp-06886",
509
+ "r10-comp-06911",
510
+ "r10-comp-06918",
511
+ "r10-comp-06927",
512
+ "r10-comp-06930",
513
+ "r10-comp-06931",
514
+ "r10-comp-06943",
515
+ "r10-comp-06959",
516
+ "r10-comp-06962",
517
+ "r10-comp-06971",
518
+ "r10-comp-06995",
519
+ "r10-comp-07009",
520
+ "r10-comp-07010",
521
+ "r10-comp-07014",
522
+ "r10-comp-07017",
523
+ "r10-comp-07018",
524
+ "r10-comp-07024",
525
+ "r10-comp-07025",
526
+ "r10-comp-07037",
527
+ "r10-comp-07069",
528
+ "r10-comp-07070",
529
+ "r10-comp-07076",
530
+ "r10-comp-07092",
531
+ "r10-comp-07097",
532
+ "r10-comp-07105",
533
+ "r10-comp-07110",
534
+ "r10-comp-07123",
535
+ "r10-comp-07127",
536
+ "r10-comp-07148",
537
+ "r10-comp-07153",
538
+ "r10-comp-07196",
539
+ "r10-comp-07224",
540
+ "r10-comp-07229",
541
+ "r10-comp-07239",
542
+ "r10-comp-07256",
543
+ "r10-comp-07268",
544
+ "r10-comp-07271",
545
+ "r10-comp-07284",
546
+ "r10-comp-07288",
547
+ "r10-comp-07293",
548
+ "r10-comp-07308",
549
+ "r10-comp-07313",
550
+ "r10-comp-07334",
551
+ "r10-comp-07347",
552
+ "r10-comp-07353",
553
+ "r10-comp-07365",
554
+ "r10-comp-07377",
555
+ "r10-comp-07387",
556
+ "r10-comp-07391",
557
+ "r10-comp-07404",
558
+ "r10-comp-07408",
559
+ "r10-comp-07410",
560
+ "r10-comp-07420",
561
+ "r10-comp-07423",
562
+ "r10-comp-07435",
563
+ "r10-comp-07441",
564
+ "r10-comp-07451",
565
+ "r10-comp-07453",
566
+ "r10-comp-07458",
567
+ "r10-comp-07466",
568
+ "r10-comp-07475",
569
+ "r10-comp-07487",
570
+ "r10-comp-07511",
571
+ "r10-comp-07517",
572
+ "r10-comp-07522",
573
+ "r10-comp-07530",
574
+ "r10-comp-07531",
575
+ "r10-comp-07533",
576
+ "r10-comp-07537",
577
+ "r10-comp-07539",
578
+ "r10-comp-07540",
579
+ "r10-comp-07548",
580
+ "r10-comp-07579",
581
+ "r10-comp-07585",
582
+ "r10-comp-07631",
583
+ "r10-comp-07641",
584
+ "r10-comp-07654",
585
+ "r10-comp-07657",
586
+ "r10-comp-07658",
587
+ "r10-comp-07660",
588
+ "r10-comp-07669",
589
+ "r10-comp-07674",
590
+ "r10-comp-07687",
591
+ "r10-comp-07750",
592
+ "r10-comp-07755",
593
+ "r10-comp-07767",
594
+ "r10-comp-07785",
595
+ "r10-comp-07786",
596
+ "r10-comp-07789",
597
+ "r10-comp-07801",
598
+ "r10-comp-07819",
599
+ "r10-comp-07821",
600
+ "r10-comp-07824",
601
+ "r10-comp-07827",
602
+ "r10-comp-07869",
603
+ "r10-comp-07884",
604
+ "r10-comp-07886",
605
+ "r10-comp-07887",
606
+ "r10-comp-07889",
607
+ "r10-comp-07895",
608
+ "r10-comp-07914",
609
+ "r10-comp-07933",
610
+ "r10-comp-07944",
611
+ "r10-comp-07972",
612
+ "r10-comp-07979",
613
+ "r10-comp-08007",
614
+ "r10-comp-08019",
615
+ "r10-comp-08026",
616
+ "r10-comp-08042",
617
+ "r10-comp-08048",
618
+ "r10-comp-08049",
619
+ "r10-comp-08050",
620
+ "r10-comp-08057",
621
+ "r10-comp-08068",
622
+ "r10-comp-08069",
623
+ "r10-comp-08070",
624
+ "r10-comp-08082",
625
+ "r10-comp-08087",
626
+ "r10-comp-08088",
627
+ "r10-comp-08117",
628
+ "r10-comp-08124",
629
+ "r10-comp-08126",
630
+ "r10-comp-08129",
631
+ "r10-comp-08132",
632
+ "r10-comp-08148",
633
+ "r10-comp-08163",
634
+ "r10-comp-08171",
635
+ "r10-comp-08189",
636
+ "r10-comp-08190",
637
+ "r10-comp-08196",
638
+ "r10-comp-08202",
639
+ "r10-comp-08212",
640
+ "r10-comp-08248",
641
+ "r10-comp-08253",
642
+ "r10-comp-08254",
643
+ "r10-comp-08257",
644
+ "r10-comp-08263",
645
+ "r10-comp-08274",
646
+ "r10-comp-08293",
647
+ "r10-comp-08294",
648
+ "r10-comp-08298",
649
+ "r10-comp-08301",
650
+ "r10-comp-08302",
651
+ "r10-comp-08333",
652
+ "r10-comp-08337",
653
+ "r10-comp-08356",
654
+ "r10-comp-08367",
655
+ "r10-comp-08378",
656
+ "r10-comp-08379",
657
+ "r10-comp-08383",
658
+ "r10-comp-08400",
659
+ "r10-comp-08408",
660
+ "r10-comp-08419",
661
+ "r10-comp-08433",
662
+ "r10-comp-08470",
663
+ "r10-comp-08477",
664
+ "r10-comp-08490",
665
+ "r10-comp-08493",
666
+ "r10-comp-08507",
667
+ "r10-comp-08515",
668
+ "r10-comp-08530",
669
+ "r10-comp-08538",
670
+ "r10-comp-08539",
671
+ "r10-comp-08556",
672
+ "r10-comp-08569",
673
+ "r10-comp-08590",
674
+ "r10-comp-08598",
675
+ "r10-comp-08634",
676
+ "r10-comp-08657",
677
+ "r10-comp-08674",
678
+ "r10-comp-08684",
679
+ "r10-comp-08686",
680
+ "r10-comp-08690",
681
+ "r10-comp-08695",
682
+ "r10-comp-08702",
683
+ "r10-comp-08706",
684
+ "r10-comp-08721",
685
+ "r10-comp-08742",
686
+ "r10-comp-08776",
687
+ "r10-comp-08795",
688
+ "r10-comp-08803",
689
+ "r10-comp-08806",
690
+ "r10-comp-08816",
691
+ "r10-comp-08817",
692
+ "r10-comp-08824",
693
+ "r10-comp-08835",
694
+ "r10-comp-08841",
695
+ "r10-comp-08843",
696
+ "r10-comp-08855",
697
+ "r10-comp-08870",
698
+ "r10-comp-08893",
699
+ "r10-comp-08909",
700
+ "r10-comp-08926",
701
+ "r10-comp-08932",
702
+ "r10-comp-08941",
703
+ "r10-comp-08946",
704
+ "r10-comp-08956",
705
+ "r10-comp-08972",
706
+ "r10-comp-08985",
707
+ "r10-comp-09006",
708
+ "r10-comp-09031",
709
+ "r10-comp-09041",
710
+ "r10-comp-09044",
711
+ "r10-comp-09051",
712
+ "r10-comp-09063",
713
+ "r10-comp-09066",
714
+ "r10-comp-09085",
715
+ "r10-comp-09086",
716
+ "r10-comp-09087",
717
+ "r10-comp-09092",
718
+ "r10-comp-09100",
719
+ "r10-comp-09137",
720
+ "r10-comp-09187",
721
+ "r10-comp-09189",
722
+ "r10-comp-09192",
723
+ "r10-comp-09196",
724
+ "r10-comp-09199",
725
+ "r10-comp-09207",
726
+ "r10-comp-09210",
727
+ "r10-comp-09222",
728
+ "r10-comp-09229",
729
+ "r10-comp-09240",
730
+ "r10-comp-09242",
731
+ "r10-comp-09249",
732
+ "r10-comp-09255",
733
+ "r10-comp-09258",
734
+ "r10-comp-09272",
735
+ "r10-comp-09273",
736
+ "r10-comp-09291",
737
+ "r10-comp-09292",
738
+ "r10-comp-09326",
739
+ "r10-comp-09329"
740
+ ],
741
+ "n_holdout": 706,
742
+ "n_train": 705,
743
+ "raw_thresholds": {
744
+ "audiobook": 0.364,
745
+ "comic": 0.364,
746
+ "course": 0.364,
747
+ "ebook": 0.364,
748
+ "game": 0.364,
749
+ "movie": 0.364,
750
+ "music": 0.364,
751
+ "software": 0.364,
752
+ "tv_show": 0.364,
753
+ "unknown": 0.364,
754
+ "xxx": 0.364
755
+ },
756
+ "tau_global": 0.364,
757
+ "thresholds": {
758
+ "audiobook": 0.364,
759
+ "comic": 0.36399999999999993,
760
+ "course": 0.364,
761
+ "ebook": 0.364,
762
+ "game": 0.364,
763
+ "movie": 0.364,
764
+ "music": 0.364,
765
+ "software": 0.364,
766
+ "tv_show": 0.364,
767
+ "unknown": 0.364,
768
+ "xxx": 0.364
769
+ },
770
+ "train_groups": [
771
+ "r10-comp-00018",
772
+ "r10-comp-00028",
773
+ "r10-comp-00030",
774
+ "r10-comp-00037",
775
+ "r10-comp-00044",
776
+ "r10-comp-00054",
777
+ "r10-comp-00068",
778
+ "r10-comp-00069",
779
+ "r10-comp-00087",
780
+ "r10-comp-00099",
781
+ "r10-comp-00102",
782
+ "r10-comp-00127",
783
+ "r10-comp-00134",
784
+ "r10-comp-00137",
785
+ "r10-comp-00144",
786
+ "r10-comp-00146",
787
+ "r10-comp-00169",
788
+ "r10-comp-00189",
789
+ "r10-comp-00207",
790
+ "r10-comp-00208",
791
+ "r10-comp-00213",
792
+ "r10-comp-00233",
793
+ "r10-comp-00268",
794
+ "r10-comp-00280",
795
+ "r10-comp-00315",
796
+ "r10-comp-00320",
797
+ "r10-comp-00335",
798
+ "r10-comp-00341",
799
+ "r10-comp-00344",
800
+ "r10-comp-00367",
801
+ "r10-comp-00387",
802
+ "r10-comp-00396",
803
+ "r10-comp-00399",
804
+ "r10-comp-00405",
805
+ "r10-comp-00409",
806
+ "r10-comp-00429",
807
+ "r10-comp-00431",
808
+ "r10-comp-00433",
809
+ "r10-comp-00464",
810
+ "r10-comp-00473",
811
+ "r10-comp-00477",
812
+ "r10-comp-00491",
813
+ "r10-comp-00533",
814
+ "r10-comp-00543",
815
+ "r10-comp-00549",
816
+ "r10-comp-00559",
817
+ "r10-comp-00564",
818
+ "r10-comp-00568",
819
+ "r10-comp-00605",
820
+ "r10-comp-00620",
821
+ "r10-comp-00684",
822
+ "r10-comp-00703",
823
+ "r10-comp-00711",
824
+ "r10-comp-00723",
825
+ "r10-comp-00745",
826
+ "r10-comp-00776",
827
+ "r10-comp-00783",
828
+ "r10-comp-00802",
829
+ "r10-comp-00811",
830
+ "r10-comp-00838",
831
+ "r10-comp-00887",
832
+ "r10-comp-00936",
833
+ "r10-comp-00950",
834
+ "r10-comp-00970",
835
+ "r10-comp-00990",
836
+ "r10-comp-00992",
837
+ "r10-comp-01000",
838
+ "r10-comp-01009",
839
+ "r10-comp-01019",
840
+ "r10-comp-01025",
841
+ "r10-comp-01029",
842
+ "r10-comp-01106",
843
+ "r10-comp-01114",
844
+ "r10-comp-01123",
845
+ "r10-comp-01141",
846
+ "r10-comp-01147",
847
+ "r10-comp-01154",
848
+ "r10-comp-01190",
849
+ "r10-comp-01307",
850
+ "r10-comp-01338",
851
+ "r10-comp-01429",
852
+ "r10-comp-01440",
853
+ "r10-comp-01483",
854
+ "r10-comp-01556",
855
+ "r10-comp-01559",
856
+ "r10-comp-01595",
857
+ "r10-comp-01601",
858
+ "r10-comp-01637",
859
+ "r10-comp-01667",
860
+ "r10-comp-01675",
861
+ "r10-comp-01735",
862
+ "r10-comp-01756",
863
+ "r10-comp-01831",
864
+ "r10-comp-01926",
865
+ "r10-comp-01949",
866
+ "r10-comp-01961",
867
+ "r10-comp-01963",
868
+ "r10-comp-02022",
869
+ "r10-comp-02069",
870
+ "r10-comp-02111",
871
+ "r10-comp-02112",
872
+ "r10-comp-02196",
873
+ "r10-comp-02198",
874
+ "r10-comp-02222",
875
+ "r10-comp-02289",
876
+ "r10-comp-02292",
877
+ "r10-comp-02310",
878
+ "r10-comp-02336",
879
+ "r10-comp-02411",
880
+ "r10-comp-02494",
881
+ "r10-comp-02557",
882
+ "r10-comp-02561",
883
+ "r10-comp-02576",
884
+ "r10-comp-02602",
885
+ "r10-comp-02699",
886
+ "r10-comp-02760",
887
+ "r10-comp-02794",
888
+ "r10-comp-02875",
889
+ "r10-comp-02880",
890
+ "r10-comp-02999",
891
+ "r10-comp-03234",
892
+ "r10-comp-03241",
893
+ "r10-comp-03251",
894
+ "r10-comp-03253",
895
+ "r10-comp-03267",
896
+ "r10-comp-03269",
897
+ "r10-comp-03272",
898
+ "r10-comp-03304",
899
+ "r10-comp-03318",
900
+ "r10-comp-03328",
901
+ "r10-comp-03348",
902
+ "r10-comp-03446",
903
+ "r10-comp-03453",
904
+ "r10-comp-03457",
905
+ "r10-comp-03493",
906
+ "r10-comp-03526",
907
+ "r10-comp-03538",
908
+ "r10-comp-03570",
909
+ "r10-comp-03585",
910
+ "r10-comp-03612",
911
+ "r10-comp-03635",
912
+ "r10-comp-03754",
913
+ "r10-comp-03766",
914
+ "r10-comp-03858",
915
+ "r10-comp-03881",
916
+ "r10-comp-03884",
917
+ "r10-comp-03885",
918
+ "r10-comp-03886",
919
+ "r10-comp-03887",
920
+ "r10-comp-03915",
921
+ "r10-comp-03929",
922
+ "r10-comp-03948",
923
+ "r10-comp-03950",
924
+ "r10-comp-03959",
925
+ "r10-comp-03993",
926
+ "r10-comp-03996",
927
+ "r10-comp-04016",
928
+ "r10-comp-04023",
929
+ "r10-comp-04025",
930
+ "r10-comp-04033",
931
+ "r10-comp-04034",
932
+ "r10-comp-04036",
933
+ "r10-comp-04047",
934
+ "r10-comp-04056",
935
+ "r10-comp-04073",
936
+ "r10-comp-04084",
937
+ "r10-comp-04088",
938
+ "r10-comp-04101",
939
+ "r10-comp-04108",
940
+ "r10-comp-04110",
941
+ "r10-comp-04120",
942
+ "r10-comp-04124",
943
+ "r10-comp-04130",
944
+ "r10-comp-04136",
945
+ "r10-comp-04151",
946
+ "r10-comp-04179",
947
+ "r10-comp-04182",
948
+ "r10-comp-04210",
949
+ "r10-comp-04211",
950
+ "r10-comp-04253",
951
+ "r10-comp-04287",
952
+ "r10-comp-04289",
953
+ "r10-comp-04292",
954
+ "r10-comp-04318",
955
+ "r10-comp-04330",
956
+ "r10-comp-04344",
957
+ "r10-comp-04396",
958
+ "r10-comp-04397",
959
+ "r10-comp-04442",
960
+ "r10-comp-04445",
961
+ "r10-comp-04476",
962
+ "r10-comp-04478",
963
+ "r10-comp-04479",
964
+ "r10-comp-04485",
965
+ "r10-comp-04521",
966
+ "r10-comp-04531",
967
+ "r10-comp-04532",
968
+ "r10-comp-04540",
969
+ "r10-comp-04552",
970
+ "r10-comp-04563",
971
+ "r10-comp-04570",
972
+ "r10-comp-04682",
973
+ "r10-comp-04704",
974
+ "r10-comp-04706",
975
+ "r10-comp-04709",
976
+ "r10-comp-04714",
977
+ "r10-comp-04716",
978
+ "r10-comp-04721",
979
+ "r10-comp-04738",
980
+ "r10-comp-04741",
981
+ "r10-comp-04798",
982
+ "r10-comp-04826",
983
+ "r10-comp-04834",
984
+ "r10-comp-04855",
985
+ "r10-comp-04863",
986
+ "r10-comp-04874",
987
+ "r10-comp-04883",
988
+ "r10-comp-04893",
989
+ "r10-comp-04901",
990
+ "r10-comp-04911",
991
+ "r10-comp-04912",
992
+ "r10-comp-04918",
993
+ "r10-comp-04935",
994
+ "r10-comp-04962",
995
+ "r10-comp-04971",
996
+ "r10-comp-04995",
997
+ "r10-comp-05006",
998
+ "r10-comp-05010",
999
+ "r10-comp-05014",
1000
+ "r10-comp-05017",
1001
+ "r10-comp-05026",
1002
+ "r10-comp-05075",
1003
+ "r10-comp-05079",
1004
+ "r10-comp-05124",
1005
+ "r10-comp-05126",
1006
+ "r10-comp-05140",
1007
+ "r10-comp-05147",
1008
+ "r10-comp-05156",
1009
+ "r10-comp-05163",
1010
+ "r10-comp-05203",
1011
+ "r10-comp-05205",
1012
+ "r10-comp-05206",
1013
+ "r10-comp-05216",
1014
+ "r10-comp-05237",
1015
+ "r10-comp-05308",
1016
+ "r10-comp-05334",
1017
+ "r10-comp-05336",
1018
+ "r10-comp-05346",
1019
+ "r10-comp-05357",
1020
+ "r10-comp-05382",
1021
+ "r10-comp-05404",
1022
+ "r10-comp-05438",
1023
+ "r10-comp-05444",
1024
+ "r10-comp-05457",
1025
+ "r10-comp-05503",
1026
+ "r10-comp-05511",
1027
+ "r10-comp-05529",
1028
+ "r10-comp-05532",
1029
+ "r10-comp-05537",
1030
+ "r10-comp-05597",
1031
+ "r10-comp-05615",
1032
+ "r10-comp-05639",
1033
+ "r10-comp-05641",
1034
+ "r10-comp-05680",
1035
+ "r10-comp-05718",
1036
+ "r10-comp-05724",
1037
+ "r10-comp-05727",
1038
+ "r10-comp-05728",
1039
+ "r10-comp-05768",
1040
+ "r10-comp-05824",
1041
+ "r10-comp-05825",
1042
+ "r10-comp-05830",
1043
+ "r10-comp-05834",
1044
+ "r10-comp-05851",
1045
+ "r10-comp-05867",
1046
+ "r10-comp-05926",
1047
+ "r10-comp-05951",
1048
+ "r10-comp-05973",
1049
+ "r10-comp-05991",
1050
+ "r10-comp-05993",
1051
+ "r10-comp-06007",
1052
+ "r10-comp-06011",
1053
+ "r10-comp-06015",
1054
+ "r10-comp-06020",
1055
+ "r10-comp-06062",
1056
+ "r10-comp-06065",
1057
+ "r10-comp-06077",
1058
+ "r10-comp-06083",
1059
+ "r10-comp-06091",
1060
+ "r10-comp-06098",
1061
+ "r10-comp-06146",
1062
+ "r10-comp-06156",
1063
+ "r10-comp-06158",
1064
+ "r10-comp-06165",
1065
+ "r10-comp-06172",
1066
+ "r10-comp-06188",
1067
+ "r10-comp-06203",
1068
+ "r10-comp-06206",
1069
+ "r10-comp-06208",
1070
+ "r10-comp-06209",
1071
+ "r10-comp-06259",
1072
+ "r10-comp-06275",
1073
+ "r10-comp-06276",
1074
+ "r10-comp-06295",
1075
+ "r10-comp-06303",
1076
+ "r10-comp-06310",
1077
+ "r10-comp-06319",
1078
+ "r10-comp-06344",
1079
+ "r10-comp-06367",
1080
+ "r10-comp-06404",
1081
+ "r10-comp-06414",
1082
+ "r10-comp-06434",
1083
+ "r10-comp-06441",
1084
+ "r10-comp-06444",
1085
+ "r10-comp-06460",
1086
+ "r10-comp-06463",
1087
+ "r10-comp-06486",
1088
+ "r10-comp-06507",
1089
+ "r10-comp-06520",
1090
+ "r10-comp-06527",
1091
+ "r10-comp-06528",
1092
+ "r10-comp-06536",
1093
+ "r10-comp-06540",
1094
+ "r10-comp-06550",
1095
+ "r10-comp-06556",
1096
+ "r10-comp-06558",
1097
+ "r10-comp-06586",
1098
+ "r10-comp-06594",
1099
+ "r10-comp-06620",
1100
+ "r10-comp-06626",
1101
+ "r10-comp-06627",
1102
+ "r10-comp-06652",
1103
+ "r10-comp-06656",
1104
+ "r10-comp-06663",
1105
+ "r10-comp-06665",
1106
+ "r10-comp-06674",
1107
+ "r10-comp-06683",
1108
+ "r10-comp-06699",
1109
+ "r10-comp-06704",
1110
+ "r10-comp-06707",
1111
+ "r10-comp-06717",
1112
+ "r10-comp-06720",
1113
+ "r10-comp-06729",
1114
+ "r10-comp-06734",
1115
+ "r10-comp-06788",
1116
+ "r10-comp-06822",
1117
+ "r10-comp-06838",
1118
+ "r10-comp-06841",
1119
+ "r10-comp-06846",
1120
+ "r10-comp-06867",
1121
+ "r10-comp-06873",
1122
+ "r10-comp-06878",
1123
+ "r10-comp-06879",
1124
+ "r10-comp-06903",
1125
+ "r10-comp-06909",
1126
+ "r10-comp-06942",
1127
+ "r10-comp-06949",
1128
+ "r10-comp-06982",
1129
+ "r10-comp-06990",
1130
+ "r10-comp-07027",
1131
+ "r10-comp-07033",
1132
+ "r10-comp-07043",
1133
+ "r10-comp-07050",
1134
+ "r10-comp-07056",
1135
+ "r10-comp-07060",
1136
+ "r10-comp-07095",
1137
+ "r10-comp-07100",
1138
+ "r10-comp-07108",
1139
+ "r10-comp-07112",
1140
+ "r10-comp-07116",
1141
+ "r10-comp-07165",
1142
+ "r10-comp-07177",
1143
+ "r10-comp-07202",
1144
+ "r10-comp-07211",
1145
+ "r10-comp-07212",
1146
+ "r10-comp-07217",
1147
+ "r10-comp-07228",
1148
+ "r10-comp-07232",
1149
+ "r10-comp-07241",
1150
+ "r10-comp-07246",
1151
+ "r10-comp-07270",
1152
+ "r10-comp-07276",
1153
+ "r10-comp-07283",
1154
+ "r10-comp-07286",
1155
+ "r10-comp-07291",
1156
+ "r10-comp-07299",
1157
+ "r10-comp-07312",
1158
+ "r10-comp-07322",
1159
+ "r10-comp-07328",
1160
+ "r10-comp-07340",
1161
+ "r10-comp-07357",
1162
+ "r10-comp-07361",
1163
+ "r10-comp-07369",
1164
+ "r10-comp-07381",
1165
+ "r10-comp-07406",
1166
+ "r10-comp-07419",
1167
+ "r10-comp-07434",
1168
+ "r10-comp-07437",
1169
+ "r10-comp-07450",
1170
+ "r10-comp-07465",
1171
+ "r10-comp-07468",
1172
+ "r10-comp-07483",
1173
+ "r10-comp-07499",
1174
+ "r10-comp-07513",
1175
+ "r10-comp-07520",
1176
+ "r10-comp-07532",
1177
+ "r10-comp-07538",
1178
+ "r10-comp-07543",
1179
+ "r10-comp-07547",
1180
+ "r10-comp-07554",
1181
+ "r10-comp-07561",
1182
+ "r10-comp-07564",
1183
+ "r10-comp-07566",
1184
+ "r10-comp-07569",
1185
+ "r10-comp-07584",
1186
+ "r10-comp-07587",
1187
+ "r10-comp-07625",
1188
+ "r10-comp-07642",
1189
+ "r10-comp-07644",
1190
+ "r10-comp-07645",
1191
+ "r10-comp-07675",
1192
+ "r10-comp-07766",
1193
+ "r10-comp-07795",
1194
+ "r10-comp-07796",
1195
+ "r10-comp-07808",
1196
+ "r10-comp-07810",
1197
+ "r10-comp-07837",
1198
+ "r10-comp-07847",
1199
+ "r10-comp-07856",
1200
+ "r10-comp-07859",
1201
+ "r10-comp-07871",
1202
+ "r10-comp-07877",
1203
+ "r10-comp-07878",
1204
+ "r10-comp-07913",
1205
+ "r10-comp-07941",
1206
+ "r10-comp-07960",
1207
+ "r10-comp-07992",
1208
+ "r10-comp-08013",
1209
+ "r10-comp-08023",
1210
+ "r10-comp-08027",
1211
+ "r10-comp-08028",
1212
+ "r10-comp-08031",
1213
+ "r10-comp-08034",
1214
+ "r10-comp-08037",
1215
+ "r10-comp-08039",
1216
+ "r10-comp-08040",
1217
+ "r10-comp-08058",
1218
+ "r10-comp-08065",
1219
+ "r10-comp-08074",
1220
+ "r10-comp-08075",
1221
+ "r10-comp-08079",
1222
+ "r10-comp-08084",
1223
+ "r10-comp-08086",
1224
+ "r10-comp-08093",
1225
+ "r10-comp-08095",
1226
+ "r10-comp-08103",
1227
+ "r10-comp-08104",
1228
+ "r10-comp-08120",
1229
+ "r10-comp-08122",
1230
+ "r10-comp-08138",
1231
+ "r10-comp-08139",
1232
+ "r10-comp-08145",
1233
+ "r10-comp-08152",
1234
+ "r10-comp-08154",
1235
+ "r10-comp-08160",
1236
+ "r10-comp-08162",
1237
+ "r10-comp-08167",
1238
+ "r10-comp-08175",
1239
+ "r10-comp-08192",
1240
+ "r10-comp-08203",
1241
+ "r10-comp-08208",
1242
+ "r10-comp-08214",
1243
+ "r10-comp-08219",
1244
+ "r10-comp-08222",
1245
+ "r10-comp-08230",
1246
+ "r10-comp-08234",
1247
+ "r10-comp-08241",
1248
+ "r10-comp-08255",
1249
+ "r10-comp-08260",
1250
+ "r10-comp-08273",
1251
+ "r10-comp-08275",
1252
+ "r10-comp-08276",
1253
+ "r10-comp-08287",
1254
+ "r10-comp-08289",
1255
+ "r10-comp-08299",
1256
+ "r10-comp-08317",
1257
+ "r10-comp-08360",
1258
+ "r10-comp-08391",
1259
+ "r10-comp-08393",
1260
+ "r10-comp-08394",
1261
+ "r10-comp-08405",
1262
+ "r10-comp-08406",
1263
+ "r10-comp-08409",
1264
+ "r10-comp-08420",
1265
+ "r10-comp-08438",
1266
+ "r10-comp-08440",
1267
+ "r10-comp-08452",
1268
+ "r10-comp-08453",
1269
+ "r10-comp-08459",
1270
+ "r10-comp-08465",
1271
+ "r10-comp-08474",
1272
+ "r10-comp-08505",
1273
+ "r10-comp-08555",
1274
+ "r10-comp-08560",
1275
+ "r10-comp-08573",
1276
+ "r10-comp-08575",
1277
+ "r10-comp-08581",
1278
+ "r10-comp-08587",
1279
+ "r10-comp-08617",
1280
+ "r10-comp-08621",
1281
+ "r10-comp-08626",
1282
+ "r10-comp-08631",
1283
+ "r10-comp-08632",
1284
+ "r10-comp-08639",
1285
+ "r10-comp-08645",
1286
+ "r10-comp-08649",
1287
+ "r10-comp-08668",
1288
+ "r10-comp-08682",
1289
+ "r10-comp-08683",
1290
+ "r10-comp-08685",
1291
+ "r10-comp-08689",
1292
+ "r10-comp-08710",
1293
+ "r10-comp-08719",
1294
+ "r10-comp-08720",
1295
+ "r10-comp-08753",
1296
+ "r10-comp-08754",
1297
+ "r10-comp-08757",
1298
+ "r10-comp-08765",
1299
+ "r10-comp-08766",
1300
+ "r10-comp-08807",
1301
+ "r10-comp-08813",
1302
+ "r10-comp-08819",
1303
+ "r10-comp-08823",
1304
+ "r10-comp-08838",
1305
+ "r10-comp-08852",
1306
+ "r10-comp-08857",
1307
+ "r10-comp-08865",
1308
+ "r10-comp-08884",
1309
+ "r10-comp-08895",
1310
+ "r10-comp-08899",
1311
+ "r10-comp-08900",
1312
+ "r10-comp-08915",
1313
+ "r10-comp-08916",
1314
+ "r10-comp-08920",
1315
+ "r10-comp-08923",
1316
+ "r10-comp-08955",
1317
+ "r10-comp-08970",
1318
+ "r10-comp-08988",
1319
+ "r10-comp-09029",
1320
+ "r10-comp-09067",
1321
+ "r10-comp-09072",
1322
+ "r10-comp-09084",
1323
+ "r10-comp-09095",
1324
+ "r10-comp-09113",
1325
+ "r10-comp-09117",
1326
+ "r10-comp-09130",
1327
+ "r10-comp-09145",
1328
+ "r10-comp-09170",
1329
+ "r10-comp-09173",
1330
+ "r10-comp-09182",
1331
+ "r10-comp-09185",
1332
+ "r10-comp-09186",
1333
+ "r10-comp-09212",
1334
+ "r10-comp-09215",
1335
+ "r10-comp-09233",
1336
+ "r10-comp-09239",
1337
+ "r10-comp-09246",
1338
+ "r10-comp-09252",
1339
+ "r10-comp-09274",
1340
+ "r10-comp-09310",
1341
+ "r10-comp-09312",
1342
+ "r10-comp-09314",
1343
+ "r10-comp-09318",
1344
+ "r10-comp-09327"
1345
+ ]
1346
+ },
1347
+ {
1348
+ "balanced_error": 0.0502750811435321,
1349
+ "counts": {
1350
+ "audiobook": 29,
1351
+ "comic": 67,
1352
+ "course": 9,
1353
+ "ebook": 34,
1354
+ "game": 78,
1355
+ "movie": 205,
1356
+ "music": 97,
1357
+ "software": 53,
1358
+ "tv_show": 134,
1359
+ "unknown": 0,
1360
+ "xxx": 0
1361
+ },
1362
+ "heldout_fold": 1,
1363
+ "holdout_groups": [
1364
+ "r10-comp-00018",
1365
+ "r10-comp-00028",
1366
+ "r10-comp-00030",
1367
+ "r10-comp-00037",
1368
+ "r10-comp-00044",
1369
+ "r10-comp-00054",
1370
+ "r10-comp-00068",
1371
+ "r10-comp-00069",
1372
+ "r10-comp-00087",
1373
+ "r10-comp-00099",
1374
+ "r10-comp-00102",
1375
+ "r10-comp-00127",
1376
+ "r10-comp-00134",
1377
+ "r10-comp-00137",
1378
+ "r10-comp-00144",
1379
+ "r10-comp-00146",
1380
+ "r10-comp-00169",
1381
+ "r10-comp-00189",
1382
+ "r10-comp-00207",
1383
+ "r10-comp-00208",
1384
+ "r10-comp-00213",
1385
+ "r10-comp-00233",
1386
+ "r10-comp-00268",
1387
+ "r10-comp-00280",
1388
+ "r10-comp-00315",
1389
+ "r10-comp-00320",
1390
+ "r10-comp-00335",
1391
+ "r10-comp-00341",
1392
+ "r10-comp-00344",
1393
+ "r10-comp-00367",
1394
+ "r10-comp-00387",
1395
+ "r10-comp-00396",
1396
+ "r10-comp-00399",
1397
+ "r10-comp-00405",
1398
+ "r10-comp-00409",
1399
+ "r10-comp-00429",
1400
+ "r10-comp-00431",
1401
+ "r10-comp-00433",
1402
+ "r10-comp-00464",
1403
+ "r10-comp-00473",
1404
+ "r10-comp-00477",
1405
+ "r10-comp-00491",
1406
+ "r10-comp-00533",
1407
+ "r10-comp-00543",
1408
+ "r10-comp-00549",
1409
+ "r10-comp-00559",
1410
+ "r10-comp-00564",
1411
+ "r10-comp-00568",
1412
+ "r10-comp-00605",
1413
+ "r10-comp-00620",
1414
+ "r10-comp-00684",
1415
+ "r10-comp-00703",
1416
+ "r10-comp-00711",
1417
+ "r10-comp-00723",
1418
+ "r10-comp-00745",
1419
+ "r10-comp-00776",
1420
+ "r10-comp-00783",
1421
+ "r10-comp-00802",
1422
+ "r10-comp-00811",
1423
+ "r10-comp-00838",
1424
+ "r10-comp-00887",
1425
+ "r10-comp-00936",
1426
+ "r10-comp-00950",
1427
+ "r10-comp-00970",
1428
+ "r10-comp-00990",
1429
+ "r10-comp-00992",
1430
+ "r10-comp-01000",
1431
+ "r10-comp-01009",
1432
+ "r10-comp-01019",
1433
+ "r10-comp-01025",
1434
+ "r10-comp-01029",
1435
+ "r10-comp-01106",
1436
+ "r10-comp-01114",
1437
+ "r10-comp-01123",
1438
+ "r10-comp-01141",
1439
+ "r10-comp-01147",
1440
+ "r10-comp-01154",
1441
+ "r10-comp-01190",
1442
+ "r10-comp-01307",
1443
+ "r10-comp-01338",
1444
+ "r10-comp-01429",
1445
+ "r10-comp-01440",
1446
+ "r10-comp-01483",
1447
+ "r10-comp-01556",
1448
+ "r10-comp-01559",
1449
+ "r10-comp-01595",
1450
+ "r10-comp-01601",
1451
+ "r10-comp-01637",
1452
+ "r10-comp-01667",
1453
+ "r10-comp-01675",
1454
+ "r10-comp-01735",
1455
+ "r10-comp-01756",
1456
+ "r10-comp-01831",
1457
+ "r10-comp-01926",
1458
+ "r10-comp-01949",
1459
+ "r10-comp-01961",
1460
+ "r10-comp-01963",
1461
+ "r10-comp-02022",
1462
+ "r10-comp-02069",
1463
+ "r10-comp-02111",
1464
+ "r10-comp-02112",
1465
+ "r10-comp-02196",
1466
+ "r10-comp-02198",
1467
+ "r10-comp-02222",
1468
+ "r10-comp-02289",
1469
+ "r10-comp-02292",
1470
+ "r10-comp-02310",
1471
+ "r10-comp-02336",
1472
+ "r10-comp-02411",
1473
+ "r10-comp-02494",
1474
+ "r10-comp-02557",
1475
+ "r10-comp-02561",
1476
+ "r10-comp-02576",
1477
+ "r10-comp-02602",
1478
+ "r10-comp-02699",
1479
+ "r10-comp-02760",
1480
+ "r10-comp-02794",
1481
+ "r10-comp-02875",
1482
+ "r10-comp-02880",
1483
+ "r10-comp-02999",
1484
+ "r10-comp-03234",
1485
+ "r10-comp-03241",
1486
+ "r10-comp-03251",
1487
+ "r10-comp-03253",
1488
+ "r10-comp-03267",
1489
+ "r10-comp-03269",
1490
+ "r10-comp-03272",
1491
+ "r10-comp-03304",
1492
+ "r10-comp-03318",
1493
+ "r10-comp-03328",
1494
+ "r10-comp-03348",
1495
+ "r10-comp-03446",
1496
+ "r10-comp-03453",
1497
+ "r10-comp-03457",
1498
+ "r10-comp-03493",
1499
+ "r10-comp-03526",
1500
+ "r10-comp-03538",
1501
+ "r10-comp-03570",
1502
+ "r10-comp-03585",
1503
+ "r10-comp-03612",
1504
+ "r10-comp-03635",
1505
+ "r10-comp-03754",
1506
+ "r10-comp-03766",
1507
+ "r10-comp-03858",
1508
+ "r10-comp-03881",
1509
+ "r10-comp-03884",
1510
+ "r10-comp-03885",
1511
+ "r10-comp-03886",
1512
+ "r10-comp-03887",
1513
+ "r10-comp-03915",
1514
+ "r10-comp-03929",
1515
+ "r10-comp-03948",
1516
+ "r10-comp-03950",
1517
+ "r10-comp-03959",
1518
+ "r10-comp-03993",
1519
+ "r10-comp-03996",
1520
+ "r10-comp-04016",
1521
+ "r10-comp-04023",
1522
+ "r10-comp-04025",
1523
+ "r10-comp-04033",
1524
+ "r10-comp-04034",
1525
+ "r10-comp-04036",
1526
+ "r10-comp-04047",
1527
+ "r10-comp-04056",
1528
+ "r10-comp-04073",
1529
+ "r10-comp-04084",
1530
+ "r10-comp-04088",
1531
+ "r10-comp-04101",
1532
+ "r10-comp-04108",
1533
+ "r10-comp-04110",
1534
+ "r10-comp-04120",
1535
+ "r10-comp-04124",
1536
+ "r10-comp-04130",
1537
+ "r10-comp-04136",
1538
+ "r10-comp-04151",
1539
+ "r10-comp-04179",
1540
+ "r10-comp-04182",
1541
+ "r10-comp-04210",
1542
+ "r10-comp-04211",
1543
+ "r10-comp-04253",
1544
+ "r10-comp-04287",
1545
+ "r10-comp-04289",
1546
+ "r10-comp-04292",
1547
+ "r10-comp-04318",
1548
+ "r10-comp-04330",
1549
+ "r10-comp-04344",
1550
+ "r10-comp-04396",
1551
+ "r10-comp-04397",
1552
+ "r10-comp-04442",
1553
+ "r10-comp-04445",
1554
+ "r10-comp-04476",
1555
+ "r10-comp-04478",
1556
+ "r10-comp-04479",
1557
+ "r10-comp-04485",
1558
+ "r10-comp-04521",
1559
+ "r10-comp-04531",
1560
+ "r10-comp-04532",
1561
+ "r10-comp-04540",
1562
+ "r10-comp-04552",
1563
+ "r10-comp-04563",
1564
+ "r10-comp-04570",
1565
+ "r10-comp-04682",
1566
+ "r10-comp-04704",
1567
+ "r10-comp-04706",
1568
+ "r10-comp-04709",
1569
+ "r10-comp-04714",
1570
+ "r10-comp-04716",
1571
+ "r10-comp-04721",
1572
+ "r10-comp-04738",
1573
+ "r10-comp-04741",
1574
+ "r10-comp-04798",
1575
+ "r10-comp-04826",
1576
+ "r10-comp-04834",
1577
+ "r10-comp-04855",
1578
+ "r10-comp-04863",
1579
+ "r10-comp-04874",
1580
+ "r10-comp-04883",
1581
+ "r10-comp-04893",
1582
+ "r10-comp-04901",
1583
+ "r10-comp-04911",
1584
+ "r10-comp-04912",
1585
+ "r10-comp-04918",
1586
+ "r10-comp-04935",
1587
+ "r10-comp-04962",
1588
+ "r10-comp-04971",
1589
+ "r10-comp-04995",
1590
+ "r10-comp-05006",
1591
+ "r10-comp-05010",
1592
+ "r10-comp-05014",
1593
+ "r10-comp-05017",
1594
+ "r10-comp-05026",
1595
+ "r10-comp-05075",
1596
+ "r10-comp-05079",
1597
+ "r10-comp-05124",
1598
+ "r10-comp-05126",
1599
+ "r10-comp-05140",
1600
+ "r10-comp-05147",
1601
+ "r10-comp-05156",
1602
+ "r10-comp-05163",
1603
+ "r10-comp-05203",
1604
+ "r10-comp-05205",
1605
+ "r10-comp-05206",
1606
+ "r10-comp-05216",
1607
+ "r10-comp-05237",
1608
+ "r10-comp-05308",
1609
+ "r10-comp-05334",
1610
+ "r10-comp-05336",
1611
+ "r10-comp-05346",
1612
+ "r10-comp-05357",
1613
+ "r10-comp-05382",
1614
+ "r10-comp-05404",
1615
+ "r10-comp-05438",
1616
+ "r10-comp-05444",
1617
+ "r10-comp-05457",
1618
+ "r10-comp-05503",
1619
+ "r10-comp-05511",
1620
+ "r10-comp-05529",
1621
+ "r10-comp-05532",
1622
+ "r10-comp-05537",
1623
+ "r10-comp-05597",
1624
+ "r10-comp-05615",
1625
+ "r10-comp-05639",
1626
+ "r10-comp-05641",
1627
+ "r10-comp-05680",
1628
+ "r10-comp-05718",
1629
+ "r10-comp-05724",
1630
+ "r10-comp-05727",
1631
+ "r10-comp-05728",
1632
+ "r10-comp-05768",
1633
+ "r10-comp-05824",
1634
+ "r10-comp-05825",
1635
+ "r10-comp-05830",
1636
+ "r10-comp-05834",
1637
+ "r10-comp-05851",
1638
+ "r10-comp-05867",
1639
+ "r10-comp-05926",
1640
+ "r10-comp-05951",
1641
+ "r10-comp-05973",
1642
+ "r10-comp-05991",
1643
+ "r10-comp-05993",
1644
+ "r10-comp-06007",
1645
+ "r10-comp-06011",
1646
+ "r10-comp-06015",
1647
+ "r10-comp-06020",
1648
+ "r10-comp-06062",
1649
+ "r10-comp-06065",
1650
+ "r10-comp-06077",
1651
+ "r10-comp-06083",
1652
+ "r10-comp-06091",
1653
+ "r10-comp-06098",
1654
+ "r10-comp-06146",
1655
+ "r10-comp-06156",
1656
+ "r10-comp-06158",
1657
+ "r10-comp-06165",
1658
+ "r10-comp-06172",
1659
+ "r10-comp-06188",
1660
+ "r10-comp-06203",
1661
+ "r10-comp-06206",
1662
+ "r10-comp-06208",
1663
+ "r10-comp-06209",
1664
+ "r10-comp-06259",
1665
+ "r10-comp-06275",
1666
+ "r10-comp-06276",
1667
+ "r10-comp-06295",
1668
+ "r10-comp-06303",
1669
+ "r10-comp-06310",
1670
+ "r10-comp-06319",
1671
+ "r10-comp-06344",
1672
+ "r10-comp-06367",
1673
+ "r10-comp-06404",
1674
+ "r10-comp-06414",
1675
+ "r10-comp-06434",
1676
+ "r10-comp-06441",
1677
+ "r10-comp-06444",
1678
+ "r10-comp-06460",
1679
+ "r10-comp-06463",
1680
+ "r10-comp-06486",
1681
+ "r10-comp-06507",
1682
+ "r10-comp-06520",
1683
+ "r10-comp-06527",
1684
+ "r10-comp-06528",
1685
+ "r10-comp-06536",
1686
+ "r10-comp-06540",
1687
+ "r10-comp-06550",
1688
+ "r10-comp-06556",
1689
+ "r10-comp-06558",
1690
+ "r10-comp-06586",
1691
+ "r10-comp-06594",
1692
+ "r10-comp-06620",
1693
+ "r10-comp-06626",
1694
+ "r10-comp-06627",
1695
+ "r10-comp-06652",
1696
+ "r10-comp-06656",
1697
+ "r10-comp-06663",
1698
+ "r10-comp-06665",
1699
+ "r10-comp-06674",
1700
+ "r10-comp-06683",
1701
+ "r10-comp-06699",
1702
+ "r10-comp-06704",
1703
+ "r10-comp-06707",
1704
+ "r10-comp-06717",
1705
+ "r10-comp-06720",
1706
+ "r10-comp-06729",
1707
+ "r10-comp-06734",
1708
+ "r10-comp-06788",
1709
+ "r10-comp-06822",
1710
+ "r10-comp-06838",
1711
+ "r10-comp-06841",
1712
+ "r10-comp-06846",
1713
+ "r10-comp-06867",
1714
+ "r10-comp-06873",
1715
+ "r10-comp-06878",
1716
+ "r10-comp-06879",
1717
+ "r10-comp-06903",
1718
+ "r10-comp-06909",
1719
+ "r10-comp-06942",
1720
+ "r10-comp-06949",
1721
+ "r10-comp-06982",
1722
+ "r10-comp-06990",
1723
+ "r10-comp-07027",
1724
+ "r10-comp-07033",
1725
+ "r10-comp-07043",
1726
+ "r10-comp-07050",
1727
+ "r10-comp-07056",
1728
+ "r10-comp-07060",
1729
+ "r10-comp-07095",
1730
+ "r10-comp-07100",
1731
+ "r10-comp-07108",
1732
+ "r10-comp-07112",
1733
+ "r10-comp-07116",
1734
+ "r10-comp-07165",
1735
+ "r10-comp-07177",
1736
+ "r10-comp-07202",
1737
+ "r10-comp-07211",
1738
+ "r10-comp-07212",
1739
+ "r10-comp-07217",
1740
+ "r10-comp-07228",
1741
+ "r10-comp-07232",
1742
+ "r10-comp-07241",
1743
+ "r10-comp-07246",
1744
+ "r10-comp-07270",
1745
+ "r10-comp-07276",
1746
+ "r10-comp-07283",
1747
+ "r10-comp-07286",
1748
+ "r10-comp-07291",
1749
+ "r10-comp-07299",
1750
+ "r10-comp-07312",
1751
+ "r10-comp-07322",
1752
+ "r10-comp-07328",
1753
+ "r10-comp-07340",
1754
+ "r10-comp-07357",
1755
+ "r10-comp-07361",
1756
+ "r10-comp-07369",
1757
+ "r10-comp-07381",
1758
+ "r10-comp-07406",
1759
+ "r10-comp-07419",
1760
+ "r10-comp-07434",
1761
+ "r10-comp-07437",
1762
+ "r10-comp-07450",
1763
+ "r10-comp-07465",
1764
+ "r10-comp-07468",
1765
+ "r10-comp-07483",
1766
+ "r10-comp-07499",
1767
+ "r10-comp-07513",
1768
+ "r10-comp-07520",
1769
+ "r10-comp-07532",
1770
+ "r10-comp-07538",
1771
+ "r10-comp-07543",
1772
+ "r10-comp-07547",
1773
+ "r10-comp-07554",
1774
+ "r10-comp-07561",
1775
+ "r10-comp-07564",
1776
+ "r10-comp-07566",
1777
+ "r10-comp-07569",
1778
+ "r10-comp-07584",
1779
+ "r10-comp-07587",
1780
+ "r10-comp-07625",
1781
+ "r10-comp-07642",
1782
+ "r10-comp-07644",
1783
+ "r10-comp-07645",
1784
+ "r10-comp-07675",
1785
+ "r10-comp-07766",
1786
+ "r10-comp-07795",
1787
+ "r10-comp-07796",
1788
+ "r10-comp-07808",
1789
+ "r10-comp-07810",
1790
+ "r10-comp-07837",
1791
+ "r10-comp-07847",
1792
+ "r10-comp-07856",
1793
+ "r10-comp-07859",
1794
+ "r10-comp-07871",
1795
+ "r10-comp-07877",
1796
+ "r10-comp-07878",
1797
+ "r10-comp-07913",
1798
+ "r10-comp-07941",
1799
+ "r10-comp-07960",
1800
+ "r10-comp-07992",
1801
+ "r10-comp-08013",
1802
+ "r10-comp-08023",
1803
+ "r10-comp-08027",
1804
+ "r10-comp-08028",
1805
+ "r10-comp-08031",
1806
+ "r10-comp-08034",
1807
+ "r10-comp-08037",
1808
+ "r10-comp-08039",
1809
+ "r10-comp-08040",
1810
+ "r10-comp-08058",
1811
+ "r10-comp-08065",
1812
+ "r10-comp-08074",
1813
+ "r10-comp-08075",
1814
+ "r10-comp-08079",
1815
+ "r10-comp-08084",
1816
+ "r10-comp-08086",
1817
+ "r10-comp-08093",
1818
+ "r10-comp-08095",
1819
+ "r10-comp-08103",
1820
+ "r10-comp-08104",
1821
+ "r10-comp-08120",
1822
+ "r10-comp-08122",
1823
+ "r10-comp-08138",
1824
+ "r10-comp-08139",
1825
+ "r10-comp-08145",
1826
+ "r10-comp-08152",
1827
+ "r10-comp-08154",
1828
+ "r10-comp-08160",
1829
+ "r10-comp-08162",
1830
+ "r10-comp-08167",
1831
+ "r10-comp-08175",
1832
+ "r10-comp-08192",
1833
+ "r10-comp-08203",
1834
+ "r10-comp-08208",
1835
+ "r10-comp-08214",
1836
+ "r10-comp-08219",
1837
+ "r10-comp-08222",
1838
+ "r10-comp-08230",
1839
+ "r10-comp-08234",
1840
+ "r10-comp-08241",
1841
+ "r10-comp-08255",
1842
+ "r10-comp-08260",
1843
+ "r10-comp-08273",
1844
+ "r10-comp-08275",
1845
+ "r10-comp-08276",
1846
+ "r10-comp-08287",
1847
+ "r10-comp-08289",
1848
+ "r10-comp-08299",
1849
+ "r10-comp-08317",
1850
+ "r10-comp-08360",
1851
+ "r10-comp-08391",
1852
+ "r10-comp-08393",
1853
+ "r10-comp-08394",
1854
+ "r10-comp-08405",
1855
+ "r10-comp-08406",
1856
+ "r10-comp-08409",
1857
+ "r10-comp-08420",
1858
+ "r10-comp-08438",
1859
+ "r10-comp-08440",
1860
+ "r10-comp-08452",
1861
+ "r10-comp-08453",
1862
+ "r10-comp-08459",
1863
+ "r10-comp-08465",
1864
+ "r10-comp-08474",
1865
+ "r10-comp-08505",
1866
+ "r10-comp-08555",
1867
+ "r10-comp-08560",
1868
+ "r10-comp-08573",
1869
+ "r10-comp-08575",
1870
+ "r10-comp-08581",
1871
+ "r10-comp-08587",
1872
+ "r10-comp-08617",
1873
+ "r10-comp-08621",
1874
+ "r10-comp-08626",
1875
+ "r10-comp-08631",
1876
+ "r10-comp-08632",
1877
+ "r10-comp-08639",
1878
+ "r10-comp-08645",
1879
+ "r10-comp-08649",
1880
+ "r10-comp-08668",
1881
+ "r10-comp-08682",
1882
+ "r10-comp-08683",
1883
+ "r10-comp-08685",
1884
+ "r10-comp-08689",
1885
+ "r10-comp-08710",
1886
+ "r10-comp-08719",
1887
+ "r10-comp-08720",
1888
+ "r10-comp-08753",
1889
+ "r10-comp-08754",
1890
+ "r10-comp-08757",
1891
+ "r10-comp-08765",
1892
+ "r10-comp-08766",
1893
+ "r10-comp-08807",
1894
+ "r10-comp-08813",
1895
+ "r10-comp-08819",
1896
+ "r10-comp-08823",
1897
+ "r10-comp-08838",
1898
+ "r10-comp-08852",
1899
+ "r10-comp-08857",
1900
+ "r10-comp-08865",
1901
+ "r10-comp-08884",
1902
+ "r10-comp-08895",
1903
+ "r10-comp-08899",
1904
+ "r10-comp-08900",
1905
+ "r10-comp-08915",
1906
+ "r10-comp-08916",
1907
+ "r10-comp-08920",
1908
+ "r10-comp-08923",
1909
+ "r10-comp-08955",
1910
+ "r10-comp-08970",
1911
+ "r10-comp-08988",
1912
+ "r10-comp-09029",
1913
+ "r10-comp-09067",
1914
+ "r10-comp-09072",
1915
+ "r10-comp-09084",
1916
+ "r10-comp-09095",
1917
+ "r10-comp-09113",
1918
+ "r10-comp-09117",
1919
+ "r10-comp-09130",
1920
+ "r10-comp-09145",
1921
+ "r10-comp-09170",
1922
+ "r10-comp-09173",
1923
+ "r10-comp-09182",
1924
+ "r10-comp-09185",
1925
+ "r10-comp-09186",
1926
+ "r10-comp-09212",
1927
+ "r10-comp-09215",
1928
+ "r10-comp-09233",
1929
+ "r10-comp-09239",
1930
+ "r10-comp-09246",
1931
+ "r10-comp-09252",
1932
+ "r10-comp-09274",
1933
+ "r10-comp-09310",
1934
+ "r10-comp-09312",
1935
+ "r10-comp-09314",
1936
+ "r10-comp-09318",
1937
+ "r10-comp-09327"
1938
+ ],
1939
+ "n_holdout": 705,
1940
+ "n_train": 706,
1941
+ "raw_thresholds": {
1942
+ "audiobook": 0.399,
1943
+ "comic": 0.399,
1944
+ "course": 0.399,
1945
+ "ebook": 0.399,
1946
+ "game": 0.399,
1947
+ "movie": 0.306,
1948
+ "music": 0.399,
1949
+ "software": 0.399,
1950
+ "tv_show": 0.399,
1951
+ "unknown": 0.399,
1952
+ "xxx": 0.399
1953
+ },
1954
+ "tau_global": 0.399,
1955
+ "thresholds": {
1956
+ "audiobook": 0.399,
1957
+ "comic": 0.399,
1958
+ "course": 0.399,
1959
+ "ebook": 0.399,
1960
+ "game": 0.399,
1961
+ "movie": 0.3178723404255319,
1962
+ "music": 0.399,
1963
+ "software": 0.399,
1964
+ "tv_show": 0.399,
1965
+ "unknown": 0.399,
1966
+ "xxx": 0.399
1967
+ },
1968
+ "train_groups": [
1969
+ "r10-comp-00005",
1970
+ "r10-comp-00049",
1971
+ "r10-comp-00059",
1972
+ "r10-comp-00067",
1973
+ "r10-comp-00084",
1974
+ "r10-comp-00092",
1975
+ "r10-comp-00122",
1976
+ "r10-comp-00176",
1977
+ "r10-comp-00204",
1978
+ "r10-comp-00221",
1979
+ "r10-comp-00236",
1980
+ "r10-comp-00238",
1981
+ "r10-comp-00240",
1982
+ "r10-comp-00269",
1983
+ "r10-comp-00311",
1984
+ "r10-comp-00343",
1985
+ "r10-comp-00371",
1986
+ "r10-comp-00379",
1987
+ "r10-comp-00384",
1988
+ "r10-comp-00393",
1989
+ "r10-comp-00450",
1990
+ "r10-comp-00469",
1991
+ "r10-comp-00542",
1992
+ "r10-comp-00553",
1993
+ "r10-comp-00558",
1994
+ "r10-comp-00560",
1995
+ "r10-comp-00563",
1996
+ "r10-comp-00610",
1997
+ "r10-comp-00614",
1998
+ "r10-comp-00619",
1999
+ "r10-comp-00626",
2000
+ "r10-comp-00646",
2001
+ "r10-comp-00675",
2002
+ "r10-comp-00705",
2003
+ "r10-comp-00732",
2004
+ "r10-comp-00749",
2005
+ "r10-comp-00768",
2006
+ "r10-comp-00769",
2007
+ "r10-comp-00777",
2008
+ "r10-comp-00817",
2009
+ "r10-comp-00848",
2010
+ "r10-comp-00874",
2011
+ "r10-comp-00892",
2012
+ "r10-comp-01011",
2013
+ "r10-comp-01014",
2014
+ "r10-comp-01039",
2015
+ "r10-comp-01047",
2016
+ "r10-comp-01136",
2017
+ "r10-comp-01277",
2018
+ "r10-comp-01309",
2019
+ "r10-comp-01320",
2020
+ "r10-comp-01341",
2021
+ "r10-comp-01353",
2022
+ "r10-comp-01356",
2023
+ "r10-comp-01405",
2024
+ "r10-comp-01441",
2025
+ "r10-comp-01445",
2026
+ "r10-comp-01447",
2027
+ "r10-comp-01485",
2028
+ "r10-comp-01566",
2029
+ "r10-comp-01622",
2030
+ "r10-comp-01626",
2031
+ "r10-comp-01647",
2032
+ "r10-comp-01693",
2033
+ "r10-comp-01725",
2034
+ "r10-comp-01751",
2035
+ "r10-comp-01760",
2036
+ "r10-comp-01795",
2037
+ "r10-comp-01808",
2038
+ "r10-comp-01816",
2039
+ "r10-comp-01826",
2040
+ "r10-comp-01827",
2041
+ "r10-comp-01860",
2042
+ "r10-comp-01893",
2043
+ "r10-comp-01907",
2044
+ "r10-comp-01941",
2045
+ "r10-comp-01972",
2046
+ "r10-comp-01981",
2047
+ "r10-comp-01999",
2048
+ "r10-comp-02018",
2049
+ "r10-comp-02137",
2050
+ "r10-comp-02146",
2051
+ "r10-comp-02206",
2052
+ "r10-comp-02236",
2053
+ "r10-comp-02302",
2054
+ "r10-comp-02305",
2055
+ "r10-comp-02341",
2056
+ "r10-comp-02361",
2057
+ "r10-comp-02373",
2058
+ "r10-comp-02396",
2059
+ "r10-comp-02421",
2060
+ "r10-comp-02448",
2061
+ "r10-comp-02497",
2062
+ "r10-comp-02548",
2063
+ "r10-comp-02553",
2064
+ "r10-comp-02591",
2065
+ "r10-comp-02593",
2066
+ "r10-comp-02600",
2067
+ "r10-comp-02622",
2068
+ "r10-comp-02640",
2069
+ "r10-comp-02654",
2070
+ "r10-comp-02698",
2071
+ "r10-comp-02764",
2072
+ "r10-comp-02824",
2073
+ "r10-comp-02839",
2074
+ "r10-comp-02930",
2075
+ "r10-comp-02983",
2076
+ "r10-comp-02990",
2077
+ "r10-comp-03005",
2078
+ "r10-comp-03040",
2079
+ "r10-comp-03133",
2080
+ "r10-comp-03188",
2081
+ "r10-comp-03229",
2082
+ "r10-comp-03239",
2083
+ "r10-comp-03260",
2084
+ "r10-comp-03265",
2085
+ "r10-comp-03274",
2086
+ "r10-comp-03282",
2087
+ "r10-comp-03298",
2088
+ "r10-comp-03300",
2089
+ "r10-comp-03317",
2090
+ "r10-comp-03384",
2091
+ "r10-comp-03436",
2092
+ "r10-comp-03535",
2093
+ "r10-comp-03565",
2094
+ "r10-comp-03586",
2095
+ "r10-comp-03590",
2096
+ "r10-comp-03597",
2097
+ "r10-comp-03603",
2098
+ "r10-comp-03633",
2099
+ "r10-comp-03648",
2100
+ "r10-comp-03649",
2101
+ "r10-comp-03681",
2102
+ "r10-comp-03711",
2103
+ "r10-comp-03823",
2104
+ "r10-comp-03838",
2105
+ "r10-comp-03846",
2106
+ "r10-comp-03847",
2107
+ "r10-comp-03857",
2108
+ "r10-comp-03859",
2109
+ "r10-comp-03877",
2110
+ "r10-comp-03892",
2111
+ "r10-comp-03894",
2112
+ "r10-comp-03916",
2113
+ "r10-comp-03924",
2114
+ "r10-comp-03942",
2115
+ "r10-comp-03957",
2116
+ "r10-comp-03963",
2117
+ "r10-comp-04000",
2118
+ "r10-comp-04029",
2119
+ "r10-comp-04049",
2120
+ "r10-comp-04114",
2121
+ "r10-comp-04119",
2122
+ "r10-comp-04139",
2123
+ "r10-comp-04197",
2124
+ "r10-comp-04238",
2125
+ "r10-comp-04251",
2126
+ "r10-comp-04270",
2127
+ "r10-comp-04271",
2128
+ "r10-comp-04278",
2129
+ "r10-comp-04286",
2130
+ "r10-comp-04297",
2131
+ "r10-comp-04322",
2132
+ "r10-comp-04340",
2133
+ "r10-comp-04342",
2134
+ "r10-comp-04349",
2135
+ "r10-comp-04365",
2136
+ "r10-comp-04385",
2137
+ "r10-comp-04393",
2138
+ "r10-comp-04438",
2139
+ "r10-comp-04450",
2140
+ "r10-comp-04494",
2141
+ "r10-comp-04505",
2142
+ "r10-comp-04534",
2143
+ "r10-comp-04544",
2144
+ "r10-comp-04555",
2145
+ "r10-comp-04594",
2146
+ "r10-comp-04596",
2147
+ "r10-comp-04605",
2148
+ "r10-comp-04637",
2149
+ "r10-comp-04656",
2150
+ "r10-comp-04657",
2151
+ "r10-comp-04660",
2152
+ "r10-comp-04671",
2153
+ "r10-comp-04701",
2154
+ "r10-comp-04717",
2155
+ "r10-comp-04725",
2156
+ "r10-comp-04740",
2157
+ "r10-comp-04742",
2158
+ "r10-comp-04753",
2159
+ "r10-comp-04755",
2160
+ "r10-comp-04762",
2161
+ "r10-comp-04772",
2162
+ "r10-comp-04776",
2163
+ "r10-comp-04783",
2164
+ "r10-comp-04796",
2165
+ "r10-comp-04812",
2166
+ "r10-comp-04827",
2167
+ "r10-comp-04845",
2168
+ "r10-comp-04871",
2169
+ "r10-comp-04876",
2170
+ "r10-comp-04879",
2171
+ "r10-comp-04887",
2172
+ "r10-comp-04898",
2173
+ "r10-comp-04913",
2174
+ "r10-comp-04919",
2175
+ "r10-comp-04956",
2176
+ "r10-comp-04966",
2177
+ "r10-comp-04986",
2178
+ "r10-comp-04990",
2179
+ "r10-comp-05023",
2180
+ "r10-comp-05027",
2181
+ "r10-comp-05029",
2182
+ "r10-comp-05038",
2183
+ "r10-comp-05061",
2184
+ "r10-comp-05070",
2185
+ "r10-comp-05080",
2186
+ "r10-comp-05081",
2187
+ "r10-comp-05083",
2188
+ "r10-comp-05099",
2189
+ "r10-comp-05106",
2190
+ "r10-comp-05111",
2191
+ "r10-comp-05151",
2192
+ "r10-comp-05157",
2193
+ "r10-comp-05162",
2194
+ "r10-comp-05168",
2195
+ "r10-comp-05178",
2196
+ "r10-comp-05218",
2197
+ "r10-comp-05224",
2198
+ "r10-comp-05234",
2199
+ "r10-comp-05276",
2200
+ "r10-comp-05309",
2201
+ "r10-comp-05310",
2202
+ "r10-comp-05316",
2203
+ "r10-comp-05337",
2204
+ "r10-comp-05349",
2205
+ "r10-comp-05350",
2206
+ "r10-comp-05352",
2207
+ "r10-comp-05356",
2208
+ "r10-comp-05366",
2209
+ "r10-comp-05369",
2210
+ "r10-comp-05370",
2211
+ "r10-comp-05381",
2212
+ "r10-comp-05383",
2213
+ "r10-comp-05384",
2214
+ "r10-comp-05389",
2215
+ "r10-comp-05390",
2216
+ "r10-comp-05427",
2217
+ "r10-comp-05440",
2218
+ "r10-comp-05452",
2219
+ "r10-comp-05465",
2220
+ "r10-comp-05466",
2221
+ "r10-comp-05481",
2222
+ "r10-comp-05486",
2223
+ "r10-comp-05490",
2224
+ "r10-comp-05539",
2225
+ "r10-comp-05542",
2226
+ "r10-comp-05545",
2227
+ "r10-comp-05547",
2228
+ "r10-comp-05553",
2229
+ "r10-comp-05606",
2230
+ "r10-comp-05621",
2231
+ "r10-comp-05630",
2232
+ "r10-comp-05653",
2233
+ "r10-comp-05668",
2234
+ "r10-comp-05682",
2235
+ "r10-comp-05685",
2236
+ "r10-comp-05688",
2237
+ "r10-comp-05691",
2238
+ "r10-comp-05711",
2239
+ "r10-comp-05723",
2240
+ "r10-comp-05725",
2241
+ "r10-comp-05737",
2242
+ "r10-comp-05744",
2243
+ "r10-comp-05751",
2244
+ "r10-comp-05755",
2245
+ "r10-comp-05783",
2246
+ "r10-comp-05789",
2247
+ "r10-comp-05802",
2248
+ "r10-comp-05849",
2249
+ "r10-comp-05877",
2250
+ "r10-comp-05880",
2251
+ "r10-comp-05892",
2252
+ "r10-comp-05908",
2253
+ "r10-comp-05928",
2254
+ "r10-comp-05957",
2255
+ "r10-comp-05958",
2256
+ "r10-comp-05980",
2257
+ "r10-comp-06003",
2258
+ "r10-comp-06019",
2259
+ "r10-comp-06024",
2260
+ "r10-comp-06032",
2261
+ "r10-comp-06051",
2262
+ "r10-comp-06057",
2263
+ "r10-comp-06079",
2264
+ "r10-comp-06089",
2265
+ "r10-comp-06109",
2266
+ "r10-comp-06139",
2267
+ "r10-comp-06140",
2268
+ "r10-comp-06151",
2269
+ "r10-comp-06194",
2270
+ "r10-comp-06196",
2271
+ "r10-comp-06216",
2272
+ "r10-comp-06230",
2273
+ "r10-comp-06231",
2274
+ "r10-comp-06286",
2275
+ "r10-comp-06293",
2276
+ "r10-comp-06306",
2277
+ "r10-comp-06330",
2278
+ "r10-comp-06354",
2279
+ "r10-comp-06366",
2280
+ "r10-comp-06391",
2281
+ "r10-comp-06412",
2282
+ "r10-comp-06415",
2283
+ "r10-comp-06420",
2284
+ "r10-comp-06423",
2285
+ "r10-comp-06451",
2286
+ "r10-comp-06457",
2287
+ "r10-comp-06467",
2288
+ "r10-comp-06498",
2289
+ "r10-comp-06505",
2290
+ "r10-comp-06523",
2291
+ "r10-comp-06524",
2292
+ "r10-comp-06545",
2293
+ "r10-comp-06579",
2294
+ "r10-comp-06590",
2295
+ "r10-comp-06595",
2296
+ "r10-comp-06638",
2297
+ "r10-comp-06654",
2298
+ "r10-comp-06676",
2299
+ "r10-comp-06691",
2300
+ "r10-comp-06702",
2301
+ "r10-comp-06711",
2302
+ "r10-comp-06721",
2303
+ "r10-comp-06744",
2304
+ "r10-comp-06779",
2305
+ "r10-comp-06785",
2306
+ "r10-comp-06802",
2307
+ "r10-comp-06815",
2308
+ "r10-comp-06843",
2309
+ "r10-comp-06863",
2310
+ "r10-comp-06874",
2311
+ "r10-comp-06885",
2312
+ "r10-comp-06886",
2313
+ "r10-comp-06911",
2314
+ "r10-comp-06918",
2315
+ "r10-comp-06927",
2316
+ "r10-comp-06930",
2317
+ "r10-comp-06931",
2318
+ "r10-comp-06943",
2319
+ "r10-comp-06959",
2320
+ "r10-comp-06962",
2321
+ "r10-comp-06971",
2322
+ "r10-comp-06995",
2323
+ "r10-comp-07009",
2324
+ "r10-comp-07010",
2325
+ "r10-comp-07014",
2326
+ "r10-comp-07017",
2327
+ "r10-comp-07018",
2328
+ "r10-comp-07024",
2329
+ "r10-comp-07025",
2330
+ "r10-comp-07037",
2331
+ "r10-comp-07069",
2332
+ "r10-comp-07070",
2333
+ "r10-comp-07076",
2334
+ "r10-comp-07092",
2335
+ "r10-comp-07097",
2336
+ "r10-comp-07105",
2337
+ "r10-comp-07110",
2338
+ "r10-comp-07123",
2339
+ "r10-comp-07127",
2340
+ "r10-comp-07148",
2341
+ "r10-comp-07153",
2342
+ "r10-comp-07196",
2343
+ "r10-comp-07224",
2344
+ "r10-comp-07229",
2345
+ "r10-comp-07239",
2346
+ "r10-comp-07256",
2347
+ "r10-comp-07268",
2348
+ "r10-comp-07271",
2349
+ "r10-comp-07284",
2350
+ "r10-comp-07288",
2351
+ "r10-comp-07293",
2352
+ "r10-comp-07308",
2353
+ "r10-comp-07313",
2354
+ "r10-comp-07334",
2355
+ "r10-comp-07347",
2356
+ "r10-comp-07353",
2357
+ "r10-comp-07365",
2358
+ "r10-comp-07377",
2359
+ "r10-comp-07387",
2360
+ "r10-comp-07391",
2361
+ "r10-comp-07404",
2362
+ "r10-comp-07408",
2363
+ "r10-comp-07410",
2364
+ "r10-comp-07420",
2365
+ "r10-comp-07423",
2366
+ "r10-comp-07435",
2367
+ "r10-comp-07441",
2368
+ "r10-comp-07451",
2369
+ "r10-comp-07453",
2370
+ "r10-comp-07458",
2371
+ "r10-comp-07466",
2372
+ "r10-comp-07475",
2373
+ "r10-comp-07487",
2374
+ "r10-comp-07511",
2375
+ "r10-comp-07517",
2376
+ "r10-comp-07522",
2377
+ "r10-comp-07530",
2378
+ "r10-comp-07531",
2379
+ "r10-comp-07533",
2380
+ "r10-comp-07537",
2381
+ "r10-comp-07539",
2382
+ "r10-comp-07540",
2383
+ "r10-comp-07548",
2384
+ "r10-comp-07579",
2385
+ "r10-comp-07585",
2386
+ "r10-comp-07631",
2387
+ "r10-comp-07641",
2388
+ "r10-comp-07654",
2389
+ "r10-comp-07657",
2390
+ "r10-comp-07658",
2391
+ "r10-comp-07660",
2392
+ "r10-comp-07669",
2393
+ "r10-comp-07674",
2394
+ "r10-comp-07687",
2395
+ "r10-comp-07750",
2396
+ "r10-comp-07755",
2397
+ "r10-comp-07767",
2398
+ "r10-comp-07785",
2399
+ "r10-comp-07786",
2400
+ "r10-comp-07789",
2401
+ "r10-comp-07801",
2402
+ "r10-comp-07819",
2403
+ "r10-comp-07821",
2404
+ "r10-comp-07824",
2405
+ "r10-comp-07827",
2406
+ "r10-comp-07869",
2407
+ "r10-comp-07884",
2408
+ "r10-comp-07886",
2409
+ "r10-comp-07887",
2410
+ "r10-comp-07889",
2411
+ "r10-comp-07895",
2412
+ "r10-comp-07914",
2413
+ "r10-comp-07933",
2414
+ "r10-comp-07944",
2415
+ "r10-comp-07972",
2416
+ "r10-comp-07979",
2417
+ "r10-comp-08007",
2418
+ "r10-comp-08019",
2419
+ "r10-comp-08026",
2420
+ "r10-comp-08042",
2421
+ "r10-comp-08048",
2422
+ "r10-comp-08049",
2423
+ "r10-comp-08050",
2424
+ "r10-comp-08057",
2425
+ "r10-comp-08068",
2426
+ "r10-comp-08069",
2427
+ "r10-comp-08070",
2428
+ "r10-comp-08082",
2429
+ "r10-comp-08087",
2430
+ "r10-comp-08088",
2431
+ "r10-comp-08117",
2432
+ "r10-comp-08124",
2433
+ "r10-comp-08126",
2434
+ "r10-comp-08129",
2435
+ "r10-comp-08132",
2436
+ "r10-comp-08148",
2437
+ "r10-comp-08163",
2438
+ "r10-comp-08171",
2439
+ "r10-comp-08189",
2440
+ "r10-comp-08190",
2441
+ "r10-comp-08196",
2442
+ "r10-comp-08202",
2443
+ "r10-comp-08212",
2444
+ "r10-comp-08248",
2445
+ "r10-comp-08253",
2446
+ "r10-comp-08254",
2447
+ "r10-comp-08257",
2448
+ "r10-comp-08263",
2449
+ "r10-comp-08274",
2450
+ "r10-comp-08293",
2451
+ "r10-comp-08294",
2452
+ "r10-comp-08298",
2453
+ "r10-comp-08301",
2454
+ "r10-comp-08302",
2455
+ "r10-comp-08333",
2456
+ "r10-comp-08337",
2457
+ "r10-comp-08356",
2458
+ "r10-comp-08367",
2459
+ "r10-comp-08378",
2460
+ "r10-comp-08379",
2461
+ "r10-comp-08383",
2462
+ "r10-comp-08400",
2463
+ "r10-comp-08408",
2464
+ "r10-comp-08419",
2465
+ "r10-comp-08433",
2466
+ "r10-comp-08470",
2467
+ "r10-comp-08477",
2468
+ "r10-comp-08490",
2469
+ "r10-comp-08493",
2470
+ "r10-comp-08507",
2471
+ "r10-comp-08515",
2472
+ "r10-comp-08530",
2473
+ "r10-comp-08538",
2474
+ "r10-comp-08539",
2475
+ "r10-comp-08556",
2476
+ "r10-comp-08569",
2477
+ "r10-comp-08590",
2478
+ "r10-comp-08598",
2479
+ "r10-comp-08634",
2480
+ "r10-comp-08657",
2481
+ "r10-comp-08674",
2482
+ "r10-comp-08684",
2483
+ "r10-comp-08686",
2484
+ "r10-comp-08690",
2485
+ "r10-comp-08695",
2486
+ "r10-comp-08702",
2487
+ "r10-comp-08706",
2488
+ "r10-comp-08721",
2489
+ "r10-comp-08742",
2490
+ "r10-comp-08776",
2491
+ "r10-comp-08795",
2492
+ "r10-comp-08803",
2493
+ "r10-comp-08806",
2494
+ "r10-comp-08816",
2495
+ "r10-comp-08817",
2496
+ "r10-comp-08824",
2497
+ "r10-comp-08835",
2498
+ "r10-comp-08841",
2499
+ "r10-comp-08843",
2500
+ "r10-comp-08855",
2501
+ "r10-comp-08870",
2502
+ "r10-comp-08893",
2503
+ "r10-comp-08909",
2504
+ "r10-comp-08926",
2505
+ "r10-comp-08932",
2506
+ "r10-comp-08941",
2507
+ "r10-comp-08946",
2508
+ "r10-comp-08956",
2509
+ "r10-comp-08972",
2510
+ "r10-comp-08985",
2511
+ "r10-comp-09006",
2512
+ "r10-comp-09031",
2513
+ "r10-comp-09041",
2514
+ "r10-comp-09044",
2515
+ "r10-comp-09051",
2516
+ "r10-comp-09063",
2517
+ "r10-comp-09066",
2518
+ "r10-comp-09085",
2519
+ "r10-comp-09086",
2520
+ "r10-comp-09087",
2521
+ "r10-comp-09092",
2522
+ "r10-comp-09100",
2523
+ "r10-comp-09137",
2524
+ "r10-comp-09187",
2525
+ "r10-comp-09189",
2526
+ "r10-comp-09192",
2527
+ "r10-comp-09196",
2528
+ "r10-comp-09199",
2529
+ "r10-comp-09207",
2530
+ "r10-comp-09210",
2531
+ "r10-comp-09222",
2532
+ "r10-comp-09229",
2533
+ "r10-comp-09240",
2534
+ "r10-comp-09242",
2535
+ "r10-comp-09249",
2536
+ "r10-comp-09255",
2537
+ "r10-comp-09258",
2538
+ "r10-comp-09272",
2539
+ "r10-comp-09273",
2540
+ "r10-comp-09291",
2541
+ "r10-comp-09292",
2542
+ "r10-comp-09326",
2543
+ "r10-comp-09329"
2544
+ ]
2545
+ }
2546
+ ],
2547
+ "m": 30.0,
2548
+ "n_folds": 2,
2549
+ "n_rows": 1411,
2550
+ "n_scored": 1411,
2551
+ "oof_balanced_error": 0.041957059005251776,
2552
+ "raw_thresholds": {
2553
+ "audiobook": 0.399,
2554
+ "comic": 0.399,
2555
+ "course": 0.399,
2556
+ "ebook": 0.399,
2557
+ "game": 0.399,
2558
+ "movie": 0.306,
2559
+ "music": 0.399,
2560
+ "software": 0.399,
2561
+ "tv_show": 0.399,
2562
+ "unknown": 0.399,
2563
+ "xxx": 0.399
2564
+ },
2565
+ "schema": "armiger-r12-owner-scope-v1",
2566
+ "split": "val",
2567
+ "tau_global": 0.399,
2568
+ "thresholds": {
2569
+ "audiobook": 0.399,
2570
+ "comic": 0.39899999999999997,
2571
+ "course": 0.399,
2572
+ "ebook": 0.399,
2573
+ "game": 0.399,
2574
+ "movie": 0.31285503685503685,
2575
+ "music": 0.399,
2576
+ "software": 0.399,
2577
+ "tv_show": 0.399,
2578
+ "unknown": 0.399,
2579
+ "xxx": 0.399
2580
+ },
2581
+ "val_sha256": "36b66664e7f6ee49e1aae13fc7338294d845b2c7842e682e71a3485b050d7777"
2582
+ },
2583
+ "r13_allow_categories": [
2584
+ "movie",
2585
+ "tv_show"
2586
+ ],
2587
+ "stratum_weighted_refit": {
2588
+ "spec_targets": {
2589
+ "allowed_coverage_at_least_98pct_test": false,
2590
+ "allowed_coverage_at_least_98pct_val": true,
2591
+ "course_leak_below_10pct": false
2592
+ },
2593
+ "val_audit": {
2594
+ "ht_weighted_global": {
2595
+ "allowed_coverage": 0.98,
2596
+ "course_leak_rate": 0.4524,
2597
+ "course_leaked": 19,
2598
+ "excluded_recall": 0.9471
2599
+ },
2600
+ "legacy_global": {
2601
+ "allowed_coverage": 0.9811,
2602
+ "course_leak_rate": 0.4524,
2603
+ "course_leaked": 19,
2604
+ "excluded_recall": 0.9471
2605
+ },
2606
+ "mondrian_by_gold_category_oracle": {
2607
+ "allowed_coverage": 0.9845,
2608
+ "course_leak_rate": 0.4524,
2609
+ "course_leaked": 19,
2610
+ "excluded_recall": 0.9471
2611
+ },
2612
+ "mondrian_by_predicted_category": {
2613
+ "allowed_coverage": 0.98,
2614
+ "course_leak_rate": 0.4524,
2615
+ "course_leaked": 19,
2616
+ "excluded_recall": 0.949
2617
+ },
2618
+ "n": 1411,
2619
+ "n_allowed": 901,
2620
+ "n_course_excluded": 42
2621
+ },
2622
+ "weights": {
2623
+ "cells": {
2624
+ "allowed_type|game": 153,
2625
+ "allowed_type|movie": 311,
2626
+ "allowed_type|software": 123,
2627
+ "allowed_type|tv_show": 314,
2628
+ "excluded|audiobook": 61,
2629
+ "excluded|comic": 163,
2630
+ "excluded|course": 42,
2631
+ "excluded|ebook": 60,
2632
+ "excluded|music": 184
2633
+ },
2634
+ "class": "owner_scope",
2635
+ "method": "horvitz_thompson_inverse_stratum w_i = 1/n_{c,k}",
2636
+ "stratum": "category"
2637
  }
2638
  },
 
2639
  "temperature_audit": {
2640
+ "anime": {
2641
+ "acc_pct": 81.8035,
2642
+ "best_T_crossfit": 0.75,
2643
+ "ece5_at_T1_pct": 3.3623,
2644
+ "ece5_guarded_pct": 1.0578,
2645
+ "fold_Ts": [
2646
+ 0.825,
2647
+ 0.675
2648
+ ],
2649
+ "n": 621,
2650
+ "shipped_T_guarded": 0.75,
2651
+ "weighted_nll_at_T1": 0.345181,
2652
+ "weighted_nll_at_best_T": 0.335153,
2653
+ "weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
2654
+ },
2655
  "category": {
2656
+ "acc_pct": 84.55,
2657
+ "best_T_crossfit": 1.2,
2658
+ "ece5_at_T1_pct": 3.2368,
2659
+ "ece5_guarded_pct": 6.287,
2660
+ "fold_Ts": [
2661
+ 1.25,
2662
+ 1.15
2663
+ ],
2664
+ "n": 1411,
2665
+ "shipped_T_guarded": 1.2,
2666
+ "weighted_nll_at_T1": 0.597522,
2667
+ "weighted_nll_at_best_T": 0.58716,
2668
+ "weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
2669
+ },
2670
+ "english_subs": {
2671
+ "acc_pct": 97.1698,
2672
+ "best_T_crossfit": 0.55,
2673
+ "ece5_at_T1_pct": 4.3801,
2674
+ "ece5_guarded_pct": 1.8806,
2675
+ "fold_Ts": [
2676
+ 0.6,
2677
+ 0.5
2678
+ ],
2679
+ "n": 212,
2680
+ "shipped_T_guarded": 0.55,
2681
+ "weighted_nll_at_T1": 0.123028,
2682
+ "weighted_nll_at_best_T": 0.087029,
2683
+ "weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
2684
  },
2685
  "lang": {
2686
+ "acc_pct": 82.7434,
2687
+ "best_T_crossfit": 1.5375,
2688
+ "ece5_at_T1_pct": 8.9621,
2689
+ "ece5_guarded_pct": 2.3818,
2690
+ "fold_Ts": [
2691
+ 1.675,
2692
+ 1.4
2693
+ ],
2694
+ "n": 226,
2695
+ "shipped_T_guarded": 1.5375,
2696
+ "weighted_nll_at_T1": 0.358405,
2697
+ "weighted_nll_at_best_T": 0.324181,
2698
+ "weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
2699
  },
2700
  "owner_scope": {
2701
+ "acc_pct": 98.4408,
2702
+ "best_T_crossfit": 1.8875,
2703
+ "ece5_at_T1_pct": 0.9298,
2704
+ "ece5_guarded_pct": 2.6221,
2705
+ "fold_Ts": [
2706
+ 2.05,
2707
+ 1.725
2708
+ ],
2709
+ "n": 1411,
2710
+ "shipped_T_guarded": 1.8875,
2711
+ "weighted_nll_at_T1": 0.200121,
2712
+ "weighted_nll_at_best_T": 0.147816,
2713
+ "weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
2714
  },
2715
  "title_language": {
2716
+ "acc_pct": 97.832,
2717
+ "best_T_crossfit": 1.0125,
2718
+ "ece5_at_T1_pct": 0.7029,
2719
+ "ece5_guarded_pct": 0.7498,
2720
+ "fold_Ts": [
2721
+ 1.175,
2722
+ 0.85
2723
+ ],
2724
+ "n": 369,
2725
+ "shipped_T_guarded": 1.0125,
2726
+ "weighted_nll_at_T1": 0.070129,
2727
+ "weighted_nll_at_best_T": 0.070032,
2728
+ "weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
2729
+ }
2730
+ },
2731
+ "temperature_scaling_stop_fit": {
2732
+ "anime": {
2733
+ "cal_temp_scaled": {
2734
+ "fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
2735
+ "n": 621
2736
+ },
2737
+ "temperature_T": 0.75
2738
  },
2739
+ "category": {
2740
+ "cal_temp_scaled": {
2741
+ "fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
2742
+ "n": 1411
2743
+ },
2744
+ "temperature_T": 1.2
 
 
 
2745
  },
2746
+ "english_subs": {
2747
+ "cal_temp_scaled": {
2748
+ "fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
2749
+ "n": 212
2750
+ },
2751
+ "temperature_T": 0.55
 
 
 
2752
  },
2753
+ "lang": {
2754
+ "cal_temp_scaled": {
2755
+ "fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
2756
+ "n": 226
2757
+ },
2758
+ "temperature_T": 1.5375
2759
  },
 
 
 
 
 
 
 
 
 
 
 
2760
  "owner_scope": {
2761
+ "cal_temp_scaled": {
2762
+ "fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
2763
+ "n": 1411
2764
+ },
2765
+ "temperature_T": 1.8875
2766
+ },
2767
+ "title_language": {
2768
+ "cal_temp_scaled": {
2769
+ "fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
2770
+ "n": 369
2771
+ },
2772
+ "temperature_T": 1.0125
2773
  }
2774
+ }
 
 
 
 
 
 
 
 
 
 
2775
  }
config.json CHANGED
@@ -1,28 +1,27 @@
1
  {
2
- "model_name": "armiger-ultra",
3
- "version": "3.0.0",
4
- "bundle_type": "unified_armiger_plus_ultra_plus_swarm",
5
- "architecture": "armiger-v3-byte-rope-transformer-9head (incl. byte-level span_bio + 7x7 BIO Viterbi) + residual-swarm-health-adapter",
 
 
 
6
  "artifacts": {
7
- "armiger_cpu_int8_onnx": {
8
- "file": "armiger.int8.onnx",
9
- "size_bytes": 15721824,
10
- "sha256": "a0dab886be6252739d4c89ce01d65d8d2135204beaec6b00159d038cf420f329"
11
  },
12
- "armiger_ultra_fp32_onnx": {
13
- "file": "armiger_ultra.fp32.onnx",
14
- "size_bytes": 32543999,
15
- "sha256": "6459a20d7bf45d46d59efbcb50ffb8a519bacda14ae36403dcc510eceaa38f44"
16
  },
17
- "armiger_ultra_pytorch": {
18
- "file": "armiger_ultra.pt",
19
- "size_bytes": 32493659,
20
- "sha256": "44cdcf1fac8ebc036262b2d3150b3ba0f27002669bf81e3480b1bc65e3ef8156"
21
  },
22
- "swarm_health_adapter": {
23
- "file": "swarm_adapter.pt",
24
- "size_bytes": 131718,
25
- "sha256": "85effcb9e70af7bff6047d4ae45d2155444e9b691dc84431f19ad14f9daee9ea"
26
  }
27
  },
28
  "heads": {
@@ -68,15 +67,6 @@
68
  "title_language": [
69
  "english_title",
70
  "non_english_title"
71
- ],
72
- "span_bio": [
73
- "O",
74
- "B-TITLE",
75
- "I-TITLE",
76
- "B-YEAR",
77
- "I-YEAR",
78
- "B-EP",
79
- "I-EP"
80
  ]
81
  },
82
  "mondrian_thresholds": {
@@ -84,118 +74,91 @@
84
  "junk": 0.101,
85
  "owner_scope": 0.477
86
  },
87
- "benchmarks": {
88
- "gold_standard_11class_category_acc": 94.85,
89
- "movie_vs_tv_acc": 95.83,
90
- "catalog_matching_rate_acc": 95.06,
91
- "audio_language_acc": 96.7,
92
- "anime_and_dub_acc": 99.61,
93
- "adult_safety_gate_acc": 98.49,
94
- "swarm_liveliness_auroc": 96.83,
95
- "fake_seeder_honeypot_recall": 100.0,
96
- "overall_6dim_composite_mean": 95.66,
97
- "gpu_batched_latency_ms": 0.5,
98
- "gpu_single_item_latency_ms": 1.12,
99
- "cpu_int8_batched_latency_ms": 0.42,
100
- "junk_malware_filter_beyond_n977_balanced_acc": 99.23,
101
- "oct15_junk_status": "n_pos = 1, not estimable",
102
- "public_tracker_swarm_auroc": 92.42,
103
- "cpu_int8_single_item_latency_ms": 2.512,
104
- "cpu_fp32_single_item_latency_ms": 2.637,
105
- "cpu_int8_batch64_items_per_sec": 396.9,
106
- "cpu_fp32_batch64_items_per_sec": 376.0
107
- },
108
- "bundled_plugins": {
109
- "armiger-swarm-survival": {
110
- "standalone_repo": "NorviTech/armiger-swarm-survival",
111
- "files": [
112
- "swarm_adapter.pt",
113
- "swarm_survival.py"
114
- ],
115
- "sha256": {
116
- "swarm_adapter.pt": "85effcb9e70af7bff6047d4ae45d2155444e9b691dc84431f19ad14f9daee9ea",
117
- "swarm_survival.py": "9dfbe7c4961f5a53bb4e40fc4450ebf958398157148d9f54da100d75dd24caf0"
118
- }
119
  },
120
- "armiger-swarm-priors": {
121
- "standalone_repo": "NorviTech/armiger-swarm-priors",
122
- "files": [
123
- "swarm_priors.py"
124
- ],
125
- "sha256": {
126
- "swarm_priors.py": "49b2b6857802606158e4a1de8cdc335e485be83aaec5405b9f2fd103ca90b394"
127
- }
128
  },
129
- "armiger-zero-cue-catalog": {
130
- "standalone_repo": "NorviTech/armiger-zero-cue-catalog",
131
- "files": [
132
- "zero_cue_plugin.py",
133
- "validated_zero_cue_catalog.json"
134
- ],
135
- "sha256": {
136
- "zero_cue_plugin.py": "90039f19d36f2bcd0f0103deefe77705caf46e60c9f170d23d4719bc4a2ca020",
137
- "validated_zero_cue_catalog.json": "8a8b7a4ba97cb22c3aa1f75ab33762052deb09580df464b1eabfe622a2837d26"
138
- }
139
  },
140
- "armiger-audio-guard": {
141
- "standalone_repo": "NorviTech/armiger-audio-guard",
142
- "files": [
143
- "audio_language_guard.py"
144
- ],
145
- "sha256": {
146
- "audio_language_guard.py": "f6f2820d4230faa4f07b9662e1e597c5b893f6855478fceb09b78d2d282ecde2"
147
- }
148
  },
149
- "armiger-calibration-dht": {
150
- "standalone_repo": "NorviTech/armiger-calibration-dht",
151
- "files": [
152
- "calibration.json",
153
- "calibration_and_dht.py",
154
- "armiger_dht_rsi.py",
155
- "test_armiger_dht_rsi.py"
156
- ],
157
- "sha256": {
158
- "calibration.json": "e5b58fc6a8380b982f09296822b0244b28aa5b9b38c32d690c9e25160a8860bb",
159
- "calibration_and_dht.py": "b5feaf9e9c5a8304c31675e6c0d4ab14a6ec3fd40bd24bafac109c5b139cbd23",
160
- "armiger_dht_rsi.py": "a765952f40c384777af82bb7a7ce4d695e7f9cf12b9487aabb08d7bc3f60c707",
161
- "test_armiger_dht_rsi.py": "af765f049fe684aedd5b20cf58e8a0cb8c4cc40fd501e50a78844bcc256ad1c4"
162
- }
163
  },
164
- "armiger-dht-db-sync": {
165
- "standalone_repo": "NorviTech/armiger-dht-db-sync",
166
- "files": [
167
- "csam_guard.py",
168
- "armiger_dht_db_sync.py",
169
- "test_dht_db_sync.py"
170
- ],
171
- "sha256": {
172
- "csam_guard.py": "fa1fc613034476247587ce8e43044433df38a8092df1210254d7715594ee23b8",
173
- "armiger_dht_db_sync.py": "a942043f4aa5887a20b9a1cb5e6632ca248c8dc8d2c905bfaa667f489d751f2d",
174
- "test_dht_db_sync.py": "3e3bd272a6adb1e217330c2732a8528cecb46a2d34beaa6b4d7d42518ea22a5d"
175
- }
176
- }
177
  },
178
- "sdk": {
179
- "entrypoint": "armiger_ultra_pipeline.py",
180
- "class_name": "ArmigerUltraPipeline",
181
- "test_suite": "test_armiger_ultra_pipeline.py",
182
- "unified_heads": [
183
- "category",
184
- "adult",
185
- "junk",
186
- "anime",
187
- "english_subs",
188
- "lang",
189
- "owner_scope",
190
- "title_language",
191
- "swarm_viability",
192
- "honeypot_trap",
193
- "csam_safety"
194
- ],
195
- "sha256": {
196
- "armiger_ultra_pipeline.py": "5a759dec66b9e13e532240d745afbb38826af90c5a9021c0eb845e6182cd1717",
197
- "test_armiger_ultra_pipeline.py": "d4aeaf2143ac192a839e5fc0e14b1b5e03c84434274526dd1f38490d4de59f47"
198
- }
199
  },
200
- "parameter_count": 8050139
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
201
  }
 
1
  {
2
+ "model_name": "armiger",
3
+ "version": "3.3.0",
4
+ "architecture": "armiger-v2-byte-rope-transformer-8head",
5
+ "checkpoint": "H350000_Curated_Wave2-seed0 (direct-side hardened junk head)",
6
+ "format": "onnx-int8-dynamic",
7
+ "size_bytes": 8283542,
8
+ "sha256": "0f7496a70565145d09a8b29b09de9b99171cf483e5c32ff32e89df1931231401",
9
  "artifacts": {
10
+ "armiger.onnx": {
11
+ "size_bytes": 17599636,
12
+ "sha256": "c0e96936038988c7465b7fe041c4dc3c92df5a8cb4f56c2666271cd4d3e79420"
 
13
  },
14
+ "calibration.json": {
15
+ "size_bytes": 76612,
16
+ "sha256": "300387779dc16292dd9e6be038ce9dd60e5bdb0ee929b383dcd4e51715f8b5f8"
 
17
  },
18
+ "vocab.json": {
19
+ "size_bytes": 39996,
20
+ "sha256": "c10afea430188e0207752e93bf0b340fbb585858684037c5134a477cbc0dd7bb"
 
21
  },
22
+ "armiger_inference.py": {
23
+ "size_bytes": 38228,
24
+ "sha256": "c345587c57cecc518626852d0d8fcacf5ebb1576c10c0df6608f851a9ec72c31"
 
25
  }
26
  },
27
  "heads": {
 
67
  "title_language": [
68
  "english_title",
69
  "non_english_title"
 
 
 
 
 
 
 
 
 
70
  ]
71
  },
72
  "mondrian_thresholds": {
 
74
  "junk": 0.101,
75
  "owner_scope": 0.477
76
  },
77
+ "benchmarks_unmutated_ground_truth": {
78
+ "oct15_confirmation_n2835": {
79
+ "standalone_neural_11class_acc": 95.06,
80
+ "fair_hybrid_with_zero_cue_plugin_11class_acc": 95.49,
81
+ "standalone_neural_movie_vs_tv_acc": 96.6,
82
+ "fair_hybrid_movie_vs_tv_acc": 97.15,
83
+ "standalone_neural_catalog_matching_acc": 95.85,
84
+ "fair_hybrid_catalog_matching_acc": 96.39,
85
+ "standalone_neural_audio_lang_binary_n545_acc": 96.88,
86
+ "with_audio_guard_binary_n545_acc": 97.8,
87
+ "audio_lang_3class_n772_acc": 92.75,
88
+ "anime_n507_acc": 99.61,
89
+ "adult_n2781_acc": 98.53,
90
+ "junk_n2170_acc": 100.0,
91
+ "owner_scope_n2669_acc": 98.65,
92
+ "title_language_n969_acc": 93.7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
93
  },
94
+ "fresh_confirmation_n1402": {
95
+ "standalone_neural_11class_acc": 94.44,
96
+ "fair_hybrid_11class_acc": 94.58,
97
+ "adult_n1393_acc": 98.21,
98
+ "junk_n1291_acc": 100.0,
99
+ "anime_n253_acc": 99.6
 
 
100
  },
101
+ "beyond_confirmation_n1025": {
102
+ "standalone_neural_11class_acc": 92.88,
103
+ "fair_hybrid_11class_acc": 95.12,
104
+ "adult_n1016_acc": 98.52,
105
+ "junk_n977_acc": 99.9,
106
+ "anime_n636_acc": 98.9,
107
+ "title_language_n604_acc": 98.51
 
 
 
108
  },
109
+ "zero_cue_2000_n2000": {
110
+ "standalone_neural_11class_acc": 87.35,
111
+ "fair_hybrid_11class_acc": 91.5,
112
+ "adult_acc": 99.9,
113
+ "junk_acc": 100.0,
114
+ "anime_n1998_acc": 95.8,
115
+ "lang_acc": 88.15,
116
+ "title_language_acc": 97.75
117
  },
118
+ "multilang_3000_n3000": {
119
+ "standalone_neural_11class_acc": 97.37,
120
+ "fair_hybrid_11class_acc": 98.4,
121
+ "adult_acc": 100.0,
122
+ "junk_acc": 100.0,
123
+ "anime_n2625_acc": 96.38,
124
+ "lang_acc": 91.47
 
 
 
 
 
 
 
125
  },
126
+ "five_split_summary_n10262": {
127
+ "weighted_mean_standalone_neural_acc": 93.95,
128
+ "weighted_mean_fair_hybrid_acc": 95.39,
129
+ "macro_mean_standalone_neural_acc": 93.42,
130
+ "macro_mean_fair_hybrid_acc": 95.0
131
+ },
132
+ "swarm_liveliness": {
133
+ "way4_priors_formula_auroc": 86.64,
134
+ "with_swarm_adapter_public_tracker_n757_auroc": 92.42,
135
+ "with_swarm_adapter_public_holdout_n493_auroc": 93.03,
136
+ "with_swarm_adapter_multi_source_corpus_n10278_auroc": 96.83
137
+ },
138
+ "overall_6dim_composite_mean": 96.5
139
  },
140
+ "hardware_latency_benchmarks": {
141
+ "apple_m4_pro_cpu_onnx_int8_single_item_p50_ms": 2.44,
142
+ "apple_m4_pro_cpu_onnx_int8_batch64_per_item_ms": 3.07,
143
+ "apple_m4_pro_cpu_onnx_int8_batch64_throughput_items_sec": 326.0,
144
+ "apple_m4_pro_mps_gpu_fp32_batch64_per_item_ms": 0.29,
145
+ "nvidia_rtx_gpu_fp32_batch64_per_item_ms": 0.21
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
146
  },
147
+ "parameter_count": 8050139,
148
+ "benchmarks": {
149
+ "heldout_10class_category_acc": 90.94,
150
+ "beyond_confirmation_category_acc": 90.44,
151
+ "catalog_matching_rate_acc": 90.34,
152
+ "audio_language_acc": 98.25,
153
+ "anime_and_dub_acc": 97.48,
154
+ "adult_safety_gate_acc": 100.0,
155
+ "junk_malware_filter_acc": 99.9,
156
+ "swarm_liveliness_public_auroc": 92.42,
157
+ "swarm_liveliness_multisource_auroc": 96.83,
158
+ "fake_seeder_honeypot_recall": 100.0,
159
+ "overall_6dim_composite_mean": 94.09,
160
+ "gpu_batched_latency_ms": 0.21,
161
+ "gpu_single_item_latency_ms": 1.12,
162
+ "cpu_int8_batched_latency_ms": 0.42
163
+ }
164
  }
r13_feature_extractor.py ADDED
@@ -0,0 +1,406 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """R13 S2 features and a read-only adapter for frozen R12 ONNX checkpoints.
2
+
3
+ There are two feature views. Category keeps the R12 enriched input; title
4
+ language sees the literal title, without file names or file statistics. S8
5
+ should train those views together, rather than transplanting a language head.
6
+
7
+ Only two semantic rules are admitted: identifier neutrality for the binary
8
+ title-language task, and a bracketed release/episode/technical-tags/video cue
9
+ for anime. ``audit_train_rules`` supplies their train-only precision gates.
10
+ Dual-audio and ASS are observable features, never standalone anime rules.
11
+ """
12
+ from __future__ import annotations
13
+
14
+ from collections import Counter
15
+ from dataclasses import dataclass
16
+ from functools import lru_cache
17
+ import hashlib
18
+ import importlib.util
19
+ import math
20
+ from pathlib import Path
21
+ import re
22
+ from typing import Any, Mapping, Sequence
23
+
24
+ import numpy as np
25
+
26
+ PROJECT_ROOT = Path(__file__).resolve().parent
27
+ SIDE_DIM = 28
28
+ BASE_STATS_DIM = 12
29
+ STATS_DIM = 40
30
+ MIN_SUPPORT = 5
31
+ MIN_PRECISION = 0.90
32
+
33
+ # These are syntax families from the session brief, not title/group catalogs.
34
+ NUMERIC_ID_RE = re.compile(r"[0-9]+(?:-[0-9]+)?", re.ASCII)
35
+ PG_ID_RE = re.compile(r"pg[0-9]+(?:-[0-9]+)?", re.I | re.ASCII)
36
+ GUT_ID_RE = re.compile(r"[a-z0-9_.-]+gut", re.I | re.ASCII)
37
+ PACKAGE_ID_RE = re.compile(r"com(?:\.[a-z][a-z0-9_]*){2,}", re.I | re.ASCII)
38
+ FANSUB_EPISODE_RE = re.compile(
39
+ r"^\[(?P<group>[^\[\]\r\n]{2,80})\]\s*(?P<title>.+?)\s+-\s+"
40
+ r"(?P<episode>[0-9]{1,3})(?:v[0-9]+)?\s*"
41
+ r"(?P<tags>(?:(?:\[[^\[\]\r\n]+\]|\([^()\r\n]+\))\s*)+)"
42
+ r"\.(?:mkv|mp4|avi|webm)$", re.I,
43
+ )
44
+ RESOLUTION_RE = re.compile(r"\b(?:480|576|720|1080|1440|2160|4320)[pi]\b", re.I)
45
+ JAPANESE_RE = re.compile(r"(?<![a-z])(?:jpn|japanese)(?![a-z])", re.I)
46
+ ENGLISH_RE = re.compile(r"(?<![a-z])(?:eng|english)(?![a-z])", re.I)
47
+ DUAL_AUDIO_RE = re.compile(r"\bdual[ ._-]*audio\b", re.I)
48
+ VIDEO_EXTS = frozenset({"mkv", "mp4", "avi", "webm"})
49
+ RULE_IDENTIFIER = "identifier_title_neutrality"
50
+ RULE_FANSUB = "bracket_episode_resolution_video"
51
+
52
+
53
+ @lru_cache(maxsize=1)
54
+ def r12_features():
55
+ """Load the named local baseline, without importing a trainer or dataset."""
56
+ path = PROJECT_ROOT / "_r12_features_base.py"
57
+ spec = importlib.util.spec_from_file_location("_s2_r12_baseline", path)
58
+ if spec is None or spec.loader is None:
59
+ raise ImportError(f"Cannot load baseline featurizer: {path}")
60
+ module = importlib.util.module_from_spec(spec)
61
+ spec.loader.exec_module(module)
62
+ if (module.SIDE_DIM, module.STATS_DIM) != (SIDE_DIM, STATS_DIM):
63
+ raise ValueError("R12 feature contract changed; requalify S2")
64
+ return module
65
+
66
+
67
+ def normalize_files(files: Any) -> list[dict[str, Any]]:
68
+ """Support SDK mappings, path/size pairs, and plain relative paths."""
69
+ result = []
70
+ for entry in files or []:
71
+ if isinstance(entry, Mapping):
72
+ path = entry.get("path") or entry.get("p") or entry.get("name") or ""
73
+ size = entry.get("size") or entry.get("s") or 0
74
+ elif isinstance(entry, (tuple, list)):
75
+ path = entry[0] if entry else ""
76
+ size = entry[1] if len(entry) > 1 else 0
77
+ else:
78
+ path, size = entry, 0
79
+ try:
80
+ size = max(0, int(size))
81
+ except (TypeError, ValueError, OverflowError):
82
+ size = 0
83
+ path = str(path or "").replace("\\", "/")
84
+ if path and "/.____padding_file/" not in f"/{path}":
85
+ result.append({"path": path, "size": size})
86
+ return result
87
+
88
+
89
+ def _safe_nonneg_int(val: Any) -> int:
90
+ try:
91
+ return max(0, int(val or 0))
92
+ except (TypeError, ValueError, OverflowError):
93
+ return 0
94
+
95
+
96
+ def inference_input(row: Mapping[str, Any]) -> dict[str, Any]:
97
+ """Use R12's production firewall: name/files/size/infohash, no language hint.
98
+
99
+ Matches active_source/armiger_bench.py::sanitize_inference_input for snapshot
100
+ mappings, while accepting SDK path/size aliases for clean external inputs.
101
+ """
102
+ source = row.get("inference_input", row)
103
+ if not isinstance(source, Mapping):
104
+ raise TypeError("inference_input must be a mapping")
105
+ return {"name": str(source.get("name") or source.get("title") or ""),
106
+ "files": normalize_files(source.get("files")),
107
+ "size": _safe_nonneg_int(source.get("size")),
108
+ "infohash": str(source.get("infohash") or "").lower()}
109
+
110
+
111
+ def _utf8_prefix(text: str, max_bytes: int) -> str:
112
+ if max_bytes < 1:
113
+ raise ValueError("max_bytes must be positive")
114
+ return str(text).encode("utf-8", "replace")[:max_bytes].decode("utf-8", "ignore")
115
+
116
+
117
+ def build_title_language_text(name: str, files: Any = None, max_bytes: int = 256) -> str:
118
+ """Title-only view. Files deliberately have no effect on this representation."""
119
+ return _utf8_prefix(name, max_bytes)
120
+
121
+
122
+ def build_enriched_name_text(
123
+ name: str, files: Any, max_bytes: int = 256, *, head: str = "category",
124
+ ) -> str:
125
+ """Keep R12 enrichment for category; expose a separate title-language view."""
126
+ if head == "title_language":
127
+ return build_title_language_text(name, max_bytes=max_bytes)
128
+ text = r12_features().build_enriched_name_text(name, normalize_files(files), max_bytes=max_bytes)
129
+ return _utf8_prefix(text, max_bytes)
130
+
131
+
132
+ def _id_stem(name: str) -> str:
133
+ stem = name.strip()
134
+ # Exact opaque file identifiers are also supported, but not arbitrary paths.
135
+ if stem.lower().endswith((".txt", ".zip")):
136
+ stem = stem.rsplit(".", 1)[0]
137
+ return stem
138
+
139
+
140
+ def identifier_family(name: str, files: Any = None) -> str | None:
141
+ """Return a nonlinguistic-ID family, never infer a work or spoken language."""
142
+ stem = _id_stem(name)
143
+ for family, pattern in (("numeric", NUMERIC_ID_RE), ("pg_numeric", PG_ID_RE),
144
+ ("gut_stem", GUT_ID_RE), ("ascii_package", PACKAGE_ID_RE)):
145
+ if pattern.fullmatch(stem):
146
+ return family
147
+ # TXT+ZIP alone is not enough: require an opaque Gutenberg-shaped stem
148
+ # and exactly matching relative path, rather than unrelated archive files.
149
+ by_ext: dict[str, set[str]] = {"txt": set(), "zip": set()}
150
+ for item in normalize_files(files):
151
+ path = item["path"].lower()
152
+ base, dot, ext = path.rpartition(".")
153
+ if dot and ext in by_ext:
154
+ leaf = base.rsplit("/", 1)[-1]
155
+ if NUMERIC_ID_RE.fullmatch(leaf) or PG_ID_RE.fullmatch(leaf):
156
+ by_ext[ext].add(base)
157
+ # A readable supplied title must retain its own language, even with book files.
158
+ opaque_name = not re.search(r"[a-z]{3,}", name, re.I | re.ASCII)
159
+ if opaque_name and by_ext["txt"] & by_ext["zip"]:
160
+ return "gutenberg_txt_zip"
161
+ return None
162
+
163
+
164
+ @dataclass(frozen=True)
165
+ class StructuralCues:
166
+ fansub_episode_keys: frozenset[int]
167
+ fansub_video_count: int
168
+ has_mkv: bool
169
+ has_ass_subtitles: bool
170
+ explicit_jpn_eng_dual_audio: bool
171
+
172
+ @property
173
+ def fansub_episode(self) -> bool:
174
+ return bool(self.fansub_episode_keys)
175
+
176
+
177
+ def extract_structural_cues(name: str, files: Any = None) -> StructuralCues:
178
+ """Parse arbitrary group tags; there are no release-group or work literals."""
179
+ entries = normalize_files(files)
180
+ texts = [name, *(f["path"].rsplit("/", 1)[-1] for f in entries)]
181
+ episodes: set[int] = set()
182
+ matched: set[str] = set()
183
+ for text in texts:
184
+ match = FANSUB_EPISODE_RE.fullmatch(text)
185
+ if match and RESOLUTION_RE.search(match["tags"]):
186
+ episodes.add(int(match["episode"]))
187
+ matched.add(text)
188
+ exts = {t.rsplit(".", 1)[-1].lower() for t in texts if "." in t}
189
+ # These fields describe metadata only. They do not assert anime/sub language.
190
+ audio_text = " ".join(texts)
191
+ dual = bool(DUAL_AUDIO_RE.search(audio_text) and JAPANESE_RE.search(audio_text)
192
+ and ENGLISH_RE.search(audio_text) and "mkv" in exts)
193
+ return StructuralCues(frozenset(episodes), len(matched), "mkv" in exts, "ass" in exts, dual)
194
+
195
+
196
+ def compute_derivative_group_features(files: Any, raw_name: str = "") -> tuple[float, ...]:
197
+ """R12 U/P/share semantics plus explicit bracketed episode/version parsing.
198
+
199
+ Episode numbers in SxxExx and fansub paths refer to the same episodes: use
200
+ the larger count rather than summing two parsers' possibly duplicate keys.
201
+ S8 receives raw counts as well as the bounded side-vector representation.
202
+ """
203
+ normalized = normalize_files(files)
204
+ if not normalized and raw_name.rsplit(".", 1)[-1].lower() in VIDEO_EXTS:
205
+ normalized = [{"path": raw_name, "size": 0}]
206
+ # The derivative view drops resolution/version tags only after a full
207
+ # structural match. Keep parent+title+episode, so separate episodes/series
208
+ # cannot collapse merely because their codec, group, or episode number agrees.
209
+ derivative_view = []
210
+ for entry in normalized:
211
+ parent, sep, leaf = entry["path"].rpartition("/")
212
+ match = FANSUB_EPISODE_RE.fullmatch(leaf)
213
+ if match and RESOLUTION_RE.search(match["tags"]):
214
+ ext = leaf.rsplit(".", 1)[-1]
215
+ leaf = f"{match['title']}.E{int(match['episode']):03d}.{ext}"
216
+ derivative_view.append({"path": parent + sep + leaf, "size": entry["size"]})
217
+ else:
218
+ derivative_view.append(entry)
219
+ u, e, p, share = r12_features().compute_derivative_group_features(derivative_view, raw_name)
220
+ fan_count = len(extract_structural_cues(raw_name, normalized).fansub_episode_keys)
221
+ e = max(e, float(fan_count))
222
+ return u, e, (0.0 if e >= 2 else p), share
223
+
224
+
225
+ def encode_torrent_features(row: Mapping[str, Any], max_name_len: int = 256,
226
+ max_files: int = 64) -> dict[str, Any]:
227
+ """S8 training/inference encoder. R12 checkpoints use the adapter below.
228
+
229
+ New keys: title_name_bytes, title_name_text, title_identifier_family,
230
+ structural_cues. Existing stats remain 40 wide. side[13]/side[15] gain
231
+ subtitle presence and a validated generic fansub cue. side[24:28] preserve
232
+ their U/E/P/share meaning; no new checkpoint width is silently introduced.
233
+ """
234
+ clean = inference_input(row)
235
+ name = str(clean.get("name") or clean.get("title") or "")
236
+ encoded = r12_features().encode_torrent_features(clean, max_name_len, max_files)
237
+ cues = extract_structural_cues(name, clean["files"])
238
+ deriv = compute_derivative_group_features(clean["files"], name)
239
+ stats = list(encoded["stats"])
240
+ if cues.fansub_episode:
241
+ stats[BASE_STATS_DIM + 15] = 1.0
242
+ if cues.has_ass_subtitles:
243
+ stats[BASE_STATS_DIM + 13] = 1.0
244
+ u, e, p, share = deriv
245
+ stats[BASE_STATS_DIM + 24:] = [min(1.0, math.log1p(u) / math.log1p(64)),
246
+ min(1.0, math.log1p(e) / math.log1p(64)), p, share]
247
+ title = build_title_language_text(name, max_bytes=max_name_len)
248
+ encoded.update(stats=stats, deriv_features=deriv, structural_cues=cues,
249
+ title_name_text=title, title_name_bytes=list(title.encode("utf-8")) or [32],
250
+ title_identifier_family=identifier_family(name, clean["files"]))
251
+ return encoded
252
+
253
+
254
+ def build_title_language_batch_inputs(rows: Sequence[Mapping[str, Any]],
255
+ stats_dim: int = STATS_DIM) -> dict[str, np.ndarray]:
256
+ """A fully separate name view: no file bytes, sizes, metadata, or language hint.
257
+
258
+ Directly replacing bytes avoids R12's file fallback for dotted names. All
259
+ non-title tensors have the same constant value for every title.
260
+ """
261
+ names = [str((r := inference_input(row)).get("name") or r.get("title") or "") for row in rows]
262
+ data = [list(build_title_language_text(n).encode("utf-8")) or [32] for n in names]
263
+ batch = len(data)
264
+ if batch == 0:
265
+ raise ValueError("rows must not be empty")
266
+ name_bytes = np.zeros((batch, max(map(len, data))), dtype=np.int64)
267
+ for i, values in enumerate(data):
268
+ name_bytes[i, :len(values)] = values
269
+ return {"name_bytes": name_bytes, "name_len": np.asarray(list(map(len, data)), dtype=np.int64),
270
+ "ext": np.zeros((batch, 1), dtype=np.int32),
271
+ "size": np.zeros((batch, 1), dtype=np.int64),
272
+ "depth": np.zeros((batch, 1), dtype=np.int64),
273
+ "words": np.zeros((batch, 1, 4), dtype=np.int32),
274
+ "stats": np.zeros((batch, stats_dim), dtype=np.float32)}
275
+
276
+
277
+ def _label(row: Mapping[str, Any], head: str) -> str | None:
278
+ target = (row.get("targets") or {}).get(head) or {}
279
+ return target.get("value") if target.get("known_mask") else None
280
+
281
+
282
+ def audit_train_rules(rows: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
283
+ """Freeze support using train labels only; unknown masks never count as gold.
284
+
285
+ The identifier rule is one pre-specified union from the user's brief.
286
+ Subgroup counts are retained so absent supervision is explicit.
287
+ """
288
+ counts = {RULE_IDENTIFIER: Counter(), RULE_FANSUB: Counter()}
289
+ subgroups: dict[str, Counter] = {}
290
+ for row in rows:
291
+ if row.get("split") != "train":
292
+ raise ValueError("Rule admission accepts train-marked rows only")
293
+ clean = inference_input(row)
294
+ name = str(clean.get("name") or clean.get("title") or "")
295
+ family = identifier_family(name, clean["files"])
296
+ if family:
297
+ value = _label(row, "title_language") or "unknown"
298
+ counts[RULE_IDENTIFIER][value] += 1
299
+ subgroups.setdefault(family, Counter())[value] += 1
300
+ if extract_structural_cues(name, clean["files"]).fansub_episode:
301
+ counts[RULE_FANSUB][_label(row, "anime") or "unknown"] += 1
302
+ report: dict[str, Any] = {}
303
+ for rule, positive in ((RULE_IDENTIFIER, "english_title"), (RULE_FANSUB, "anime")):
304
+ c = counts[rule]
305
+ support = sum(v for k, v in c.items() if k != "unknown")
306
+ precision = c[positive] / support if support else 0.0
307
+ report[rule] = {"counts": dict(c), "known_support": support, "precision": precision,
308
+ "admitted": support >= MIN_SUPPORT and precision >= MIN_PRECISION}
309
+ report["identifier_subgroups"] = {k: dict(v) for k, v in subgroups.items()}
310
+ report["dual_audio_ass_standalone_rule_admitted"] = False
311
+ return report
312
+
313
+
314
+ def apply_head_guards(row: Mapping[str, Any], predictions: Mapping[str, str],
315
+ train_audit: Mapping[str, Any], *,
316
+ enable_identifier_guard: bool = False) -> tuple[dict[str, str], dict[str, str]]:
317
+ """Copy predictions, modify only title_language/anime, and expose provenance.
318
+
319
+ english_title is the existing binary task's compatibility value for an
320
+ identifier without a readable title. It is not an assertion about book text,
321
+ spoken language, subtitle language, country, or the title behind an ID.
322
+ """
323
+ clean = inference_input(row)
324
+ name = str(clean.get("name") or clean.get("title") or "")
325
+ result, reasons = dict(predictions), {}
326
+ family = identifier_family(name, clean["files"])
327
+ if (enable_identifier_guard and family
328
+ and train_audit.get(RULE_IDENTIFIER, {}).get("admitted")):
329
+ result["title_language"] = "english_title"
330
+ reasons["title_language"] = f"{RULE_IDENTIFIER}:{family}"
331
+ if (extract_structural_cues(name, clean["files"]).fansub_episode
332
+ and train_audit.get(RULE_FANSUB, {}).get("admitted")):
333
+ result["anime"] = "anime"
334
+ reasons["anime"] = RULE_FANSUB
335
+ return result, reasons
336
+
337
+
338
+ class R13FrozenR12Adapter:
339
+ """Head-local compatibility probe, never writes/changes model weights.
340
+
341
+ Category and all unrelated heads come from the original R12 feature feed.
342
+ Title-only inference is used for opaque IDs or names R12 enriched; ordinary
343
+ titles retain the frozen model's trained feature distribution. S8 can use
344
+ title_name_bytes for every row during native joint training.
345
+
346
+ Blanket identifier neutrality is an opt-in compatibility candidate: it
347
+ passes the pooled train gate but regresses known validation titles. Always
348
+ expose its ablation separately; the safe default uses title-only inference.
349
+ """
350
+ def __init__(self, model_path: str | Path, train_audit: Mapping[str, Any], threads: int = 2,
351
+ *, enable_identifier_guard: bool = False):
352
+ import onnxruntime as ort
353
+ options = ort.SessionOptions()
354
+ options.intra_op_num_threads = threads
355
+ options.inter_op_num_threads = 1
356
+ self.session = ort.InferenceSession(str(model_path), sess_options=options,
357
+ providers=["CPUExecutionProvider"])
358
+ self.stats_dim = next(i.shape[1] for i in self.session.get_inputs() if i.name == "stats")
359
+ self.train_audit = train_audit
360
+ self.enable_identifier_guard = enable_identifier_guard
361
+
362
+ def predict_batch(self, rows: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
363
+ if not rows:
364
+ return []
365
+ baseline = r12_features()
366
+ clean = [inference_input(r) for r in rows]
367
+ head_names = [o.name for o in self.session.get_outputs()
368
+ if o.name in baseline.NEURAL_HEAD_CLASSES and len(o.shape) == 2]
369
+ feed = baseline.build_onnx_batch_inputs(clean, stats_dim=self.stats_dim)
370
+ logits = dict(zip(head_names, self.session.run(head_names, feed)))
371
+ routed = []
372
+ for i, row in enumerate(clean):
373
+ name = str(row.get("name") or row.get("title") or "")
374
+ if (identifier_family(name, row["files"])
375
+ or baseline.build_enriched_name_text(name, row["files"]) != name):
376
+ routed.append(i)
377
+ title_logits = None
378
+ if routed:
379
+ title_feed = build_title_language_batch_inputs([clean[i] for i in routed], self.stats_dim)
380
+ title_logits = self.session.run(["title_language"], title_feed)[0]
381
+ route_index = {index: j for j, index in enumerate(routed)}
382
+ result = []
383
+ for i, row in enumerate(clean):
384
+ raw = {h: baseline.NEURAL_HEAD_CLASSES[h][int(np.argmax(values[i]))]
385
+ for h, values in logits.items()}
386
+ view = dict(raw)
387
+ if i in route_index:
388
+ view["title_language"] = baseline.NEURAL_HEAD_CLASSES["title_language"][
389
+ int(np.argmax(title_logits[route_index[i]]))]
390
+ guarded, reasons = apply_head_guards(row, view, self.train_audit,
391
+ enable_identifier_guard=self.enable_identifier_guard)
392
+ candidate, candidate_reasons = apply_head_guards(row, view, self.train_audit,
393
+ enable_identifier_guard=True)
394
+ result.append({"baseline": raw, "title_only": view, "predictions": guarded,
395
+ "guard_reasons": reasons, "title_view_routed": i in route_index,
396
+ "identifier_policy_candidate": candidate,
397
+ "identifier_policy_guard_reasons": candidate_reasons})
398
+ return result
399
+
400
+
401
+ def sha256_file(path: str | Path) -> str:
402
+ digest = hashlib.sha256()
403
+ with Path(path).open("rb") as stream:
404
+ for chunk in iter(lambda: stream.read(1024 * 1024), b""):
405
+ digest.update(chunk)
406
+ return digest.hexdigest()
r13_features.py ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Explicit R13 tensor ABI: 40 S2 features plus raw title byte length.
2
+
3
+ The last scalar is a mask boundary, never a learned feature. Both views share
4
+ the transformer; the title head cannot attend to appended file names.
5
+ """
6
+
7
+ from __future__ import annotations
8
+
9
+ import numpy as np
10
+
11
+ try:
12
+ from . import r13_feature_extractor as s2
13
+ except ImportError:
14
+ try:
15
+ import r13_feature_extractor as s2
16
+ except ImportError:
17
+ from r13_master_program.s2_neural_features import r13_feature_extractor as s2
18
+
19
+ BASE_STATS_DIM, SIDE_DIM, STATS_DIM = 12, 28, 41
20
+ INPUTS = ("name_bytes", "name_len", "ext", "size", "depth", "words", "stats")
21
+
22
+
23
+ def encode_torrent_features(row, max_name_len=256, max_files=64):
24
+ encoded = s2.encode_torrent_features(row, max_name_len, max_files)
25
+ title = encoded["title_name_bytes"]
26
+ # S2's enriched representation starts with the supplied title. Empty titles
27
+ # have a stable space token rather than consuming any file-derived byte.
28
+ if not str(
29
+ s2.inference_input(row).get("name")
30
+ or s2.inference_input(row).get("title")
31
+ or ""
32
+ ):
33
+ encoded["name_bytes"] = [32]
34
+ if encoded["name_bytes"][: len(title)] != title:
35
+ raise ValueError("Enriched input no longer preserves the raw title prefix")
36
+ encoded["stats"] = list(encoded["stats"]) + [float(len(title))]
37
+ return encoded
38
+
39
+
40
+ def build_onnx_batch_inputs(rows, max_name_len=256, max_files=64, stats_dim=STATS_DIM):
41
+ if isinstance(stats_dim, bool) or not isinstance(stats_dim, (int, np.integer)) or stats_dim != STATS_DIM:
42
+ raise ValueError("R13 requires exactly 41 stats components; padding or truncation changes its feature ABI")
43
+ if not rows:
44
+ raise ValueError("Cannot collate an empty batch")
45
+ enc = [encode_torrent_features(r, max_name_len, max_files) for r in rows]
46
+ b, length, files = (
47
+ len(enc),
48
+ max(len(e["name_bytes"]) for e in enc),
49
+ max(1, max(len(e["ext"]) for e in enc)),
50
+ )
51
+ out = {
52
+ "name_bytes": np.zeros((b, length), np.int64),
53
+ "name_len": np.zeros(b, np.int64),
54
+ "ext": np.zeros((b, files), np.int32),
55
+ "size": np.zeros((b, files), np.int64),
56
+ "depth": np.zeros((b, files), np.int64),
57
+ "words": np.zeros((b, files, 4), np.int32),
58
+ "stats": np.zeros((b, stats_dim), np.float32),
59
+ }
60
+ for i, e in enumerate(enc):
61
+ n, f = len(e["name_bytes"]), len(e["ext"])
62
+ out["name_bytes"][i, :n], out["name_len"][i] = e["name_bytes"], n
63
+ for k in ("ext", "size", "depth", "words"):
64
+ out[k][i, :f] = e[k]
65
+ out["stats"][i] = e["stats"]
66
+ return out