Release Armiger v3.3.0 (v3.0-gold-r13.3) model weights, updated charts, and concise model cards
Browse files- .gitattributes +1 -0
- README.md +11 -43
- _r12_features_base.py +932 -0
- armiger.int8.onnx +2 -2
- armiger.onnx +2 -2
- armiger_dht_rsi.py +506 -38
- armiger_inference.py +24 -6
- assets/armiger_overview.png +3 -0
- calibration.json +2721 -134
- config.json +99 -136
- r13_feature_extractor.py +406 -0
- r13_features.py +66 -0
.gitattributes
CHANGED
|
@@ -41,3 +41,4 @@ assets/armiger_vs_torrent_classifiers_6dims.png filter=lfs diff=lfs merge=lfs -t
|
|
| 41 |
assets/armiger_categorical_breakdown.png filter=lfs diff=lfs merge=lfs -text
|
| 42 |
assets/armiger_vs_llms_overall_and_6dims.png filter=lfs diff=lfs merge=lfs -text
|
| 43 |
assets/armiger_vs_llms_cost_comparison.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 41 |
assets/armiger_categorical_breakdown.png filter=lfs diff=lfs merge=lfs -text
|
| 42 |
assets/armiger_vs_llms_overall_and_6dims.png filter=lfs diff=lfs merge=lfs -text
|
| 43 |
assets/armiger_vs_llms_cost_comparison.png filter=lfs diff=lfs merge=lfs -text
|
| 44 |
+
assets/armiger_overview.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -21,28 +21,18 @@ tags:
|
|
| 21 |
- cpu-inference
|
| 22 |
---
|
| 23 |
|
| 24 |
-
# Armiger `v3.
|
| 25 |
|
| 26 |
-
**Release:** [`v3.
|
| 27 |
|
| 28 |
-
**Armiger (`NorviTech/armiger`)** is the CPU
|
| 29 |
|
| 30 |
-
|
| 31 |
|
| 32 |
-
|
| 33 |
-
- **INT8 accuracy (all 3 suites):**
|
| 34 |
-
- **`Suite B LOTO N=1,025`:** **`96.85%`** 6-dimension composite (`95% CI [96.33%, 97.36%]` with plugins) vs. Gemini Flash (`94.56%`), OpenAI GPT Fast Tier (`93.49%`, 5-dim; Junk N/A (Unprompted)), Claude Haiku 4.5 (`92.88%`).
|
| 35 |
-
- **`W5A-Test N=703` (Rule-3 Gold):** **`97.01%`** 11-class accuracy (`95% Wilson CI [95.47%, 98.04%]`); **`96.87%`** 6-core-head exact match.
|
| 36 |
-
- **`R5-Test N=996` (7-key zero-leakage):** **`91.16%`** INT8 9-class accuracy (FP32 `91.27%`; `99.90%` parity), **`99.22%`** Title Language (`n=511`), **`99.09%`** English Subtitles (`n=110`), **`98.90%`** Owner Scope (`n=996`), **`90.32%`** BIO Balanced Byte Accuracy (`n=882`), **`85.84%`** all-heads exact match (`n=996`).
|
| 37 |
-
|
| 38 |
-

|
| 39 |
-
|
| 40 |
-
## Quickstart (Standalone CPU ONNX Runtime)
|
| 41 |
-
|
| 42 |
-
### Option A — Install the Python package (`v3.2.1`) with verified model download
|
| 43 |
|
| 44 |
```bash
|
| 45 |
-
python -m pip install "armiger @ git+https://github.com/spencernorton/armiger.git@v3.
|
| 46 |
python -m armiger download --directory ./models-cpu --variant cpu
|
| 47 |
```
|
| 48 |
|
|
@@ -62,31 +52,9 @@ res = model.predict({
|
|
| 62 |
print(res["heads"]["category"], res["matching_identity"], res["mondrian_gates"]["passed_all"])
|
| 63 |
```
|
| 64 |
|
| 65 |
-
##
|
| 66 |
-
|
| 67 |
-
```python
|
| 68 |
-
from armiger_inference import ArmigerStandaloneONNX
|
| 69 |
-
|
| 70 |
-
model = ArmigerStandaloneONNX() # auto-loads armiger.onnx, vocab.json, calibration.json
|
| 71 |
-
res = model.predict({
|
| 72 |
-
"name": "Tears.of.Steel.2012.1080p.BluRay.x264-OPEN",
|
| 73 |
-
"size": 8_500_000_000,
|
| 74 |
-
"files": [{"path": "Tears.of.Steel.2012.mkv", "size": 8_500_000_000}],
|
| 75 |
-
})
|
| 76 |
-
print(res["heads"]["category"], res["matching_identity"], res["mondrian_gates"]["passed_all"])
|
| 77 |
-
```
|
| 78 |
-
|
| 79 |
-
## Ready-Made Docker Setups & Integrations (`v3.2.1`)
|
| 80 |
|
| 81 |
-
- **Full FP32 + 6-
|
| 82 |
-
- **Core
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
- **[Standalone Armiger Ultra HTTP Container](https://github.com/spencernorton/armiger/blob/main/docs/quickstart.md):** Prebuilt multi-arch image `ghcr.io/spencernorton/armiger:3.2.1` with `/v1/classify` and `/healthz`
|
| 86 |
-
- **[Python API & Component Reference](https://github.com/spencernorton/armiger/blob/main/docs/reference.md)** · **[Evaluation & Limitations](https://github.com/spencernorton/armiger/blob/main/docs/evaluation.md)**
|
| 87 |
-
- **Standalone sidecars & connectors (`v1.1.1`):** [`spencernorton/Armiger-Connectors`](https://github.com/spencernorton/Armiger-Connectors)
|
| 88 |
-
- **Stock Bitmagnet (`bitmagnet-io/bitmagnet`):** [`bitmagnet/bitmagnet_connector.py`](https://github.com/spencernorton/Armiger-Connectors/blob/main/bitmagnet/bitmagnet_connector.py) — polls untagged torrents via GraphQL, runs vectorized Armiger inference, prunes `.exe`/`.scr`/`.lnk` malware and honeypot traps, writes `armiger-*` tags.
|
| 89 |
-
- **Stock Rats Search (`librats/rats-search`):** [`rats-search/rats_search_connector.py`](https://github.com/spencernorton/Armiger-Connectors/blob/main/rats-search/rats_search_connector.py) — drop-in REST/SQLite sidecar classifier and safety filter.
|
| 90 |
-
- **DH-Waypoint `processorapi/v1` plugin** ([`spencernorton/DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint)): [`dh-waypoint/`](https://github.com/spencernorton/Armiger-Connectors/tree/main/dh-waypoint) — admission (`/v1/admit`), batch (`/v1/classify`), 4-tier feedback (`/v1/feedback`).
|
| 91 |
-
- **Torznab pre-grab proxy (`Prowlarr`/`Sonarr`/`Radarr`/`Lidarr`/`Readarr`):** [`arr-stack/armiger_torznab_proxy.py`](https://github.com/spencernorton/Armiger-Connectors/blob/main/arr-stack/armiger_torznab_proxy.py) — filters malware lures, dead swarms, and unrequested foreign-audio releases before grab.
|
| 92 |
-
- **Post-import `*Arr`/qBittorrent webhook:** [`arr-stack/armiger_arr_webhook_receiver.py`](https://github.com/spencernorton/Armiger-Connectors/blob/main/arr-stack/armiger_arr_webhook_receiver.py) — `ffprobe` ground truth for local conformal calibration and opt-in anonymous BEP-44/46 updates.
|
|
|
|
| 21 |
- cpu-inference
|
| 22 |
---
|
| 23 |
|
| 24 |
+
# Armiger `v3.3.0` — INT8 ONNX CPU Classifier
|
| 25 |
|
| 26 |
+
**Release:** [`v3.3.0`](https://github.com/spencernorton/armiger/releases/tag/v3.3.0) · **Artifact:** `17.6 MB` INT8 ONNX (`armiger.int8.onnx` / `armiger.onnx`) · **Parameters:** `8.05M` · **License:** Apache-2.0
|
| 27 |
|
| 28 |
+
**Armiger (`NorviTech/armiger`)** is the standalone CPU INT8 edition of the byte-level RoPE Transformer (`numpy` + `onnxruntime` only). For FP32 weights, pre-built `armiger-ultra` containers, and the 6-plugin pipeline, see [`NorviTech/armiger-ultra`](https://huggingface.co/NorviTech/armiger-ultra).
|
| 29 |
|
| 30 |
+

|
| 31 |
|
| 32 |
+
## Quickstart
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
```bash
|
| 35 |
+
python -m pip install "armiger @ git+https://github.com/spencernorton/armiger.git@v3.3.0"
|
| 36 |
python -m armiger download --directory ./models-cpu --variant cpu
|
| 37 |
```
|
| 38 |
|
|
|
|
| 52 |
print(res["heads"]["category"], res["matching_identity"], res["mondrian_gates"]["passed_all"])
|
| 53 |
```
|
| 54 |
|
| 55 |
+
## Links & Ecosystem
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
|
| 57 |
+
- **Full FP32 + 6-Plugin Suite:** [`NorviTech/armiger-ultra`](https://huggingface.co/NorviTech/armiger-ultra)
|
| 58 |
+
- **Core Package, Containers & Evaluation (`v3.3.0`):** [`spencernorton/armiger`](https://github.com/spencernorton/armiger) (`ghcr.io/spencernorton/armiger-ultra:3.3.0`)
|
| 59 |
+
- **Indexer Sidecars (Bitmagnet, Rats Search, `*Arr` Torznab):** [`spencernorton/Armiger-Connectors`](https://github.com/spencernorton/Armiger-Connectors)
|
| 60 |
+
- **DH-Waypoint DHT Crawler & Indexer:** [`spencernorton/DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
_r12_features_base.py
ADDED
|
@@ -0,0 +1,932 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Standalone Armiger v2.1 / Armiger Ultra ONNX Featurizer & Inference Runtime (`armiger_inference.py`).
|
| 2 |
+
|
| 3 |
+
Implements exact feature parity with the Armiger v2.1 training pipeline (`sift/prepare.py` +
|
| 4 |
+
`enrich_side21_v2` 36-dim `stats` vector + `vocab.json` 4,096-extension lookup + CRC32
|
| 5 |
+
digit-folded path-word hashing) with zero external codebase dependencies beyond `numpy`
|
| 6 |
+
and `onnxruntime`.
|
| 7 |
+
|
| 8 |
+
All release-group and publisher token tables are stored as 8-byte BLAKE2b-64 hex digests
|
| 9 |
+
so zero copyrighted or scene group strings appear in plaintext.
|
| 10 |
+
|
| 11 |
+
R12 Session 2: derivative-group file-tree features
|
| 12 |
+
(`U_deriv_groups`, `E_episode_keys`, `P_multipart_idx`, `s_max`) are computed by
|
| 13 |
+
`compute_derivative_group_features` and appended to the side vector at
|
| 14 |
+
`side[24..27]`; negated subtitle clauses are stripped before the `side[13]`
|
| 15 |
+
subtitle cue; opaque top-level names gain a `|FILES: ...` primary-media suffix
|
| 16 |
+
in `name_bytes`. Legacy `side[0..23]` semantics are unchanged (only previously
|
| 17 |
+
dead `side[13]` becomes live), so `stats[:36]` remains the legacy R10/R11
|
| 18 |
+
prefix. Training (R12 Sessions 5 & 7) and inference must import the shared
|
| 19 |
+
`SIDE_DIM` / `STATS_DIM` constants below instead of hard-coding `24` / `36`.
|
| 20 |
+
"""
|
| 21 |
+
from __future__ import annotations
|
| 22 |
+
|
| 23 |
+
import hashlib
|
| 24 |
+
import json
|
| 25 |
+
import math
|
| 26 |
+
import os
|
| 27 |
+
from pathlib import Path
|
| 28 |
+
import re
|
| 29 |
+
from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple
|
| 30 |
+
import zlib
|
| 31 |
+
|
| 32 |
+
try:
|
| 33 |
+
import numpy as np
|
| 34 |
+
except ImportError:
|
| 35 |
+
np = None # type: ignore[assignment]
|
| 36 |
+
|
| 37 |
+
SPAN_BIO_CLASSES: Tuple[str, ...] = (
|
| 38 |
+
"O",
|
| 39 |
+
"B-TITLE",
|
| 40 |
+
"I-TITLE",
|
| 41 |
+
"B-YEAR",
|
| 42 |
+
"I-YEAR",
|
| 43 |
+
"B-EP",
|
| 44 |
+
"I-EP",
|
| 45 |
+
)
|
| 46 |
+
|
| 47 |
+
# Side/stats dimensionality contract (R12 Session 2). `stats = base_stats + side`.
|
| 48 |
+
BASE_STATS_DIM: int = 12
|
| 49 |
+
LEGACY_SIDE_DIM: int = 24
|
| 50 |
+
SIDE_DIM: int = 28 # 24 legacy slots + 4 derivative-group features at side[24..27]
|
| 51 |
+
N_SIDE: int = SIDE_DIM
|
| 52 |
+
STATS_DIM: int = BASE_STATS_DIM + SIDE_DIM # 40 (legacy R10/R11 width was 36)
|
| 53 |
+
LEGACY_STATS_DIM: int = BASE_STATS_DIM + LEGACY_SIDE_DIM # 36
|
| 54 |
+
SIDE_FEATURE_NAMES: Tuple[str, ...] = (
|
| 55 |
+
"ebook_share", "pdf_share", "m4b_share", "comic_share", "console_share",
|
| 56 |
+
"app_share", "exec_share", "iso_share", "archive_share", "html_share",
|
| 57 |
+
"script_ext_present", "trap_flag", "small_video_flag", "subtitle_cue",
|
| 58 |
+
"sub_only_group", "anime_keyword", "dub_keyword", "game_group",
|
| 59 |
+
"software_group", "course_keyword", "book_comic_keyword",
|
| 60 |
+
"lang_english", "lang_non_english", "lang_unknown",
|
| 61 |
+
"deriv_u_log", "deriv_e_log", "deriv_p_multipart", "deriv_s_max",
|
| 62 |
+
)
|
| 63 |
+
|
| 64 |
+
NEURAL_HEAD_CLASSES: Dict[str, List[str]] = {
|
| 65 |
+
"category": [
|
| 66 |
+
"movie", "tv_show", "xxx", "music", "comic",
|
| 67 |
+
"game", "audiobook", "software", "ebook", "course", "unknown",
|
| 68 |
+
],
|
| 69 |
+
"adult": ["non_adult", "adult"],
|
| 70 |
+
"junk": ["clean", "junk"],
|
| 71 |
+
"anime": ["non_anime", "anime"],
|
| 72 |
+
"english_subs": ["no_eng_subs", "has_eng_subs", "unknown"],
|
| 73 |
+
"lang": ["english", "non_english", "multi_or_unknown"],
|
| 74 |
+
"owner_scope": ["allowed_type", "excluded"],
|
| 75 |
+
"title_language": ["english_title", "non_english_title"],
|
| 76 |
+
"span_bio": list(SPAN_BIO_CLASSES),
|
| 77 |
+
}
|
| 78 |
+
|
| 79 |
+
_VOCAB_EXT_MAP: Optional[Dict[str, int]] = None
|
| 80 |
+
_PREP_KIND_EXTS = {
|
| 81 |
+
0: frozenset("mkv mp4 avi ts m2ts mts vob wmv mpg mpeg mov flv rmvb rm m4v divx webm 3gp asf f4v ogv mxf m2v bik".split()),
|
| 82 |
+
1: frozenset("mp3 flac m4a m4b aac ogg opus wav wma ape alac aif aiff aifc ac3 dts mka mpc wv shn mid midi dff dsf".split()),
|
| 83 |
+
2: frozenset("zip rar 7z gz tgz bz2 xz zst lz lzma tar cab iso img dmg mdf mds nrg wim par par2 z".split()),
|
| 84 |
+
3: frozenset("exe msi apk ipa dll bat cmd sh vbs ps1 com scr jar deb rpm appimage run lnk".split()),
|
| 85 |
+
4: frozenset("jpg jpeg png gif webp bmp tif tiff avif jxl heic heif psd ai svg ico tga cr2 nef arw dng raw eps".split()),
|
| 86 |
+
5: frozenset("pdf epub mobi azw azw3 djvu djv fb2 txt rtf doc docx odt xls xlsx ppt pptx csv chm cbz cbr cb7 html htm mht mhtml md tex lit".split()),
|
| 87 |
+
6: frozenset("srt ass ssa sub idx vtt sup smi sbv ttml dfxp usf scc stl mks".split()),
|
| 88 |
+
}
|
| 89 |
+
_PREP_EXT_TO_KIND = {e: k for k, es in _PREP_KIND_EXTS.items() for e in es}
|
| 90 |
+
_PREP_SPLIT_ARC_RE = re.compile(r"[rsz]\d{2,3}|\d{3}")
|
| 91 |
+
_PREP_WORD_RE = re.compile(r"[^\W_]+")
|
| 92 |
+
_PREP_DIGIT_RE = re.compile(r"\d")
|
| 93 |
+
_PREP_SAMPLE_RE = re.compile(r"(?<![a-z])samples?(?![a-z])", re.IGNORECASE)
|
| 94 |
+
|
| 95 |
+
# BLAKE2b-64 digest sets for release-tag disambiguation (zero plaintext group names)
|
| 96 |
+
_SUB_ONLY_GRP_H = frozenset(['00d47045a1f1cd70', '0380c309e464115a', '0665f92d474878ff', '0f8f3809c6074d35', '1bc080c5d14b4eae', '1bf62e9b91086379', '22325081668ed28b', '26503f6826942362', '33b21e80d5dbfa62', '3e370e64b7539495', '4297edae7e25ba02', '4491be5b41a66832', '44e08227cfda9421', '4b956de9c462ee70', '519230ab4d0911f7', '581a8006ea8bd801', '65dfe6849ae4bb0d', '66ced9df20d8f558', '6a437c446b5778d6', '6cff013c3e752ac8', '6d3b6a62beedc878', '7bc52c5f863149cc', '867c636203d0524d', 'a7637cb5c531d8e9', 'b61d75b73d8e0c59', 'bed3322685febfa5', 'c603a601b7cb9d4c', 'd4eaba2bc7ef10d6', 'e20a8d50ddf1aa2e', 'e3fd326207da261c', 'fc94bcdf9fcb0164'])
|
| 97 |
+
_DUAL_DUB_GRP_H = frozenset(['16bdcdbbbd493d9e', '17addcc669201a7b', '1cb486cd76cea8c6', '1cc2f6b1aa24d6f7', '236a71bfa2d47ca3', '24eee113ee4bddb0', '341556a463a4784b', '3658912d4f05f762', '6149de24f7784f27', '627dcb38527b8e4c', '6f09d409143a679c', '727c5e1f0f05ad51', '8b7c320e1df4bf3e', '9e64e0c8c213bf00', 'ac3675ff37aaf11e', 'b2c26515563660f5', 'bd672057ff15b985', 'd9621931df7eb664', 'e20db1736162ba0c', 'e2ee075c6b5545ad', 'eb446df9e0b5f645', 'f7a175e4cf1907fb', 'fe0fe2b14bb48506'])
|
| 98 |
+
_GAME_GRP_H = frozenset(['03f22fe2a782da12', '0ae36a7939ac0be1', '1217fb0deda20335', '1c8f3166691fa2c6', '306372c75c8ceaa2', '32ba70bb6b5ae022', '4509f87fe97d6ed9', '4afe068be7dcf016', '6f7a14ce24f000a0', '7403ff2453c8a173', '7de6d237db23b5fa', '9641051915abdbb6', 'a491e84e39e2064e', 'a65a1816debe0915', 'aa89d2a9d5d18f01', 'aaac60aa2d75341b', 'bb98906accdd6e05', 'c200372b38fb6d26', 'c430ddffd526c993', 'cc81415c4d797870', 'e37a0da20d8d24ac', 'ed5d3b0f00026714', 'ee7b45399bcf968a', 'f0982ea7c2028c50', 'f1737c198253be9d', 'f766dbb74e38db04', 'fbe41c3bbd33ac19'])
|
| 99 |
+
_SW_GRP_H = frozenset(['09455d377a262608', '0e30392b37df5495', '103064b4fea08f1c', '11a120607b793e4c', '158ba6de7392b368', '193a01c7e1bd0f62', '2199b8b3ff553486', '240843438078b817', '2508ee34b17c2225', '2a4eb6e114be0fab', '352836dac99f04b7', '391bd8a58914224a', '4185301675b9d80c', '4472c2744066912a', '4c4684b615a127b2', '522a470219d3a4f1', '5a6bb90382b6c1f6', '5ca7613c608a9ea5', '63f15e4687bfc135', '66a7555e70f0872e', '73d877e885d34562', '77c343b865d05e31', '780409423e866d09', '7e0a43f55a6714fd', '97409999bccb98f4', '996bd865c828c8f9', '9b60cdfff0e2ab1c', 'a330b94802e4b6c6', 'a9c5a9029bc6a596', 'b48e9326bf6316ef', 'c595d125bfd5ebe9', 'c8575232dac0347d', 'd64fa92611ac0eb2', 'e182e241e26cb119', 'e88087f130a6e7d6'])
|
| 100 |
+
_BOOK_COMIC_GRP_H = frozenset(['0cad54552908fa65', '338c3fad71b18311', '5913147687b3c1e5', '712f82b4b85131a0', '7f5fc23af41380b2', '8d8b63a8bb00b7e4', '913bd80742a62e95', 'c00722c39b756d92', 'ecfebbed5476af20', 'ee2807f40b38f779', 'f89c7ccfb7b1338c', 'fdffac028227e18e', 'ffb2c2ead4c7ba7e'])
|
| 101 |
+
_BENIGN_README_EXE_H = frozenset(['062b96e72d602b6e', '9a9f15179701c6ae', 'ddf05591355cf7e0'])
|
| 102 |
+
|
| 103 |
+
_TOKEN_SCAN_RE = re.compile(r"[A-Za-z0-9]+(?:-[A-Za-z0-9]+)*")
|
| 104 |
+
_SIDE_ANIME_KW_RE = re.compile(
|
| 105 |
+
r"(?i)\b(?:anime|ova|ona|donghua|aeni|bdrip[ ._-]*1080p[ ._-]*flac|dual[ ._-]*audio[ ._-]*\[?(?:jpn|jap))\b"
|
| 106 |
+
)
|
| 107 |
+
_SIDE_VID_CLAIM_RE = re.compile(
|
| 108 |
+
r"(?i)\b(?:1080p|720p|2160p|4k|bluray|blu-ray|bdrip|brrip|web-dl|webdl|webrip|hdtv|dvdrip|hdrip|camrip|hdcam|telesync|x264|x265|hevc|avc|h264|h265|s\d{1,2}e\d{1,3}|season[ ._-]*\d+|fc2-?ppv)\b|高清首发|更新至第"
|
| 109 |
+
)
|
| 110 |
+
_SIDE_DECEPTIVE_SUB_STRIP_RE = re.compile(
|
| 111 |
+
r"(?i)\b(?:eng(?:lish)?|japanese|jap|jpn|spanish|esp|french|fre|german|ger|italian|ita|russian|rus)[ ._-]+(?:hard|soft)?sub(?:s|bed|titled|titles)?\b|"
|
| 112 |
+
r"\b(?:hard|soft)?sub(?:s|bed|titled|titles)?[ ._-]+(?:eng|english|esp|ja|jpn|fr|de|it|ru)\b|"
|
| 113 |
+
r"\b(?:m[ ._-]?subs?|multi[ ._-]?subs?|esubs?|vostfr|hardsubs?)\b"
|
| 114 |
+
)
|
| 115 |
+
_SIDE_ANIME_DUB_KW_RE = re.compile(
|
| 116 |
+
r"(?i)\b(?:dual[ ._-]?audio|dubbed|eng(?:lish)?[ ._-]?dub(?:bed)?|multi[ ._-]?audio)\b"
|
| 117 |
+
)
|
| 118 |
+
_SIDE_SW_APP_TITLE_RE = re.compile(
|
| 119 |
+
r"(?i)\b(?:v?\d+\.\d+\.\d+|repack|portable|cracked|keygen|patch|patcher|activat(?:ed|or)|"
|
| 120 |
+
r"multilingual|win(?:32|64)|x64|x86|macos|macosx|android|apk|pc[ ._-]?game|nintendo[ ._-]?switch|ps4|ps5|"
|
| 121 |
+
r"emulator|driver|firmware|setup|installer|downloader|studio|vst3?|"
|
| 122 |
+
r"pro[ ._-]+v?\d|ce[ ._-]+v?\d)\b"
|
| 123 |
+
)
|
| 124 |
+
_SIDE_COURSE_RE = re.compile(
|
| 125 |
+
r"(?i)\b(?:complete[ ._-]+course|video[ ._-]+course|masterclass|bootcamp|tutorial[ ._-]+series|certification[ ._-]+course|hands-on[ ._-]+training)\b"
|
| 126 |
+
)
|
| 127 |
+
_SIDE_BOOK_COMIC_KW_RE = re.compile(
|
| 128 |
+
r"(?i)\b(?:audiobooks?|audible|m4b|unabridged|narrated[ ._-]?by|ebook|e-book|epub|mobi|azw3|calibre)\b"
|
| 129 |
+
)
|
| 130 |
+
_SIDE_VPN_TRAP_RE = re.compile(
|
| 131 |
+
r"(?i)免费安装下载vpn|打不开网页|codec[ ._-]?(?:pack|setup|install)|player[ ._-]?setup|instructions?[ ._-]?to[ ._-]?watch"
|
| 132 |
+
)
|
| 133 |
+
_SIDE_CJK_RE = re.compile(r"[\u4e00-\u9fff]")
|
| 134 |
+
_SIDE_EP_ZIP_RE = re.compile(
|
| 135 |
+
r"(?i)(?:^\[[^\[\]]{2,30}\].*?(?:\s-\s*\d{1,3}\b|\bS\d{1,2}E\d{1,3}\b).*?\b(?:1080p|720p|2160p|480p)\b|"
|
| 136 |
+
r"\bS\d{1,2}E\d{1,3}\b.*?\b(?:1080p|720p|2160p|480p|WEB-DL|WEBRip|BluRay|BDRip|HDTV|x264|x265|HEVC)\b|"
|
| 137 |
+
r"\b(?:1080p|720p|2160p|480p|576p|4k)\b.*?\b(?:WEB-DL|WEBRip|BluRay|BDRip|BRRip|HDRip|HDTV|HDCAM|CAMRip|DVDRip|x264|x265|H\.?264|H\.?265|HEVC|AV1|XviD|DivX)\b|"
|
| 138 |
+
r"\b(?:WEB-DL|WEBRip|BDRip|BRRip|HDRip|HDTV|HDCAM|CAMRip|DVDRip)\b(?:.*?\b(?:x264|x265|H\.?264|H\.?265|HEVC|AV1|XviD|DivX))?)"
|
| 139 |
+
)
|
| 140 |
+
_SIDE_BENIGN_ZIP_RE = re.compile(
|
| 141 |
+
r"(?i)\b(?:batch|complete|season[ ._-]*\d+|s\d{1,2}[ ._-]*complete|e?\d{1,3}[ ._-]*[~-][ ._-]*e?\d{1,3}|시즌\s*\d+|완결|全集|合集|"
|
| 142 |
+
r"ost|soundtrack|artbook|scans?|wallpapers?|fonts?|subs?|subtitles?)\b"
|
| 143 |
+
)
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
# --- R12 Session 2: derivative-group file-tree features ---------------------
|
| 147 |
+
# Brief-mandated video set (.mkv .mp4 .avi .ogv .mpg .mpeg .m4v .ts .wmv .mov
|
| 148 |
+
# .webm .vob .iso) plus same-family containers seen in the gold corpus
|
| 149 |
+
# (.m2ts/.mts/.divx/.flv/.rmvb/.3gp/.asf) so primary videos are never missed.
|
| 150 |
+
_DERIV_VIDEO_EXTS = frozenset(
|
| 151 |
+
"mkv mp4 avi ogv mpg mpeg m4v ts wmv mov webm vob iso m2ts mts divx flv rmvb 3gp asf".split()
|
| 152 |
+
)
|
| 153 |
+
_DERIV_SAMPLE_RE = re.compile(r"(?i)\bsample\b")
|
| 154 |
+
# IA/transcode rendition suffixes stripped repeatedly from a video stem.
|
| 155 |
+
_DERIV_SUFFIX_RE = re.compile(
|
| 156 |
+
r"(?i)(?:\.(?:ia|512kb|h264|h\.264|x264|hd|sd)|_(?:archive|thumbs|files|512kb|h264|ia))+$"
|
| 157 |
+
)
|
| 158 |
+
_EP_SXE_RE = re.compile(r"(?i)(?<![a-z0-9])s(\d{1,2})[\s._\-]*e(\d{1,3})(?![0-9])")
|
| 159 |
+
_EP_NXN_RE = re.compile(r"(?<![0-9a-zA-Z])(\d{1,2})x(\d{2})(?![0-9a-zA-Z])", re.IGNORECASE)
|
| 160 |
+
_EP_WORD_RE = re.compile(r"(?i)\b(?:episode)[\s._\-]*(\d{1,3})(?![0-9])")
|
| 161 |
+
_EP_SHORT_RE = re.compile(r"(?i)(?<![a-z0-9])ep\.?\s*(\d{1,3})(?![0-9])")
|
| 162 |
+
_EP_BARE_E_RE = re.compile(r"(?i)(?<![a-z0-9])e(\d{1,3})(?![a-z0-9])")
|
| 163 |
+
# Anime fansub dash numbering (`[Group] Show - 01v2 [CRC].mkv`), train-validated in WS-1.
|
| 164 |
+
_EP_DASH_NUM_RE = re.compile(r"\s-\s(\d{1,3})(?:v\d)?(?![0-9a-zA-Z])")
|
| 165 |
+
_SEASON_HINT_RE = re.compile(
|
| 166 |
+
r"(?i)(?:(?<![a-z0-9])s(\d{1,2})(?![0-9])|\bseason[\s._\-]*(\d{1,2})(?![0-9]))"
|
| 167 |
+
)
|
| 168 |
+
_MULTIPART_TOKEN_RE = re.compile(
|
| 169 |
+
r"(?i)\b(?:cd|disc|disk|pt|part|side)[\s._\-]*([1-4])\b"
|
| 170 |
+
)
|
| 171 |
+
_VTS_STEM_RE = re.compile(r"(?i)^vts_\d+_\d+$")
|
| 172 |
+
_PILOT_RE = re.compile(r"(?i)\b(?:unaired[\s._\-]*)?pilot\b")
|
| 173 |
+
_YEAR_TOKEN_RE = re.compile(r"(?<![0-9])(?:19[2-9]\d|20[0-2]\d)(?![0-9])")
|
| 174 |
+
# Negated subtitle clauses are removed before the side[13] subtitle cue so
|
| 175 |
+
# "no english subs" / "without eng subs" / "non-english subtitles" cannot fire it.
|
| 176 |
+
_NEGATED_SUB_RE = re.compile(
|
| 177 |
+
r"(?i)\b(?:no|without|non|none|pas\s+de|kein[e]?|sin)[\s._\-]+"
|
| 178 |
+
r"(?:[-\w]+[\s._\-]+){0,2}?(?:eng(?:lish)?[\s._\-]+)?"
|
| 179 |
+
r"(?:sub(?:title)?s?|srt|ass|ssa|cc)\b"
|
| 180 |
+
r"|\bsub(?:title)?s?[\s:_-]*(?:no|none|n/a|nil|false)\b"
|
| 181 |
+
)
|
| 182 |
+
_SUB_CUE_RE = re.compile(
|
| 183 |
+
r"(?i)(?:\b(?:eng(?:lish)?|en|hard|soft|multi)[\s._\-]*"
|
| 184 |
+
r"(?:sub(?:title)?s?|srt|ass|ssa|cc)\b"
|
| 185 |
+
r"|\b(?:subtitles?|subs|subbed|esubs?|closed[\s._\-]*captions?)\b"
|
| 186 |
+
r"|\.(?:srt|ass|ssa)\b)"
|
| 187 |
+
)
|
| 188 |
+
_MEDIA_SUMMARY_EXTS = frozenset(
|
| 189 |
+
set(_DERIV_VIDEO_EXTS)
|
| 190 |
+
| set("mp3 flac m4b m4a ogg opus epub mobi azw azw3 pdf cbz cbr cb7 "
|
| 191 |
+
"exe msi apk dmg nsp xci xcz nsz".split())
|
| 192 |
+
)
|
| 193 |
+
_DERIV_LOG_NORM: float = math.log1p(64.0)
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
def _normalize_file_entries(files: Any) -> List[Tuple[str, int]]:
|
| 197 |
+
"""Normalize torrent `files` (dicts / (path, size) tuples / strings) to `(path, size)` pairs."""
|
| 198 |
+
out: List[Tuple[str, int]] = []
|
| 199 |
+
for f in files or []:
|
| 200 |
+
if isinstance(f, Mapping):
|
| 201 |
+
p = str(f.get("path") or f.get("p") or f.get("name") or "")
|
| 202 |
+
try:
|
| 203 |
+
sz = int(f.get("size") or f.get("s") or 0)
|
| 204 |
+
except (TypeError, ValueError, OverflowError):
|
| 205 |
+
sz = 0
|
| 206 |
+
elif isinstance(f, (tuple, list)) and len(f) >= 1:
|
| 207 |
+
p = str(f[0] or "")
|
| 208 |
+
try:
|
| 209 |
+
sz = int(f[1]) if len(f) > 1 else 0
|
| 210 |
+
except (TypeError, ValueError, OverflowError):
|
| 211 |
+
sz = 0
|
| 212 |
+
else:
|
| 213 |
+
p, sz = str(f or ""), 0
|
| 214 |
+
if p and "/.____padding_file/" not in p and not p.startswith(".____padding_file"):
|
| 215 |
+
out.append((p, max(sz, 0)))
|
| 216 |
+
return out
|
| 217 |
+
|
| 218 |
+
|
| 219 |
+
def _deriv_stem_key(parent: str, stem: str) -> Tuple[str, str]:
|
| 220 |
+
"""Collapse transcode renditions of one work to a single derivative-group key."""
|
| 221 |
+
s = stem
|
| 222 |
+
for _ in range(4):
|
| 223 |
+
stripped = _DERIV_SUFFIX_RE.sub("", s)
|
| 224 |
+
if stripped == s:
|
| 225 |
+
break
|
| 226 |
+
s = stripped
|
| 227 |
+
norm_s = re.sub(r"[\W_]+", " ", s.lower(), flags=re.UNICODE).strip() or s.lower().strip()
|
| 228 |
+
norm_p = re.sub(r"[\W_]+", " ", parent.lower(), flags=re.UNICODE).strip()
|
| 229 |
+
return norm_p, norm_s
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
def _episode_keys_from_text(text: str, default_season: Optional[int] = None) -> set:
|
| 233 |
+
"""Distinct episode keys in one path/name string (multipart CD/Part tokens never count)."""
|
| 234 |
+
keys: set = set()
|
| 235 |
+
if not text:
|
| 236 |
+
return keys
|
| 237 |
+
seasons = {
|
| 238 |
+
int(g) for m in _SEASON_HINT_RE.finditer(text)
|
| 239 |
+
for g in m.groups() if g is not None
|
| 240 |
+
}
|
| 241 |
+
for m_sxe in _EP_SXE_RE.finditer(text):
|
| 242 |
+
seasons.add(int(m_sxe.group(1)))
|
| 243 |
+
for m_nxn in _EP_NXN_RE.finditer(text):
|
| 244 |
+
seasons.add(int(m_nxn.group(1)))
|
| 245 |
+
season = seasons.pop() if len(seasons) == 1 else (default_season if not seasons else None)
|
| 246 |
+
|
| 247 |
+
def add_bare(num: int) -> None:
|
| 248 |
+
keys.add(f"s{season:02d}e{num:03d}" if season is not None else f"ep{num:04d}")
|
| 249 |
+
|
| 250 |
+
for m in _EP_SXE_RE.finditer(text):
|
| 251 |
+
keys.add(f"s{int(m.group(1)):02d}e{int(m.group(2)):03d}")
|
| 252 |
+
for m in _EP_NXN_RE.finditer(text):
|
| 253 |
+
keys.add(f"s{int(m.group(1)):02d}e{int(m.group(2)):03d}")
|
| 254 |
+
for m in _EP_WORD_RE.finditer(text):
|
| 255 |
+
add_bare(int(m.group(1)))
|
| 256 |
+
for m in _EP_SHORT_RE.finditer(text):
|
| 257 |
+
add_bare(int(m.group(1)))
|
| 258 |
+
# Bare `E01` stems (brief's `Ep?` with optional `p`), e.g. `Show.S01/E01.mkv`.
|
| 259 |
+
# The lookbehind already excludes the `E` inside `S01E01` (digit before it).
|
| 260 |
+
for m in _EP_BARE_E_RE.finditer(text):
|
| 261 |
+
add_bare(int(m.group(1)))
|
| 262 |
+
for m in _EP_DASH_NUM_RE.finditer(text):
|
| 263 |
+
add_bare(int(m.group(1)))
|
| 264 |
+
if not keys and _PILOT_RE.search(text):
|
| 265 |
+
keys.add("pilot")
|
| 266 |
+
return keys
|
| 267 |
+
|
| 268 |
+
|
| 269 |
+
def compute_derivative_group_features(
|
| 270 |
+
files: Any,
|
| 271 |
+
raw_name: str = "",
|
| 272 |
+
) -> Tuple[float, float, float, float]:
|
| 273 |
+
"""Compute `(U_deriv_groups, E_episode_keys, P_multipart_idx, s_max)` for a torrent file tree.
|
| 274 |
+
|
| 275 |
+
- `U`: unique normalized video derivative stems; IA transcodes of one stem
|
| 276 |
+
(`.mp4`/`.ogv`/`.ia.mp4`/`.512kb` ...) collapse to one group and all
|
| 277 |
+
`VIDEO_TS/VTS_*` files collapse to one group.
|
| 278 |
+
- `E`: distinct episode keys (`SxxExx`, `NxNN`, `Ep N`, `Episode N`, bare
|
| 279 |
+
`ENN` under a season folder). Pure multipart splits (`CD1/CD2`,
|
| 280 |
+
`Part 1/2` of one film) contribute `0`; a two-part TV pilot carrying an
|
| 281 |
+
explicit `SxxExx`/`Pilot`/season marker counts its parts (`E >= 2`).
|
| 282 |
+
- `P`: `1.0` for a multi-disc/part movie split or `VIDEO_TS` with `E <= 1`.
|
| 283 |
+
- `s_max`: largest deduplicated primary stem share, using the largest file
|
| 284 |
+
per derivative stem so IA renditions cannot depress it. Sample files
|
| 285 |
+
(`sample` token, or `< 15 MB` when total video exceeds `100 MB`) are ignored.
|
| 286 |
+
"""
|
| 287 |
+
entries = _normalize_file_entries(files)
|
| 288 |
+
videos: List[Tuple[str, str, str, int]] = [] # (path, parent, stem, size)
|
| 289 |
+
for path, sz in entries:
|
| 290 |
+
p_norm = path.replace("\\", "/")
|
| 291 |
+
parent, _, base = p_norm.rpartition("/")
|
| 292 |
+
if "." not in base:
|
| 293 |
+
continue
|
| 294 |
+
stem, ext = base.rsplit(".", 1)
|
| 295 |
+
if ext.lower() in _DERIV_VIDEO_EXTS:
|
| 296 |
+
videos.append((p_norm, parent, stem, sz))
|
| 297 |
+
|
| 298 |
+
total_video = sum(sz for _, _, _, sz in videos)
|
| 299 |
+
kept = [
|
| 300 |
+
v for v in videos
|
| 301 |
+
if not _DERIV_SAMPLE_RE.search(v[0].rsplit("/", 1)[-1])
|
| 302 |
+
and not (0 < v[3] < 15_000_000 and total_video > 100_000_000)
|
| 303 |
+
]
|
| 304 |
+
if kept:
|
| 305 |
+
videos = kept
|
| 306 |
+
|
| 307 |
+
rep_bytes: Dict[Tuple[str, str], int] = {}
|
| 308 |
+
ep_keys: set = set()
|
| 309 |
+
part_indices: set = set()
|
| 310 |
+
has_video_ts = False
|
| 311 |
+
tv_marker = bool(_EP_SXE_RE.search(raw_name or "") or _PILOT_RE.search(raw_name or "")
|
| 312 |
+
or _SEASON_HINT_RE.search(raw_name or ""))
|
| 313 |
+
title_seasons = {
|
| 314 |
+
int(g) for m in _SEASON_HINT_RE.finditer(raw_name or "")
|
| 315 |
+
for g in m.groups() if g is not None
|
| 316 |
+
} | {int(m.group(1)) for m in _EP_SXE_RE.finditer(raw_name or "")} | {int(m.group(1)) for m in _EP_NXN_RE.finditer(raw_name or "")}
|
| 317 |
+
title_default_season = next(iter(title_seasons)) if len(title_seasons) == 1 else None
|
| 318 |
+
ep_keys.update(_episode_keys_from_text(raw_name or "", default_season=title_default_season))
|
| 319 |
+
|
| 320 |
+
for p_norm, parent, stem, sz in videos:
|
| 321 |
+
ep_keys.update(_episode_keys_from_text(p_norm, default_season=title_default_season))
|
| 322 |
+
part_indices.update(int(m.group(1)) for m in _MULTIPART_TOKEN_RE.finditer(p_norm))
|
| 323 |
+
if _EP_SXE_RE.search(p_norm) or _PILOT_RE.search(p_norm) or _SEASON_HINT_RE.search(p_norm):
|
| 324 |
+
tv_marker = True
|
| 325 |
+
if "/video_ts/" in f"/{p_norm.lower()}" or _VTS_STEM_RE.match(stem):
|
| 326 |
+
has_video_ts = True
|
| 327 |
+
key = ("__video_ts__", "")
|
| 328 |
+
else:
|
| 329 |
+
key = _deriv_stem_key(parent, stem)
|
| 330 |
+
rep_bytes[key] = max(rep_bytes.get(key, 0), sz)
|
| 331 |
+
|
| 332 |
+
e_count = len(ep_keys)
|
| 333 |
+
# Two-part TV pilot: explicit TV marker + distinct parts of one episode.
|
| 334 |
+
if tv_marker and len(part_indices) >= 2 and e_count <= 1:
|
| 335 |
+
e_count = len(part_indices)
|
| 336 |
+
|
| 337 |
+
u_count = len(rep_bytes)
|
| 338 |
+
p_idx = 1.0 if (has_video_ts or len(part_indices) >= 2) and e_count <= 1 else 0.0
|
| 339 |
+
rep_total = sum(rep_bytes.values())
|
| 340 |
+
if u_count == 0:
|
| 341 |
+
s_max = 0.0
|
| 342 |
+
elif rep_total > 0:
|
| 343 |
+
s_max = max(rep_bytes.values()) / rep_total
|
| 344 |
+
else: # sizes unknown: equal-share assumption over derivative groups
|
| 345 |
+
s_max = 1.0 / u_count
|
| 346 |
+
return float(u_count), float(e_count), float(p_idx), float(s_max)
|
| 347 |
+
|
| 348 |
+
|
| 349 |
+
def primary_media_file_summary(files: Any, top_n: int = 3) -> str:
|
| 350 |
+
"""Top-`top_n` largest primary media relative paths, ` ; `-joined (largest first)."""
|
| 351 |
+
media = [
|
| 352 |
+
(p, sz) for p, sz in _normalize_file_entries(files)
|
| 353 |
+
if "." in p.rsplit("/", 1)[-1]
|
| 354 |
+
and p.rsplit("/", 1)[-1].rsplit(".", 1)[-1].lower() in _MEDIA_SUMMARY_EXTS
|
| 355 |
+
]
|
| 356 |
+
media.sort(key=lambda x: -x[1])
|
| 357 |
+
return " ; ".join(p for p, _ in media[:top_n])
|
| 358 |
+
|
| 359 |
+
|
| 360 |
+
def build_enriched_name_text(name: str, files: Any, max_bytes: int = 256) -> str:
|
| 361 |
+
"""Append |FILES: <top-3 media basenames> when name is opaque or hides multi-episode files."""
|
| 362 |
+
summary = primary_media_file_summary(files, top_n=3)
|
| 363 |
+
if not summary:
|
| 364 |
+
return name
|
| 365 |
+
_, e_count, _, _ = compute_derivative_group_features(files, name)
|
| 366 |
+
name_has_ep = bool(_episode_keys_from_text(name))
|
| 367 |
+
is_opaque_slug = not bool(re.search(r"[A-Za-z]{3,}", name))
|
| 368 |
+
hidden_ep_pack = (e_count >= 2 and not name_has_ep)
|
| 369 |
+
if not (is_opaque_slug or hidden_ep_pack):
|
| 370 |
+
return name
|
| 371 |
+
basenames = " ; ".join(
|
| 372 |
+
p.rsplit("/", 1)[-1] for p in summary.split(" ; ")
|
| 373 |
+
)
|
| 374 |
+
return f"{name} |FILES: {basenames}"
|
| 375 |
+
|
| 376 |
+
|
| 377 |
+
def _h64(s: str) -> str:
|
| 378 |
+
return hashlib.blake2b(s.strip().lower().encode("utf-8"), digest_size=8).hexdigest()
|
| 379 |
+
|
| 380 |
+
|
| 381 |
+
def _extract_token_hashes(text: str) -> frozenset[str]:
|
| 382 |
+
toks = [m.group(0).lower() for m in _TOKEN_SCAN_RE.finditer(text)]
|
| 383 |
+
out = {_h64(t) for t in toks}
|
| 384 |
+
for i in range(len(toks) - 1):
|
| 385 |
+
out.add(_h64(f"{toks[i]} {toks[i + 1]}"))
|
| 386 |
+
return frozenset(out)
|
| 387 |
+
|
| 388 |
+
|
| 389 |
+
_VOCAB_CACHE_BY_PATH: Dict[str, Dict[str, int]] = {}
|
| 390 |
+
|
| 391 |
+
|
| 392 |
+
def _get_vocab_ext_map(vocab_path: Optional[Path] = None) -> Dict[str, int]:
|
| 393 |
+
global _VOCAB_EXT_MAP
|
| 394 |
+
if vocab_path is not None:
|
| 395 |
+
vp = Path(vocab_path)
|
| 396 |
+
if vp.exists():
|
| 397 |
+
key = str(vp.resolve())
|
| 398 |
+
if key not in _VOCAB_CACHE_BY_PATH:
|
| 399 |
+
data = json.loads(vp.read_text(encoding="utf-8"))
|
| 400 |
+
_VOCAB_CACHE_BY_PATH[key] = {e: i + 1 for i, e in enumerate(data.get("exts", []))}
|
| 401 |
+
_VOCAB_EXT_MAP = _VOCAB_CACHE_BY_PATH[key]
|
| 402 |
+
return _VOCAB_EXT_MAP
|
| 403 |
+
if vocab_path is None and _VOCAB_EXT_MAP is not None and len(_VOCAB_EXT_MAP) > 64:
|
| 404 |
+
return _VOCAB_EXT_MAP
|
| 405 |
+
vpath = Path(__file__).resolve().with_name("vocab.json")
|
| 406 |
+
if not vpath.exists() and os.environ.get("ARMIGER_MODEL_DIR"):
|
| 407 |
+
vpath = Path(os.environ["ARMIGER_MODEL_DIR"]) / "vocab.json"
|
| 408 |
+
if vpath.exists():
|
| 409 |
+
key = str(vpath.resolve())
|
| 410 |
+
if key not in _VOCAB_CACHE_BY_PATH:
|
| 411 |
+
data = json.loads(vpath.read_text(encoding="utf-8"))
|
| 412 |
+
_VOCAB_CACHE_BY_PATH[key] = {e: i + 1 for i, e in enumerate(data.get("exts", []))}
|
| 413 |
+
_VOCAB_EXT_MAP = _VOCAB_CACHE_BY_PATH[key]
|
| 414 |
+
else:
|
| 415 |
+
_VOCAB_EXT_MAP = {
|
| 416 |
+
"mp3": 2, "mp4": 3, "mkv": 4, "avi": 7, "srt": 10, "zip": 12,
|
| 417 |
+
"rar": 17, "exe": 23, "flac": 24, "png": 25, "dll": 31, "m4b": 53,
|
| 418 |
+
"epub": 81, "cbz": 96, "lnk": 150, "scr": 310,
|
| 419 |
+
}
|
| 420 |
+
return _VOCAB_EXT_MAP
|
| 421 |
+
|
| 422 |
+
|
| 423 |
+
def _hash_ext(ext: str) -> int:
|
| 424 |
+
if not ext:
|
| 425 |
+
return 4096
|
| 426 |
+
return _get_vocab_ext_map().get(ext.lower(), 4096)
|
| 427 |
+
|
| 428 |
+
|
| 429 |
+
def _hash_word(w: str) -> int:
|
| 430 |
+
if not w:
|
| 431 |
+
return 0
|
| 432 |
+
folded = _PREP_DIGIT_RE.sub("0", w.lower())
|
| 433 |
+
return (zlib.crc32(folded.encode("utf-8")) % 32767) + 1
|
| 434 |
+
|
| 435 |
+
|
| 436 |
+
def _log2_size_bucket(sz: int) -> int:
|
| 437 |
+
return min(47, max(int(sz), 1).bit_length() - 1)
|
| 438 |
+
|
| 439 |
+
|
| 440 |
+
def _prep_split_path(path: str) -> Tuple[str, str, str, int]:
|
| 441 |
+
p_norm = path.replace("\\", "/")
|
| 442 |
+
parent, _, base = p_norm.rpartition("/")
|
| 443 |
+
i = base.rfind(".")
|
| 444 |
+
ext = base[i + 1:].lower() if i > 0 else ""
|
| 445 |
+
if len(ext) <= 10 and ext.isascii() and ext.isalnum():
|
| 446 |
+
stem = base[:i]
|
| 447 |
+
else:
|
| 448 |
+
ext, stem = "", base
|
| 449 |
+
return parent.rpartition("/")[2], stem, ext, p_norm.count("/")
|
| 450 |
+
|
| 451 |
+
|
| 452 |
+
def _prep_path_words(parent: str, stem: str) -> List[int]:
|
| 453 |
+
p_words = [_hash_word(w) for w in _PREP_WORD_RE.findall(parent.lower())][-2:] if parent else []
|
| 454 |
+
s_words = [_hash_word(w) for w in _PREP_WORD_RE.findall(stem.lower())][-(4 - len(p_words)):]
|
| 455 |
+
res = p_words + s_words
|
| 456 |
+
while len(res) < 4:
|
| 457 |
+
res.append(0)
|
| 458 |
+
return res[:4]
|
| 459 |
+
|
| 460 |
+
|
| 461 |
+
def encode_torrent_features(
|
| 462 |
+
row: Mapping[str, Any],
|
| 463 |
+
max_name_len: int = 256,
|
| 464 |
+
max_files: int = 64,
|
| 465 |
+
) -> Dict[str, Any]:
|
| 466 |
+
"""Encode a torrent metadata dictionary into the exact ONNX tensor inputs for Armiger v2.1."""
|
| 467 |
+
name = str(row.get("name") or row.get("title") or "")
|
| 468 |
+
|
| 469 |
+
raw_files = row.get("files") or []
|
| 470 |
+
try:
|
| 471 |
+
total_size = max(0, int(row.get("size") or 0))
|
| 472 |
+
except (TypeError, ValueError, OverflowError):
|
| 473 |
+
total_size = 0
|
| 474 |
+
all_items: List[Tuple[str, int]] = _normalize_file_entries(raw_files)
|
| 475 |
+
all_items.sort(key=lambda x: -x[1])
|
| 476 |
+
items: List[Tuple[str, int]] = all_items[:max_files]
|
| 477 |
+
if not items and (row.get("ext") or row.get("extension") or "." in name):
|
| 478 |
+
items.append((name, total_size))
|
| 479 |
+
|
| 480 |
+
# R12: opaque/folder names expose their primary media filenames to the
|
| 481 |
+
# byte/char/word encoders; derivative-group features use the full tree.
|
| 482 |
+
name_text = build_enriched_name_text(name, all_items or items, max_bytes=max_name_len)
|
| 483 |
+
b_name = name_text.encode("utf-8", "replace")
|
| 484 |
+
if len(b_name) > max_name_len:
|
| 485 |
+
b_name = b_name[:max_name_len].decode("utf-8", "ignore").encode("utf-8")
|
| 486 |
+
raw_bytes = list(b_name) if b_name else [32]
|
| 487 |
+
deriv_features = compute_derivative_group_features(all_items or items, name)
|
| 488 |
+
|
| 489 |
+
ext_map = _get_vocab_ext_map()
|
| 490 |
+
exts: List[int] = []
|
| 491 |
+
sizes: List[int] = []
|
| 492 |
+
depths: List[int] = []
|
| 493 |
+
words_list: List[List[int]] = []
|
| 494 |
+
f_exts_str: List[str] = []
|
| 495 |
+
|
| 496 |
+
kbytes = [0.0] * 8
|
| 497 |
+
kcount = [0] * 8
|
| 498 |
+
has_sample = False
|
| 499 |
+
|
| 500 |
+
for path, sz in items:
|
| 501 |
+
parent, stem, ext, depth = _prep_split_path(path)
|
| 502 |
+
f_exts_str.append(ext)
|
| 503 |
+
exts.append(ext_map.get(ext, 4096))
|
| 504 |
+
sizes.append(_log2_size_bucket(sz))
|
| 505 |
+
depths.append(min(depth, 7))
|
| 506 |
+
words_list.append(_prep_path_words(parent, stem))
|
| 507 |
+
k = _PREP_EXT_TO_KIND.get(ext)
|
| 508 |
+
if k is None:
|
| 509 |
+
k = 2 if _PREP_SPLIT_ARC_RE.fullmatch(ext) else 7
|
| 510 |
+
kbytes[k] += max(sz, 0)
|
| 511 |
+
kcount[k] += 1
|
| 512 |
+
has_sample = has_sample or (_PREP_SAMPLE_RE.search(path) is not None)
|
| 513 |
+
|
| 514 |
+
tot_k = sum(kbytes)
|
| 515 |
+
weights = kbytes if tot_k > 0 else [float(c) for c in kcount]
|
| 516 |
+
wsum = sum(weights)
|
| 517 |
+
try:
|
| 518 |
+
raw_fc = max(0, int(row.get("file_count") or 0))
|
| 519 |
+
except (TypeError, ValueError, OverflowError):
|
| 520 |
+
raw_fc = 0
|
| 521 |
+
count = max(raw_fc, len(all_items), len(items))
|
| 522 |
+
biggest = max((max(s, 0) for _, s in items), default=0)
|
| 523 |
+
max_share = (biggest / tot_k if tot_k > 0 else (1.0 / len(items) if items else 0.0))
|
| 524 |
+
|
| 525 |
+
base_stats = [
|
| 526 |
+
math.log1p(max(count, 0)),
|
| 527 |
+
math.log1p(max(total_size if total_size > 0 else int(tot_k), 0)),
|
| 528 |
+
]
|
| 529 |
+
base_stats += [w / wsum if wsum > 0 else 0.0 for w in weights]
|
| 530 |
+
base_stats += [max_share, float(has_sample)]
|
| 531 |
+
|
| 532 |
+
side = [0.0] * SIDE_DIM
|
| 533 |
+
n_f = float(max(1, len(items)))
|
| 534 |
+
|
| 535 |
+
m_grp = re.match(r"^\[([^\[\]]{2,30})\]", name)
|
| 536 |
+
grp_lc = m_grp.group(1).strip().lower() if m_grp else ""
|
| 537 |
+
grp_h = _h64(grp_lc) if grp_lc else ""
|
| 538 |
+
title_tok_h = _extract_token_hashes(name)
|
| 539 |
+
|
| 540 |
+
has_sub_grp = (grp_h in _SUB_ONLY_GRP_H) or bool(title_tok_h & _SUB_ONLY_GRP_H)
|
| 541 |
+
has_dub_grp = (grp_h in _DUAL_DUB_GRP_H) or bool(title_tok_h & _DUAL_DUB_GRP_H)
|
| 542 |
+
has_game_grp = bool(title_tok_h & _GAME_GRP_H)
|
| 543 |
+
has_sw_grp = bool(title_tok_h & _SW_GRP_H)
|
| 544 |
+
has_book_grp = bool(title_tok_h & _BOOK_COMIC_GRP_H)
|
| 545 |
+
|
| 546 |
+
if items:
|
| 547 |
+
ebk_set = {"epub", "mobi", "azw3", "azw", "djvu", "fb2", "lit", "iba"}
|
| 548 |
+
pdf_set = {"pdf"}
|
| 549 |
+
m4b_set = {"m4b", "aa", "aax"}
|
| 550 |
+
com_set = {"cbz", "cbr", "cb7", "cbt"}
|
| 551 |
+
con_set = {"nsp", "xci", "xcz", "nsz", "wbfs", "rvz", "gcz", "cia", "3ds", "vpk", "ps4", "fpkg"}
|
| 552 |
+
app_set = {"dmg", "pkg", "apk", "ipa", "deb", "rpm", "appimage"}
|
| 553 |
+
exe_dll_set = {"exe", "msi", "bat", "cmd", "sh", "com", "dll"}
|
| 554 |
+
iso_set = {"iso", "bin", "cue", "mdf", "mds", "nrg", "img"}
|
| 555 |
+
arc_set = {"rar", "zip", "7z", "tar", "gz", "xz", "bz2", "001"}
|
| 556 |
+
htm_set = {"html", "htm", "url", "mht", "torrent"}
|
| 557 |
+
scr_set = {"scr", "lnk", "vbs", "hta", "pif", "ps1", "wsf", "reg"}
|
| 558 |
+
vid_set = {"mkv", "mp4", "avi", "ts", "m2ts", "wmv", "mov", "webm", "flv", "mpg", "mpeg", "m4v", "vob", "ogv", "rmvb", "3gp", "asf", "divx"}
|
| 559 |
+
bd_game_benign = {"vcdiff", "apk", "bdmv", "clpi", "mpls", "ifo", "bup", "vob", "npa", "vpk", "pck", "umd", "pak", "xp3", "rgss3a", "wolf", "nsp", "xci", "wbfs"}
|
| 560 |
+
|
| 561 |
+
side[0] = sum(1 for e in f_exts_str if e in ebk_set) / n_f
|
| 562 |
+
side[1] = sum(1 for e in f_exts_str if e in pdf_set) / n_f
|
| 563 |
+
side[2] = sum(1 for e in f_exts_str if e in m4b_set) / n_f
|
| 564 |
+
side[3] = sum(1 for e in f_exts_str if e in com_set) / n_f
|
| 565 |
+
side[4] = sum(1 for e in f_exts_str if e in con_set) / n_f
|
| 566 |
+
side[5] = sum(1 for e in f_exts_str if e in app_set) / n_f
|
| 567 |
+
|
| 568 |
+
vc = bool(_SIDE_VID_CLAIM_RE.search(name) or _SIDE_ANIME_KW_RE.search(name) or has_sub_grp or has_dub_grp)
|
| 569 |
+
has_any_vid_inf = any(e in vid_set for e in f_exts_str)
|
| 570 |
+
has_renpy_structure = (
|
| 571 |
+
not vc
|
| 572 |
+
and not has_any_vid_inf
|
| 573 |
+
and any(
|
| 574 |
+
p_i.replace("\\", "/").lower().endswith((".rpa", ".rpyb")) and int(sz_i or 0) >= 65536
|
| 575 |
+
for p_i, sz_i in items
|
| 576 |
+
)
|
| 577 |
+
)
|
| 578 |
+
renpy_say_re = re.compile(r"(?i)(?:^|/)lib/(?:py\d+-)?windows-[^/]+/say\.vbs$")
|
| 579 |
+
|
| 580 |
+
n_real_exec_dll = 0
|
| 581 |
+
has_real_exe = False
|
| 582 |
+
has_dll = False
|
| 583 |
+
has_scr_trap = False
|
| 584 |
+
has_oversized_lnk = False
|
| 585 |
+
has_root_launcher = False
|
| 586 |
+
has_unconditional_trap_ext = False
|
| 587 |
+
has_any_scr_ext = False
|
| 588 |
+
vid_depths: List[int] = []
|
| 589 |
+
|
| 590 |
+
for (p_i, _), e_str, s_b, d_i, w_i in zip(items, f_exts_str, sizes, depths, words_list):
|
| 591 |
+
p_posix = p_i.replace("\\", "/")
|
| 592 |
+
if has_renpy_structure and renpy_say_re.search(p_posix):
|
| 593 |
+
continue
|
| 594 |
+
if e_str in scr_set:
|
| 595 |
+
has_any_scr_ext = True
|
| 596 |
+
if e_str in {"scr", "vbs", "vbe", "hta", "pif", "wsf", "wsh", "cpl"}:
|
| 597 |
+
has_unconditional_trap_ext = True
|
| 598 |
+
p_tok_h = _extract_token_hashes(p_i)
|
| 599 |
+
is_benign_stub = (e_str == "exe") and (s_b < 15 or 16570 in w_i or bool(p_tok_h & _BENIGN_README_EXE_H))
|
| 600 |
+
if e_str in exe_dll_set and not is_benign_stub:
|
| 601 |
+
n_real_exec_dll += 1
|
| 602 |
+
if e_str == "exe" and not is_benign_stub and s_b >= 15:
|
| 603 |
+
has_real_exe = True
|
| 604 |
+
if d_i == 0:
|
| 605 |
+
has_root_launcher = True
|
| 606 |
+
elif e_str == "dll":
|
| 607 |
+
has_dll = True
|
| 608 |
+
if e_str in {"scr", "lnk", "hta", "vbs", "pif", "ps1", "wsf", "cmd", "com", "one", "xll", "xlam"} and s_b >= 12:
|
| 609 |
+
has_scr_trap = True
|
| 610 |
+
if d_i == 0:
|
| 611 |
+
has_root_launcher = True
|
| 612 |
+
if e_str == "lnk" and s_b >= 15:
|
| 613 |
+
has_oversized_lnk = True
|
| 614 |
+
if e_str in vid_set:
|
| 615 |
+
vid_depths.append(d_i)
|
| 616 |
+
|
| 617 |
+
side[6] = n_real_exec_dll / n_f
|
| 618 |
+
side[7] = sum(1 for e in f_exts_str if e in iso_set) / n_f
|
| 619 |
+
side[8] = sum(1 for e in f_exts_str if e in arc_set) / n_f
|
| 620 |
+
side[9] = sum(1 for e in f_exts_str if e in htm_set) / n_f
|
| 621 |
+
side[10] = 1.0 if has_any_scr_ext else 0.0
|
| 622 |
+
|
| 623 |
+
is_sw_or_game = bool(
|
| 624 |
+
has_game_grp
|
| 625 |
+
or has_sw_grp
|
| 626 |
+
or (not vc and _SIDE_SW_APP_TITLE_RE.search(name))
|
| 627 |
+
or any(e in bd_game_benign for e in f_exts_str)
|
| 628 |
+
)
|
| 629 |
+
vid_share = base_stats[2]
|
| 630 |
+
flagged = has_unconditional_trap_ext
|
| 631 |
+
if not is_sw_or_game:
|
| 632 |
+
if vc and has_real_exe and has_dll:
|
| 633 |
+
flagged = True
|
| 634 |
+
elif vc and len(vid_depths) >= 1 and (has_real_exe or has_root_launcher or has_scr_trap):
|
| 635 |
+
flagged = True
|
| 636 |
+
elif (vc or len(vid_depths) > 0) and has_oversized_lnk:
|
| 637 |
+
flagged = True
|
| 638 |
+
elif vc and vid_share == 0.0 and (has_real_exe or has_scr_trap):
|
| 639 |
+
flagged = True
|
| 640 |
+
elif len(items) <= 2 and any(e in ("zip", "zipx", "rar", "7z") for e in f_exts_str) and vid_share == 0.0:
|
| 641 |
+
if _SIDE_EP_ZIP_RE.search(name) and not _SIDE_BENIGN_ZIP_RE.search(name):
|
| 642 |
+
flagged = True
|
| 643 |
+
elif len(items) <= 2 and any(e in ("iso", "img", "vhd", "vhdx", "udf") for e in f_exts_str) and vid_share == 0.0:
|
| 644 |
+
name_stem = re.sub(r"\.(?:iso|img|vhd|vhdx|udf|dmg)$", "", name.strip(), flags=re.IGNORECASE)
|
| 645 |
+
if _SIDE_EP_ZIP_RE.search(name_stem) and not re.search(r"(?i)\b(?:dvd|bd25|bd50|remux|untouched|pal|ntsc|iso)\b", name_stem):
|
| 646 |
+
flagged = True
|
| 647 |
+
|
| 648 |
+
if flagged or _SIDE_VPN_TRAP_RE.search(name) or (vc and vid_share == 0.0 and (side[6] > 0 or side[9] > 0 or side[10] > 0)):
|
| 649 |
+
side[11] = 1.0
|
| 650 |
+
if vc and vid_share > 0 and 0 < base_stats[1] < 14.22:
|
| 651 |
+
side[12] = 1.0
|
| 652 |
+
|
| 653 |
+
# R12: strip negated subtitle clauses ("no english subs", ...) before the
|
| 654 |
+
# side[13] subtitle cue; previously side[13] was never populated at all.
|
| 655 |
+
if _SUB_CUE_RE.search(_NEGATED_SUB_RE.sub(" ", name)):
|
| 656 |
+
side[13] = 1.0
|
| 657 |
+
|
| 658 |
+
ag = bool(_SIDE_ANIME_KW_RE.search(name)) or has_sub_grp or has_dub_grp
|
| 659 |
+
if ag:
|
| 660 |
+
side[15] = 1.0
|
| 661 |
+
clean_audio_name = _SIDE_DECEPTIVE_SUB_STRIP_RE.sub(" ", name)
|
| 662 |
+
clean_tok_h = _extract_token_hashes(clean_audio_name)
|
| 663 |
+
ad = bool(_SIDE_ANIME_DUB_KW_RE.search(clean_audio_name)) or (grp_h in _DUAL_DUB_GRP_H) or bool(clean_tok_h & _DUAL_DUB_GRP_H)
|
| 664 |
+
if ad:
|
| 665 |
+
side[16] = 1.0
|
| 666 |
+
if (grp_h in _SUB_ONLY_GRP_H) and not ad:
|
| 667 |
+
side[14] = 1.0
|
| 668 |
+
if has_game_grp:
|
| 669 |
+
side[17] = 1.0
|
| 670 |
+
if has_sw_grp and not _SIDE_CJK_RE.search(name):
|
| 671 |
+
side[18] = 1.0
|
| 672 |
+
if _SIDE_COURSE_RE.search(name) and not ag:
|
| 673 |
+
side[19] = 1.0
|
| 674 |
+
if _SIDE_BOOK_COMIC_KW_RE.search(name) or has_book_grp:
|
| 675 |
+
side[20] = 1.0
|
| 676 |
+
|
| 677 |
+
lang_str = (row.get("language") or "").strip().lower()
|
| 678 |
+
if lang_str in ("ja", "ru", "fr", "es", "de", "it", "pt", "zh", "ko", "hi", "pl", "nl", "tr", "ar") or _SIDE_CJK_RE.search(name) or ((grp_h in _SUB_ONLY_GRP_H) and not ad):
|
| 679 |
+
side[22] = 1.0
|
| 680 |
+
elif lang_str in ("en", "eng", "english") or ad:
|
| 681 |
+
side[21] = 1.0
|
| 682 |
+
else:
|
| 683 |
+
side[23] = 1.0
|
| 684 |
+
|
| 685 |
+
# R12 derivative-group features (raw counts stay in `deriv_features`;
|
| 686 |
+
# the network sees bounded log-normalized U/E, binary P, raw s_max).
|
| 687 |
+
_u, _e, _p, _smax = deriv_features
|
| 688 |
+
side[24] = min(1.0, math.log1p(_u) / _DERIV_LOG_NORM)
|
| 689 |
+
side[25] = min(1.0, math.log1p(_e) / _DERIV_LOG_NORM)
|
| 690 |
+
side[26] = float(_p)
|
| 691 |
+
side[27] = float(_smax)
|
| 692 |
+
|
| 693 |
+
return {
|
| 694 |
+
"name_bytes": raw_bytes,
|
| 695 |
+
"ext": exts if exts else [0],
|
| 696 |
+
"size": sizes if sizes else [0],
|
| 697 |
+
"depth": depths if depths else [0],
|
| 698 |
+
"words": words_list if words_list else [[0, 0, 0, 0]],
|
| 699 |
+
"stats": base_stats + side,
|
| 700 |
+
"deriv_features": deriv_features,
|
| 701 |
+
"name_text": name_text,
|
| 702 |
+
}
|
| 703 |
+
|
| 704 |
+
|
| 705 |
+
def build_onnx_batch_inputs(
|
| 706 |
+
rows: Sequence[Mapping[str, Any]],
|
| 707 |
+
max_name_len: int = 256,
|
| 708 |
+
max_files: int = 64,
|
| 709 |
+
stats_dim: Optional[int] = None,
|
| 710 |
+
) -> Dict[str, np.ndarray]:
|
| 711 |
+
"""Collate a sequence of torrent dicts into padded NumPy arrays for ONNX Runtime.
|
| 712 |
+
|
| 713 |
+
`stats_dim` defaults to `STATS_DIM` (40). Pass a legacy checkpoint's width
|
| 714 |
+
(e.g. `36`) to emit the legacy `stats` prefix during the R12 transition;
|
| 715 |
+
`ArmigerStandaloneONNX.predict_batch` does this automatically from the
|
| 716 |
+
loaded session's `stats` input shape.
|
| 717 |
+
"""
|
| 718 |
+
encoded = [encode_torrent_features(r, max_name_len=max_name_len, max_files=max_files) for r in rows]
|
| 719 |
+
bsz = len(encoded)
|
| 720 |
+
max_l = max(len(e["name_bytes"]) for e in encoded)
|
| 721 |
+
max_f = max(len(e["ext"]) for e in encoded)
|
| 722 |
+
width = STATS_DIM if stats_dim is None else int(stats_dim)
|
| 723 |
+
|
| 724 |
+
name_b = np.zeros((bsz, max_l), dtype=np.int64)
|
| 725 |
+
name_l = np.zeros((bsz,), dtype=np.int64)
|
| 726 |
+
ext_t = np.zeros((bsz, max_f), dtype=np.int32)
|
| 727 |
+
size_t = np.zeros((bsz, max_f), dtype=np.int64)
|
| 728 |
+
depth_t = np.zeros((bsz, max_f), dtype=np.int64)
|
| 729 |
+
words_t = np.zeros((bsz, max_f, 4), dtype=np.int32)
|
| 730 |
+
stats_t = np.zeros((bsz, width), dtype=np.float32)
|
| 731 |
+
|
| 732 |
+
for i, e in enumerate(encoded):
|
| 733 |
+
lb = len(e["name_bytes"])
|
| 734 |
+
lf = len(e["ext"])
|
| 735 |
+
name_b[i, :lb] = e["name_bytes"]
|
| 736 |
+
name_l[i] = lb
|
| 737 |
+
ext_t[i, :lf] = e["ext"]
|
| 738 |
+
size_t[i, :lf] = e["size"]
|
| 739 |
+
depth_t[i, :lf] = e["depth"]
|
| 740 |
+
words_t[i, :lf, :] = e["words"]
|
| 741 |
+
stats_t[i, : min(width, len(e["stats"]))] = e["stats"][:width]
|
| 742 |
+
|
| 743 |
+
return {
|
| 744 |
+
"name_bytes": name_b,
|
| 745 |
+
"name_len": name_l,
|
| 746 |
+
"ext": ext_t,
|
| 747 |
+
"size": size_t,
|
| 748 |
+
"depth": depth_t,
|
| 749 |
+
"words": words_t,
|
| 750 |
+
"stats": stats_t,
|
| 751 |
+
}
|
| 752 |
+
|
| 753 |
+
|
| 754 |
+
def _softmax(logits: np.ndarray) -> np.ndarray:
|
| 755 |
+
shifted = logits - np.max(logits, axis=-1, keepdims=True)
|
| 756 |
+
exps = np.exp(shifted)
|
| 757 |
+
return exps / np.sum(exps, axis=-1, keepdims=True)
|
| 758 |
+
|
| 759 |
+
|
| 760 |
+
_DEFAULT_MONDRIAN_THRESHOLDS: Dict[str, float] = {"adult": 0.5451, "junk": 0.1010, "owner_scope": 0.4770}
|
| 761 |
+
|
| 762 |
+
|
| 763 |
+
def _mondrian_value(v: Any) -> Optional[float]:
|
| 764 |
+
return float(v) if isinstance(v, (int, float)) and not isinstance(v, bool) and 0.0 < float(v) < 1.0 else None
|
| 765 |
+
|
| 766 |
+
|
| 767 |
+
def load_mondrian_thresholds(calibration_path: str | Path | None) -> Tuple[Dict[str, float], Dict[str, Dict[str, float]]]:
|
| 768 |
+
"""(global gate thresholds, per-gate strata) from `calibration.json`; defaults only where nothing is fitted.
|
| 769 |
+
|
| 770 |
+
Same precedence as `calibration_and_dht.ArmigerCalibrator`: `mondrian_thresholds[gate]` (a dict replaces the
|
| 771 |
+
legacy block entirely), else legacy `mondrian_conformal_thresholds[gate]["threshold" | "threshold_*"]`;
|
| 772 |
+
strata from `mondrian_stratified_thresholds[gate]["strata"][<category>]` (fitted per predicted category).
|
| 773 |
+
"""
|
| 774 |
+
thresholds = dict(_DEFAULT_MONDRIAN_THRESHOLDS)
|
| 775 |
+
strata: Dict[str, Dict[str, float]] = {}
|
| 776 |
+
try:
|
| 777 |
+
cfg = json.loads(Path(calibration_path).read_text(encoding="utf-8")) if calibration_path else {}
|
| 778 |
+
except (OSError, ValueError):
|
| 779 |
+
cfg = {}
|
| 780 |
+
if not isinstance(cfg, dict):
|
| 781 |
+
return thresholds, strata
|
| 782 |
+
flat = cfg.get("mondrian_thresholds")
|
| 783 |
+
if not isinstance(flat, dict):
|
| 784 |
+
legacy = cfg.get("mondrian_conformal_thresholds")
|
| 785 |
+
flat = {}
|
| 786 |
+
for gate, block in (legacy.items() if isinstance(legacy, dict) else ()):
|
| 787 |
+
if isinstance(block, dict):
|
| 788 |
+
keys = ["threshold"] + sorted(k for k in block if str(k).startswith("threshold_"))
|
| 789 |
+
flat[gate] = next((block[k] for k in keys if _mondrian_value(block.get(k)) is not None), None)
|
| 790 |
+
else:
|
| 791 |
+
flat[gate] = block
|
| 792 |
+
for gate in thresholds:
|
| 793 |
+
v = _mondrian_value(flat.get(gate, flat.get("scope") if gate == "owner_scope" else None))
|
| 794 |
+
if v is not None:
|
| 795 |
+
thresholds[gate] = v
|
| 796 |
+
raw_strata = cfg.get("mondrian_stratified_thresholds")
|
| 797 |
+
for gate, block in (raw_strata.items() if isinstance(raw_strata, dict) else ()):
|
| 798 |
+
per = block.get("strata") if isinstance(block, dict) else None
|
| 799 |
+
if isinstance(per, dict):
|
| 800 |
+
g_key = "owner_scope" if gate == "scope" else str(gate)
|
| 801 |
+
strata[g_key] = {str(k): t for k, v in per.items() if (t := _mondrian_value(v)) is not None}
|
| 802 |
+
return thresholds, strata
|
| 803 |
+
|
| 804 |
+
|
| 805 |
+
class ArmigerStandaloneONNX:
|
| 806 |
+
"""Self-contained ONNX Runtime classifier for `NorviTech/armiger` and `NorviTech/armiger-ultra`."""
|
| 807 |
+
|
| 808 |
+
def __init__(
|
| 809 |
+
self,
|
| 810 |
+
onnx_path: Optional[str | Path] = None,
|
| 811 |
+
vocab_path: Optional[str | Path] = None,
|
| 812 |
+
calibration_path: Optional[str | Path] = None,
|
| 813 |
+
) -> None:
|
| 814 |
+
import onnxruntime as ort
|
| 815 |
+
|
| 816 |
+
base = Path(__file__).resolve().parent
|
| 817 |
+
if onnx_path is None:
|
| 818 |
+
for cand in ("armiger.onnx", "armiger.int8.onnx", "armiger_ultra.fp32.onnx"):
|
| 819 |
+
if (base / cand).exists():
|
| 820 |
+
onnx_path = base / cand
|
| 821 |
+
break
|
| 822 |
+
if onnx_path is None:
|
| 823 |
+
raise FileNotFoundError("Could not find armiger.onnx or armiger_ultra.fp32.onnx")
|
| 824 |
+
self.onnx_path = Path(onnx_path)
|
| 825 |
+
default_vocab = base / "vocab.json"
|
| 826 |
+
if not default_vocab.exists() and self.onnx_path.with_name("vocab.json").exists():
|
| 827 |
+
default_vocab = self.onnx_path.with_name("vocab.json")
|
| 828 |
+
self.vocab_path = Path(vocab_path) if vocab_path else default_vocab
|
| 829 |
+
_get_vocab_ext_map(self.vocab_path)
|
| 830 |
+
default_cal = base / "calibration.json"
|
| 831 |
+
if not default_cal.exists() and self.onnx_path.with_name("calibration.json").exists():
|
| 832 |
+
default_cal = self.onnx_path.with_name("calibration.json")
|
| 833 |
+
self.calibration_path = Path(calibration_path) if calibration_path else default_cal
|
| 834 |
+
self.mondrian_thresholds, self.mondrian_strata = load_mondrian_thresholds(self.calibration_path)
|
| 835 |
+
self.session = ort.InferenceSession(str(self.onnx_path), providers=["CPUExecutionProvider"])
|
| 836 |
+
self.head_names = [o.name for o in self.session.get_outputs()]
|
| 837 |
+
|
| 838 |
+
@staticmethod
|
| 839 |
+
def _apply_v3_post_neural_guard(row: Mapping[str, Any], item_res: Dict[str, Any]) -> Dict[str, Any]:
|
| 840 |
+
return _apply_v3_post_neural_guard(row, item_res)
|
| 841 |
+
|
| 842 |
+
def mondrian_threshold(self, gate: str, stratum: Optional[str] = None) -> float:
|
| 843 |
+
"""Stratified Mondrian threshold for `gate` (per predicted category when fitted, else global)."""
|
| 844 |
+
per = self.mondrian_strata.get(gate, {})
|
| 845 |
+
if stratum is not None and str(stratum) in per:
|
| 846 |
+
return per[str(stratum)]
|
| 847 |
+
return self.mondrian_thresholds[gate]
|
| 848 |
+
|
| 849 |
+
def _session_stats_dim(self) -> int:
|
| 850 |
+
"""`stats` width expected by the loaded checkpoint (legacy 36 or R12 40)."""
|
| 851 |
+
try:
|
| 852 |
+
for inp in self.session.get_inputs():
|
| 853 |
+
if inp.name == "stats" and len(inp.shape) == 2 and isinstance(inp.shape[1], int):
|
| 854 |
+
return int(inp.shape[1])
|
| 855 |
+
except Exception:
|
| 856 |
+
pass
|
| 857 |
+
return STATS_DIM
|
| 858 |
+
|
| 859 |
+
def predict_batch(self, rows: Sequence[Mapping[str, Any]]) -> List[Dict[str, Any]]:
|
| 860 |
+
feed = build_onnx_batch_inputs(rows, stats_dim=self._session_stats_dim())
|
| 861 |
+
raw_outs = self.session.run(self.head_names, feed)
|
| 862 |
+
logits_by_head = dict(zip(self.head_names, raw_outs))
|
| 863 |
+
results: List[Dict[str, Any]] = []
|
| 864 |
+
for i in range(len(rows)):
|
| 865 |
+
item_res: Dict[str, Any] = {"heads": {}, "probabilities": {}}
|
| 866 |
+
for h, arr in logits_by_head.items():
|
| 867 |
+
vec = arr[i].astype(np.float64)
|
| 868 |
+
if vec.ndim == 2:
|
| 869 |
+
# 3D batch tensor (e.g. span_bio [B, 256, 7]) -> 2D per-item [256, 7]
|
| 870 |
+
shifted = vec - np.max(vec, axis=-1, keepdims=True)
|
| 871 |
+
exps = np.exp(shifted)
|
| 872 |
+
probs_2d = exps / np.maximum(1e-12, np.sum(exps, axis=-1, keepdims=True))
|
| 873 |
+
raw_nm = str(rows[i].get("name") or rows[i].get("title") or "")
|
| 874 |
+
try:
|
| 875 |
+
try:
|
| 876 |
+
from byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans
|
| 877 |
+
except ImportError:
|
| 878 |
+
from byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans
|
| 879 |
+
raw_b = raw_nm.encode("utf-8", errors="replace")[: vec.shape[0]]
|
| 880 |
+
tag_ids = [int(x) for x in decode_bio_logits_viterbi(vec, raw_b).tolist()]
|
| 881 |
+
except Exception:
|
| 882 |
+
tag_ids = [int(x) for x in np.argmax(probs_2d, axis=-1).tolist()]
|
| 883 |
+
classes = NEURAL_HEAD_CLASSES.get(
|
| 884 |
+
h,
|
| 885 |
+
list(SPAN_BIO_CLASSES),
|
| 886 |
+
)
|
| 887 |
+
tag_labels = [classes[idx_c] if idx_c < len(classes) else str(idx_c) for idx_c in tag_ids]
|
| 888 |
+
item_res["heads"][h] = tag_labels
|
| 889 |
+
mean_probs = np.mean(probs_2d, axis=0)
|
| 890 |
+
item_res["probabilities"][h] = {
|
| 891 |
+
classes[k]: round(float(mean_probs[k]), 6) for k in range(min(len(classes), len(mean_probs)))
|
| 892 |
+
}
|
| 893 |
+
try:
|
| 894 |
+
item_res["span_bio"] = decode_bio_tags_to_spans(raw_nm, tag_ids)
|
| 895 |
+
except Exception:
|
| 896 |
+
item_res["span_bio"] = {"tag_ids": tag_ids, "tags": tag_labels}
|
| 897 |
+
continue
|
| 898 |
+
probs = _softmax(vec)
|
| 899 |
+
classes = NEURAL_HEAD_CLASSES.get(h, [str(k) for k in range(len(probs))])
|
| 900 |
+
idx_c = int(np.argmax(probs))
|
| 901 |
+
item_res["heads"][h] = classes[idx_c] if idx_c < len(classes) else str(idx_c)
|
| 902 |
+
item_res["probabilities"][h] = {
|
| 903 |
+
classes[k]: round(float(probs[k]), 6) for k in range(min(len(classes), len(probs)))
|
| 904 |
+
}
|
| 905 |
+
self._apply_v3_post_neural_guard(rows[i], item_res)
|
| 906 |
+
p_junk = item_res["probabilities"].get("junk", {}).get("junk", 0.0)
|
| 907 |
+
p_adult = item_res["probabilities"].get("adult", {}).get("adult", 0.0)
|
| 908 |
+
# Calibrated excluded probability is kept as-is: the conformal gate (not the argmax label) decides.
|
| 909 |
+
p_excl = item_res["probabilities"].get("owner_scope", {}).get("excluded", 0.0)
|
| 910 |
+
stratum = item_res["heads"].get("category")
|
| 911 |
+
thr = {g: self.mondrian_threshold(g, stratum) for g in ("adult", "junk", "owner_scope")}
|
| 912 |
+
item_res["p_junk"] = p_junk
|
| 913 |
+
item_res["mondrian_gates"] = {
|
| 914 |
+
"adult_flagged": bool(p_adult >= thr["adult"]),
|
| 915 |
+
"junk_flagged": bool(p_junk >= thr["junk"]),
|
| 916 |
+
"scope_excluded": bool(p_excl >= thr["owner_scope"]),
|
| 917 |
+
"passed_all": bool(p_adult < thr["adult"] and p_junk < thr["junk"] and p_excl < thr["owner_scope"]),
|
| 918 |
+
"thresholds": thr,
|
| 919 |
+
"owner_scope_stratum": stratum,
|
| 920 |
+
}
|
| 921 |
+
results.append(item_res)
|
| 922 |
+
return results
|
| 923 |
+
|
| 924 |
+
def predict(self, row: Mapping[str, Any]) -> Dict[str, Any]:
|
| 925 |
+
return self.predict_batch([row])[0]
|
| 926 |
+
|
| 927 |
+
|
| 928 |
+
ArmigerONNXClassifier = ArmigerStandaloneONNX
|
| 929 |
+
|
| 930 |
+
|
| 931 |
+
def _apply_v3_post_neural_guard(row, result):
|
| 932 |
+
return result
|
armiger.int8.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c0e96936038988c7465b7fe041c4dc3c92df5a8cb4f56c2666271cd4d3e79420
|
| 3 |
+
size 17599636
|
armiger.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c0e96936038988c7465b7fe041c4dc3c92df5a8cb4f56c2666271cd4d3e79420
|
| 3 |
+
size 17599636
|
armiger_dht_rsi.py
CHANGED
|
@@ -45,14 +45,14 @@ from cryptography.hazmat.primitives.kdf.hkdf import HKDF
|
|
| 45 |
|
| 46 |
|
| 47 |
import sys
|
| 48 |
-
_THIS_DIR = pathlib.Path(__file__).resolve().parent
|
| 49 |
-
if str(_THIS_DIR) not in sys.path:
|
| 50 |
-
sys.path.insert(0, str(_THIS_DIR))
|
| 51 |
|
| 52 |
from dht_rsi_models_and_privacy import (
|
|
|
|
| 53 |
AnonymityGuard,
|
| 54 |
AnonymityViolationError,
|
| 55 |
AnonymizedTelemetryRecord,
|
|
|
|
|
|
|
| 56 |
BlindedTitleStem,
|
| 57 |
BundleFormat,
|
| 58 |
CanaryItem,
|
|
@@ -62,6 +62,9 @@ from dht_rsi_models_and_privacy import (
|
|
| 62 |
ContinuousAggregationResult,
|
| 63 |
ContinuousDeltaSubmission,
|
| 64 |
DHTRSIMode,
|
|
|
|
|
|
|
|
|
|
| 65 |
EncryptedTelemetryBundle,
|
| 66 |
MediaCategory,
|
| 67 |
PeerAttestation,
|
|
@@ -73,6 +76,7 @@ from dht_rsi_models_and_privacy import (
|
|
| 73 |
VerifiedObservation,
|
| 74 |
ZeroCueCatalogDelta,
|
| 75 |
_bep44_canonical_sign_buffer,
|
|
|
|
| 76 |
pure_laplace_noise,
|
| 77 |
resolve_opt_in_sharing,
|
| 78 |
)
|
|
@@ -91,6 +95,7 @@ from dht_rsi_crypto_and_adapters import (
|
|
| 91 |
_OptInAdapterBase,
|
| 92 |
)
|
| 93 |
from dht_rsi_verification import (
|
|
|
|
| 94 |
AuthoritativeASNVerifier,
|
| 95 |
PhysicalGroundTruthVerifier,
|
| 96 |
SafeFormatValidator,
|
|
@@ -136,9 +141,13 @@ class ArmigerDHTRSIEngine:
|
|
| 136 |
dp_sigma: float = 0.05,
|
| 137 |
max_seen_delta_digests: int = 4096,
|
| 138 |
max_tracked_senders: int = 1024,
|
|
|
|
| 139 |
delta_freshness_window_seconds: int = 14 * 86400,
|
| 140 |
max_telemetry_queue_size: int = 1024,
|
| 141 |
max_item_encryption_retries: int = 3,
|
|
|
|
|
|
|
|
|
|
| 142 |
) -> None:
|
| 143 |
self.opt_in_sharing: bool = resolve_opt_in_sharing(opt_in_sharing)
|
| 144 |
if mode is None:
|
|
@@ -151,9 +160,13 @@ class ArmigerDHTRSIEngine:
|
|
| 151 |
|
| 152 |
self.mode = DHTRSIMode(resolved_mode)
|
| 153 |
self.current_seq = int(current_seq)
|
|
|
|
| 154 |
self._telemetry_seq = int(current_seq)
|
| 155 |
self.trusted_signers = trusted_signers or {}
|
| 156 |
self.trusted_pq_signers = trusted_pq_signers or {}
|
|
|
|
|
|
|
|
|
|
| 157 |
self.require_pq = bool(require_pq)
|
| 158 |
self.threshold_m = int(threshold_m)
|
| 159 |
self.min_distinct_asns = int(min_distinct_asns)
|
|
@@ -169,7 +182,12 @@ class ArmigerDHTRSIEngine:
|
|
| 169 |
self.max_item_encryption_retries = max(int(max_item_encryption_retries), 1)
|
| 170 |
self._seen_delta_digests: Dict[str, Tuple[int, Tuple[str, bytes]]] = {}
|
| 171 |
self._inbound_sender_seq: Dict[Tuple[str, bytes], Tuple[int, int]] = {}
|
| 172 |
-
self.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 173 |
|
| 174 |
self.verifier = PhysicalGroundTruthVerifier()
|
| 175 |
self.anonymizer = TelemetryAnonymizer(
|
|
@@ -188,6 +206,8 @@ class ArmigerDHTRSIEngine:
|
|
| 188 |
self._outbound_bundles_shared: int = 0
|
| 189 |
self._outbound_packets_blocked_opt_out: int = 0
|
| 190 |
self._outbound_bytes_shared: int = 0
|
|
|
|
|
|
|
| 191 |
self._telemetry_encryption_errors: int = 0
|
| 192 |
self._telemetry_dropped_backlog: int = 0
|
| 193 |
self._last_telemetry_error: str = ""
|
|
@@ -259,6 +279,8 @@ class ArmigerDHTRSIEngine:
|
|
| 259 |
"outbound_bundles_shared": int(self._outbound_bundles_shared),
|
| 260 |
"outbound_packets_blocked_opt_out": int(self._outbound_packets_blocked_opt_out),
|
| 261 |
"outbound_bytes_shared": int(self._outbound_bytes_shared),
|
|
|
|
|
|
|
| 262 |
"ephemeral_identity_enabled": bool(self.anonymity_guard.ephemeral_identity),
|
| 263 |
"ephemeral_pubkey_hex": self.anonymity_guard.current_pubkey_hex() if self.opt_in_sharing else None,
|
| 264 |
"ephemeral_rotations": int(self.anonymity_guard.rotation_count),
|
|
@@ -371,28 +393,82 @@ class ArmigerDHTRSIEngine:
|
|
| 371 |
for sk in expired_senders:
|
| 372 |
self._inbound_sender_seq.pop(sk, None)
|
| 373 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 374 |
def ingest_signed_delta_bundle(
|
| 375 |
self,
|
| 376 |
signed_bundle: Mapping[str, Any],
|
|
|
|
|
|
|
| 377 |
) -> Tuple[bool, str, Optional[Dict[str, Any]]]:
|
| 378 |
"""Verify Ed25519 signature and AnonymityGuard schema of an inbound delta bundle (works in pull_only mode)."""
|
| 379 |
if self.mode == DHTRSIMode.OFF:
|
| 380 |
return False, "plugin_mode_off", None
|
| 381 |
|
|
|
|
|
|
|
|
|
|
| 382 |
try:
|
| 383 |
v_raw = signed_bundle["v"]
|
| 384 |
-
|
| 385 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 386 |
salt = signed_bundle.get("salt", b"armiger-rsi-delta")
|
|
|
|
|
|
|
| 387 |
if isinstance(salt, str):
|
| 388 |
-
|
| 389 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 390 |
if len(salt_bytes) > 64:
|
| 391 |
return False, f"bep44_salt_exceeds_64_bytes:{len(salt_bytes)}", None
|
| 392 |
-
|
| 393 |
-
|
| 394 |
-
|
| 395 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 396 |
except Exception as exc:
|
| 397 |
return False, f"malformed_bundle_envelope:{exc}", None
|
| 398 |
|
|
@@ -406,10 +482,10 @@ class ArmigerDHTRSIEngine:
|
|
| 406 |
self._prune_inbound_delta_tracking(now_ts)
|
| 407 |
|
| 408 |
sender_key = (pub_bytes.hex(), salt_bytes)
|
| 409 |
-
prev_entry = self._inbound_sender_seq.get(sender_key)
|
| 410 |
min_seq = max(
|
| 411 |
self.current_seq,
|
| 412 |
-
prev_entry[0] if prev_entry is not None else
|
| 413 |
)
|
| 414 |
if seq <= min_seq:
|
| 415 |
return False, f"stale_or_replayed_delta_seq:{seq}<={min_seq}", None
|
|
@@ -427,25 +503,33 @@ class ArmigerDHTRSIEngine:
|
|
| 427 |
if epoch_val <= 0 or abs(now_ts - epoch_val) > self.delta_freshness_window_seconds:
|
| 428 |
return False, f"stale_or_future_delta_epoch:{epoch_val}", None
|
| 429 |
|
| 430 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 431 |
superseded_digests = [
|
| 432 |
d for d, (_, sk) in self._seen_delta_digests.items() if sk == sender_key
|
| 433 |
]
|
| 434 |
-
|
| 435 |
-
|
| 436 |
-
|
| 437 |
-
|
| 438 |
-
|
| 439 |
-
|
| 440 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 441 |
|
| 442 |
-
if sender_key in self._inbound_sender_seq:
|
| 443 |
-
self._inbound_sender_seq.pop(sender_key, None)
|
| 444 |
-
while len(self._inbound_sender_seq) >= self.max_tracked_senders:
|
| 445 |
-
oldest_sk = next(iter(self._inbound_sender_seq))
|
| 446 |
-
evicted_seq, _ = self._inbound_sender_seq.pop(oldest_sk)
|
| 447 |
-
self._evicted_sender_seq_floor = max(self._evicted_sender_seq_floor, evicted_seq)
|
| 448 |
-
self._inbound_sender_seq[sender_key] = (seq, now_ts)
|
| 449 |
scale = float(parsed.get("scale", 1.0 / 127.0))
|
| 450 |
w_q8 = parsed.get("delta_w_q8", [])
|
| 451 |
b_q8 = parsed.get("delta_b_q8", [])
|
|
@@ -459,14 +543,48 @@ class ArmigerDHTRSIEngine:
|
|
| 459 |
"reliability_bins": parsed.get("reliability_bins", []),
|
| 460 |
"sender_ephemeral_pubkey_hex": pub_bytes.hex(),
|
| 461 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 462 |
return True, "verified_and_ingested", decoded
|
| 463 |
|
| 464 |
def ingest_signed_bundle(
|
| 465 |
self,
|
| 466 |
signed_bundle: Mapping[str, Any],
|
|
|
|
|
|
|
| 467 |
) -> Tuple[bool, str, Optional[Dict[str, Any]]]:
|
| 468 |
"""Alias for `ingest_signed_delta_bundle`."""
|
| 469 |
-
return self.ingest_signed_delta_bundle(signed_bundle)
|
| 470 |
|
| 471 |
async def publish_to_dht(
|
| 472 |
self,
|
|
@@ -608,6 +726,14 @@ class ArmigerDHTRSIEngine:
|
|
| 608 |
require_pq: Optional[bool] = None,
|
| 609 |
require_authoritative_asn: Optional[bool] = None,
|
| 610 |
asn_verifier: Optional[AuthoritativeASNVerifier] = None,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 611 |
) -> Tuple[bool, str, Optional[CanaryRegressionReport]]:
|
| 612 |
"""Verify signed DHT update manifest, check ASN quorum, validate safe formats, and run 500-canary gate."""
|
| 613 |
if self.mode == DHTRSIMode.OFF:
|
|
@@ -624,6 +750,27 @@ class ArmigerDHTRSIEngine:
|
|
| 624 |
for wots_pk_hex in WOTSPlusSHAKE256._ED25519_TO_WOTS_PUBS.get(norm_ed, ()):
|
| 625 |
eff_pq_signers[wots_pk_hex] = identity
|
| 626 |
require_pq_flag = self.require_pq if require_pq is None else bool(require_pq)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 627 |
ok, reason = ThresholdSignatureVerifier.verify(
|
| 628 |
manifest=manifest,
|
| 629 |
trusted_signers=self.trusted_signers,
|
|
@@ -631,6 +778,9 @@ class ArmigerDHTRSIEngine:
|
|
| 631 |
current_seq=self.current_seq,
|
| 632 |
trusted_pq_signers=eff_pq_signers,
|
| 633 |
require_pq=require_pq_flag,
|
|
|
|
|
|
|
|
|
|
| 634 |
)
|
| 635 |
if not ok:
|
| 636 |
return False, f"manifest_threshold_failed:{reason}", None
|
|
@@ -640,14 +790,58 @@ class ArmigerDHTRSIEngine:
|
|
| 640 |
if actual_sha.lower() != manifest.bundle_sha256.lower():
|
| 641 |
return False, "bundle_sha256_digest_mismatch", None
|
| 642 |
|
| 643 |
-
#
|
| 644 |
-
|
| 645 |
-
|
| 646 |
-
|
| 647 |
-
|
| 648 |
-
asn_verifier=asn_verifier,
|
| 649 |
-
require_authoritative_asn=require_authoritative_asn,
|
| 650 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 651 |
if not q_ok:
|
| 652 |
return False, f"sybil_quorum_failed:{q_reason}", None
|
| 653 |
|
|
@@ -664,6 +858,280 @@ class ArmigerDHTRSIEngine:
|
|
| 664 |
if not canary_report.passed:
|
| 665 |
return False, f"golden_canary_gate_rejected:{canary_report.reason}", canary_report
|
| 666 |
|
| 667 |
-
# All checks passed! Advance monotonic sequence
|
| 668 |
self.current_seq = manifest.seq
|
|
|
|
| 669 |
return True, "update_staged_and_verified", canary_report
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
|
| 46 |
|
| 47 |
import sys
|
|
|
|
|
|
|
|
|
|
| 48 |
|
| 49 |
from dht_rsi_models_and_privacy import (
|
| 50 |
+
AcyclicReleaseEnvelope,
|
| 51 |
AnonymityGuard,
|
| 52 |
AnonymityViolationError,
|
| 53 |
AnonymizedTelemetryRecord,
|
| 54 |
+
AuthenticatedConformalBaseline,
|
| 55 |
+
BenchmarkAttestationPredicate,
|
| 56 |
BlindedTitleStem,
|
| 57 |
BundleFormat,
|
| 58 |
CanaryItem,
|
|
|
|
| 62 |
ContinuousAggregationResult,
|
| 63 |
ContinuousDeltaSubmission,
|
| 64 |
DHTRSIMode,
|
| 65 |
+
DSSE_TYPE_ATTESTATION_V2,
|
| 66 |
+
DSSE_TYPE_BASELINE_V1,
|
| 67 |
+
DSSE_TYPE_RELEASE_V2,
|
| 68 |
EncryptedTelemetryBundle,
|
| 69 |
MediaCategory,
|
| 70 |
PeerAttestation,
|
|
|
|
| 76 |
VerifiedObservation,
|
| 77 |
ZeroCueCatalogDelta,
|
| 78 |
_bep44_canonical_sign_buffer,
|
| 79 |
+
dsse_pae,
|
| 80 |
pure_laplace_noise,
|
| 81 |
resolve_opt_in_sharing,
|
| 82 |
)
|
|
|
|
| 95 |
_OptInAdapterBase,
|
| 96 |
)
|
| 97 |
from dht_rsi_verification import (
|
| 98 |
+
AcyclicProvenanceVerifier,
|
| 99 |
AuthoritativeASNVerifier,
|
| 100 |
PhysicalGroundTruthVerifier,
|
| 101 |
SafeFormatValidator,
|
|
|
|
| 141 |
dp_sigma: float = 0.05,
|
| 142 |
max_seen_delta_digests: int = 4096,
|
| 143 |
max_tracked_senders: int = 1024,
|
| 144 |
+
max_sender_tombstones: Optional[int] = None,
|
| 145 |
delta_freshness_window_seconds: int = 14 * 86400,
|
| 146 |
max_telemetry_queue_size: int = 1024,
|
| 147 |
max_item_encryption_retries: int = 3,
|
| 148 |
+
trusted_evaluator_signers: Optional[Dict[str, str]] = None,
|
| 149 |
+
trusted_contributor_signers: Optional[Dict[str, str]] = None,
|
| 150 |
+
require_acyclic_envelope: bool = True,
|
| 151 |
) -> None:
|
| 152 |
self.opt_in_sharing: bool = resolve_opt_in_sharing(opt_in_sharing)
|
| 153 |
if mode is None:
|
|
|
|
| 160 |
|
| 161 |
self.mode = DHTRSIMode(resolved_mode)
|
| 162 |
self.current_seq = int(current_seq)
|
| 163 |
+
self.current_bundle_sha256: Optional[str] = None
|
| 164 |
self._telemetry_seq = int(current_seq)
|
| 165 |
self.trusted_signers = trusted_signers or {}
|
| 166 |
self.trusted_pq_signers = trusted_pq_signers or {}
|
| 167 |
+
self.trusted_evaluator_signers = dict(trusted_evaluator_signers or {})
|
| 168 |
+
self.trusted_contributor_signers = dict(trusted_contributor_signers or {})
|
| 169 |
+
self.require_acyclic_envelope = bool(require_acyclic_envelope)
|
| 170 |
self.require_pq = bool(require_pq)
|
| 171 |
self.threshold_m = int(threshold_m)
|
| 172 |
self.min_distinct_asns = int(min_distinct_asns)
|
|
|
|
| 182 |
self.max_item_encryption_retries = max(int(max_item_encryption_retries), 1)
|
| 183 |
self._seen_delta_digests: Dict[str, Tuple[int, Tuple[str, bytes]]] = {}
|
| 184 |
self._inbound_sender_seq: Dict[Tuple[str, bytes], Tuple[int, int]] = {}
|
| 185 |
+
self._evicted_sender_tombstones: Dict[Tuple[str, bytes], Tuple[int, int]] = {}
|
| 186 |
+
self.max_sender_tombstones: int = (
|
| 187 |
+
max(int(max_sender_tombstones), 1)
|
| 188 |
+
if max_sender_tombstones is not None
|
| 189 |
+
else max(self.max_tracked_senders * 8, 256)
|
| 190 |
+
)
|
| 191 |
|
| 192 |
self.verifier = PhysicalGroundTruthVerifier()
|
| 193 |
self.anonymizer = TelemetryAnonymizer(
|
|
|
|
| 206 |
self._outbound_bundles_shared: int = 0
|
| 207 |
self._outbound_packets_blocked_opt_out: int = 0
|
| 208 |
self._outbound_bytes_shared: int = 0
|
| 209 |
+
self._outbound_udp_packets_sent: int = 0
|
| 210 |
+
self._outbound_udp_puts_succeeded: int = 0
|
| 211 |
self._telemetry_encryption_errors: int = 0
|
| 212 |
self._telemetry_dropped_backlog: int = 0
|
| 213 |
self._last_telemetry_error: str = ""
|
|
|
|
| 279 |
"outbound_bundles_shared": int(self._outbound_bundles_shared),
|
| 280 |
"outbound_packets_blocked_opt_out": int(self._outbound_packets_blocked_opt_out),
|
| 281 |
"outbound_bytes_shared": int(self._outbound_bytes_shared),
|
| 282 |
+
"outbound_udp_packets_sent": int(getattr(self, "_outbound_udp_packets_sent", 0)),
|
| 283 |
+
"outbound_udp_puts_succeeded": int(getattr(self, "_outbound_udp_puts_succeeded", 0)),
|
| 284 |
"ephemeral_identity_enabled": bool(self.anonymity_guard.ephemeral_identity),
|
| 285 |
"ephemeral_pubkey_hex": self.anonymity_guard.current_pubkey_hex() if self.opt_in_sharing else None,
|
| 286 |
"ephemeral_rotations": int(self.anonymity_guard.rotation_count),
|
|
|
|
| 393 |
for sk in expired_senders:
|
| 394 |
self._inbound_sender_seq.pop(sk, None)
|
| 395 |
|
| 396 |
+
expired_tombstones = [
|
| 397 |
+
sk for sk, (_, seen_ts) in self._evicted_sender_tombstones.items() if seen_ts < cutoff
|
| 398 |
+
]
|
| 399 |
+
for sk in expired_tombstones:
|
| 400 |
+
self._evicted_sender_tombstones.pop(sk, None)
|
| 401 |
+
|
| 402 |
def ingest_signed_delta_bundle(
|
| 403 |
self,
|
| 404 |
signed_bundle: Mapping[str, Any],
|
| 405 |
+
*,
|
| 406 |
+
commit: bool = True,
|
| 407 |
) -> Tuple[bool, str, Optional[Dict[str, Any]]]:
|
| 408 |
"""Verify Ed25519 signature and AnonymityGuard schema of an inbound delta bundle (works in pull_only mode)."""
|
| 409 |
if self.mode == DHTRSIMode.OFF:
|
| 410 |
return False, "plugin_mode_off", None
|
| 411 |
|
| 412 |
+
if not isinstance(signed_bundle, Mapping):
|
| 413 |
+
return False, "malformed_bundle_envelope:not_mapping", None
|
| 414 |
+
|
| 415 |
try:
|
| 416 |
v_raw = signed_bundle["v"]
|
| 417 |
+
if isinstance(v_raw, str):
|
| 418 |
+
if len(v_raw) > 2 * 1024 * 1024:
|
| 419 |
+
return False, "malformed_bundle_envelope:v_too_large", None
|
| 420 |
+
v_bytes = v_raw.encode("utf-8")
|
| 421 |
+
elif isinstance(v_raw, (bytes, bytearray, memoryview)):
|
| 422 |
+
if len(v_raw) > 2 * 1024 * 1024:
|
| 423 |
+
return False, "malformed_bundle_envelope:v_too_large", None
|
| 424 |
+
v_bytes = bytes(v_raw)
|
| 425 |
+
else:
|
| 426 |
+
return False, f"malformed_bundle_envelope:invalid_v_type:{type(v_raw).__name__}", None
|
| 427 |
+
|
| 428 |
+
raw_seq = signed_bundle["seq"]
|
| 429 |
+
if isinstance(raw_seq, bool) or not isinstance(raw_seq, int) or raw_seq < 1 or raw_seq > (2**63 - 1):
|
| 430 |
+
return False, f"malformed_bundle_envelope:invalid_seq:{raw_seq!r}", None
|
| 431 |
+
seq = int(raw_seq)
|
| 432 |
+
|
| 433 |
salt = signed_bundle.get("salt", b"armiger-rsi-delta")
|
| 434 |
+
if salt is None:
|
| 435 |
+
salt = b"armiger-rsi-delta"
|
| 436 |
if isinstance(salt, str):
|
| 437 |
+
if len(salt) > 256:
|
| 438 |
+
return False, f"bep44_salt_exceeds_64_bytes:{len(salt)}", None
|
| 439 |
+
salt_bytes = salt.encode("utf-8")
|
| 440 |
+
elif isinstance(salt, (bytes, bytearray, memoryview)):
|
| 441 |
+
if len(salt) > 64:
|
| 442 |
+
return False, f"bep44_salt_exceeds_64_bytes:{len(salt)}", None
|
| 443 |
+
salt_bytes = bytes(salt)
|
| 444 |
+
else:
|
| 445 |
+
return False, f"malformed_bundle_envelope:invalid_salt_type:{type(salt).__name__}", None
|
| 446 |
if len(salt_bytes) > 64:
|
| 447 |
return False, f"bep44_salt_exceeds_64_bytes:{len(salt_bytes)}", None
|
| 448 |
+
|
| 449 |
+
pub_raw = signed_bundle["k"] if signed_bundle.get("k") is not None else signed_bundle.get("pubkey_hex")
|
| 450 |
+
if isinstance(pub_raw, str):
|
| 451 |
+
if len(pub_raw.strip()) != 64:
|
| 452 |
+
return False, "malformed_bundle_envelope:invalid_pubkey_hex_length", None
|
| 453 |
+
pub_bytes = bytes.fromhex(pub_raw.strip())
|
| 454 |
+
elif isinstance(pub_raw, (bytes, bytearray, memoryview)):
|
| 455 |
+
if len(pub_raw) != 32:
|
| 456 |
+
return False, f"malformed_bundle_envelope:invalid_pubkey_length:{len(pub_raw)}", None
|
| 457 |
+
pub_bytes = bytes(pub_raw)
|
| 458 |
+
else:
|
| 459 |
+
return False, f"malformed_bundle_envelope:invalid_pubkey_type:{type(pub_raw).__name__}", None
|
| 460 |
+
|
| 461 |
+
sig_raw = signed_bundle["sig"] if signed_bundle.get("sig") is not None else signed_bundle.get("sig_hex")
|
| 462 |
+
if isinstance(sig_raw, str):
|
| 463 |
+
if len(sig_raw.strip()) != 128:
|
| 464 |
+
return False, "malformed_bundle_envelope:invalid_sig_hex_length", None
|
| 465 |
+
sig_bytes = bytes.fromhex(sig_raw.strip())
|
| 466 |
+
elif isinstance(sig_raw, (bytes, bytearray, memoryview)):
|
| 467 |
+
if len(sig_raw) != 64:
|
| 468 |
+
return False, f"malformed_bundle_envelope:invalid_sig_length:{len(sig_raw)}", None
|
| 469 |
+
sig_bytes = bytes(sig_raw)
|
| 470 |
+
else:
|
| 471 |
+
return False, f"malformed_bundle_envelope:invalid_sig_type:{type(sig_raw).__name__}", None
|
| 472 |
except Exception as exc:
|
| 473 |
return False, f"malformed_bundle_envelope:{exc}", None
|
| 474 |
|
|
|
|
| 482 |
self._prune_inbound_delta_tracking(now_ts)
|
| 483 |
|
| 484 |
sender_key = (pub_bytes.hex(), salt_bytes)
|
| 485 |
+
prev_entry = self._inbound_sender_seq.get(sender_key) or self._evicted_sender_tombstones.get(sender_key)
|
| 486 |
min_seq = max(
|
| 487 |
self.current_seq,
|
| 488 |
+
prev_entry[0] if prev_entry is not None else 0,
|
| 489 |
)
|
| 490 |
if seq <= min_seq:
|
| 491 |
return False, f"stale_or_replayed_delta_seq:{seq}<={min_seq}", None
|
|
|
|
| 503 |
if epoch_val <= 0 or abs(now_ts - epoch_val) > self.delta_freshness_window_seconds:
|
| 504 |
return False, f"stale_or_future_delta_epoch:{epoch_val}", None
|
| 505 |
|
| 506 |
+
# Never evict an unexpired sender high-water mark: if tracking capacity (active + tombstones)
|
| 507 |
+
# is full of unexpired senders, reject previously unseen sender identities fail-closed.
|
| 508 |
+
is_known_sender = sender_key in self._inbound_sender_seq or sender_key in self._evicted_sender_tombstones
|
| 509 |
+
if (
|
| 510 |
+
not is_known_sender
|
| 511 |
+
and len(self._inbound_sender_seq) >= self.max_tracked_senders
|
| 512 |
+
and len(self._evicted_sender_tombstones) >= self.max_sender_tombstones
|
| 513 |
+
):
|
| 514 |
+
return False, "replay_tracking_capacity_exhausted", None
|
| 515 |
+
|
| 516 |
superseded_digests = [
|
| 517 |
d for d, (_, sk) in self._seen_delta_digests.items() if sk == sender_key
|
| 518 |
]
|
| 519 |
+
projected_digest_count = len(self._seen_delta_digests) - len(superseded_digests)
|
| 520 |
+
if projected_digest_count >= self.max_seen_delta_digests:
|
| 521 |
+
evictable_d = next(
|
| 522 |
+
(
|
| 523 |
+
d
|
| 524 |
+
for d, (_, sk) in self._seen_delta_digests.items()
|
| 525 |
+
if d not in superseded_digests
|
| 526 |
+
and (sk in self._inbound_sender_seq or sk in self._evicted_sender_tombstones or sk == sender_key)
|
| 527 |
+
),
|
| 528 |
+
None,
|
| 529 |
+
)
|
| 530 |
+
if evictable_d is None:
|
| 531 |
+
return False, "replay_digest_capacity_exhausted", None
|
| 532 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 533 |
scale = float(parsed.get("scale", 1.0 / 127.0))
|
| 534 |
w_q8 = parsed.get("delta_w_q8", [])
|
| 535 |
b_q8 = parsed.get("delta_b_q8", [])
|
|
|
|
| 543 |
"reliability_bins": parsed.get("reliability_bins", []),
|
| 544 |
"sender_ephemeral_pubkey_hex": pub_bytes.hex(),
|
| 545 |
}
|
| 546 |
+
if not commit:
|
| 547 |
+
return True, "verified_candidate", decoded
|
| 548 |
+
|
| 549 |
+
for d in superseded_digests:
|
| 550 |
+
self._seen_delta_digests.pop(d, None)
|
| 551 |
+
|
| 552 |
+
while len(self._seen_delta_digests) >= self.max_seen_delta_digests:
|
| 553 |
+
evictable_d = next(
|
| 554 |
+
(
|
| 555 |
+
d
|
| 556 |
+
for d, (_, sk) in self._seen_delta_digests.items()
|
| 557 |
+
if sk in self._inbound_sender_seq or sk in self._evicted_sender_tombstones or sk == sender_key
|
| 558 |
+
),
|
| 559 |
+
None,
|
| 560 |
+
)
|
| 561 |
+
if evictable_d is None:
|
| 562 |
+
return False, "replay_digest_capacity_exhausted", None
|
| 563 |
+
self._seen_delta_digests.pop(evictable_d, None)
|
| 564 |
+
|
| 565 |
+
eff_retention_ts = max(now_ts, epoch_val)
|
| 566 |
+
self._seen_delta_digests[bundle_digest] = (eff_retention_ts, sender_key)
|
| 567 |
+
|
| 568 |
+
self._evicted_sender_tombstones.pop(sender_key, None)
|
| 569 |
+
if sender_key in self._inbound_sender_seq:
|
| 570 |
+
self._inbound_sender_seq.pop(sender_key, None)
|
| 571 |
+
while len(self._inbound_sender_seq) >= self.max_tracked_senders:
|
| 572 |
+
if len(self._evicted_sender_tombstones) >= self.max_sender_tombstones:
|
| 573 |
+
return False, "replay_tracking_capacity_exhausted", None
|
| 574 |
+
oldest_sk = next(iter(self._inbound_sender_seq))
|
| 575 |
+
evicted_seq, evicted_ts = self._inbound_sender_seq.pop(oldest_sk)
|
| 576 |
+
self._evicted_sender_tombstones[oldest_sk] = (evicted_seq, evicted_ts)
|
| 577 |
+
self._inbound_sender_seq[sender_key] = (seq, eff_retention_ts)
|
| 578 |
return True, "verified_and_ingested", decoded
|
| 579 |
|
| 580 |
def ingest_signed_bundle(
|
| 581 |
self,
|
| 582 |
signed_bundle: Mapping[str, Any],
|
| 583 |
+
*,
|
| 584 |
+
commit: bool = True,
|
| 585 |
) -> Tuple[bool, str, Optional[Dict[str, Any]]]:
|
| 586 |
"""Alias for `ingest_signed_delta_bundle`."""
|
| 587 |
+
return self.ingest_signed_delta_bundle(signed_bundle, commit=commit)
|
| 588 |
|
| 589 |
async def publish_to_dht(
|
| 590 |
self,
|
|
|
|
| 726 |
require_pq: Optional[bool] = None,
|
| 727 |
require_authoritative_asn: Optional[bool] = None,
|
| 728 |
asn_verifier: Optional[AuthoritativeASNVerifier] = None,
|
| 729 |
+
bootstrap_checkpoint_seq: Optional[int] = None,
|
| 730 |
+
authenticated_baseline: Optional[AuthenticatedConformalBaseline] = None,
|
| 731 |
+
expected_parent_bundle_sha256: Optional[str] = None,
|
| 732 |
+
expected_canary_suite_sha256: Optional[str] = None,
|
| 733 |
+
release_envelope: Optional[AcyclicReleaseEnvelope] = None,
|
| 734 |
+
trusted_evaluator_signers: Optional[Dict[str, str]] = None,
|
| 735 |
+
trusted_contributor_signers: Optional[Dict[str, str]] = None,
|
| 736 |
+
require_acyclic_envelope: Optional[bool] = None,
|
| 737 |
) -> Tuple[bool, str, Optional[CanaryRegressionReport]]:
|
| 738 |
"""Verify signed DHT update manifest, check ASN quorum, validate safe formats, and run 500-canary gate."""
|
| 739 |
if self.mode == DHTRSIMode.OFF:
|
|
|
|
| 750 |
for wots_pk_hex in WOTSPlusSHAKE256._ED25519_TO_WOTS_PUBS.get(norm_ed, ()):
|
| 751 |
eff_pq_signers[wots_pk_hex] = identity
|
| 752 |
require_pq_flag = self.require_pq if require_pq is None else bool(require_pq)
|
| 753 |
+
q_ok, q_reason, asns = SybilQuorumVerifier.verify_quorum(
|
| 754 |
+
attestations=attestations,
|
| 755 |
+
target_hash=manifest.bundle_sha256,
|
| 756 |
+
min_distinct_asns=max(self.min_distinct_asns, manifest.min_asn_quorum),
|
| 757 |
+
asn_verifier=asn_verifier,
|
| 758 |
+
require_authoritative_asn=require_authoritative_asn,
|
| 759 |
+
)
|
| 760 |
+
if authenticated_baseline is not None:
|
| 761 |
+
b_ok, b_reason = authenticated_baseline.verify_signature(self.trusted_signers)
|
| 762 |
+
if not b_ok:
|
| 763 |
+
return False, f"authenticated_baseline_failed:{b_reason}", None
|
| 764 |
+
eff_parent_sha = (expected_parent_bundle_sha256 or self.current_bundle_sha256 or "").strip().lower()
|
| 765 |
+
if not eff_parent_sha:
|
| 766 |
+
return False, "authenticated_baseline_missing_expected_parent_bundle_sha256", None
|
| 767 |
+
if authenticated_baseline.parent_bundle_sha256.strip().lower() != eff_parent_sha:
|
| 768 |
+
return False, "authenticated_baseline_parent_bundle_mismatch", None
|
| 769 |
+
eff_suite_sha = (expected_canary_suite_sha256 or self.canary_gate.suite_sha256()).strip().lower()
|
| 770 |
+
if authenticated_baseline.canary_suite_sha256.strip().lower() != eff_suite_sha:
|
| 771 |
+
return False, "authenticated_baseline_canary_suite_mismatch", None
|
| 772 |
+
baseline_metrics = authenticated_baseline.to_metrics_dict()
|
| 773 |
+
|
| 774 |
ok, reason = ThresholdSignatureVerifier.verify(
|
| 775 |
manifest=manifest,
|
| 776 |
trusted_signers=self.trusted_signers,
|
|
|
|
| 778 |
current_seq=self.current_seq,
|
| 779 |
trusted_pq_signers=eff_pq_signers,
|
| 780 |
require_pq=require_pq_flag,
|
| 781 |
+
asn_verifier=asn_verifier,
|
| 782 |
+
verified_asn_quorum=asns if q_ok else None,
|
| 783 |
+
bootstrap_checkpoint_seq=bootstrap_checkpoint_seq,
|
| 784 |
)
|
| 785 |
if not ok:
|
| 786 |
return False, f"manifest_threshold_failed:{reason}", None
|
|
|
|
| 790 |
if actual_sha.lower() != manifest.bundle_sha256.lower():
|
| 791 |
return False, "bundle_sha256_digest_mismatch", None
|
| 792 |
|
| 793 |
+
# 2b. Enforce acyclic provenance envelope (disjoint contributors / evaluators / release signers)
|
| 794 |
+
eff_req_env = (
|
| 795 |
+
self.require_acyclic_envelope
|
| 796 |
+
if require_acyclic_envelope is None
|
| 797 |
+
else bool(require_acyclic_envelope)
|
|
|
|
|
|
|
| 798 |
)
|
| 799 |
+
if eff_req_env or release_envelope is not None:
|
| 800 |
+
if release_envelope is None:
|
| 801 |
+
return False, "missing_acyclic_release_envelope", None
|
| 802 |
+
if (
|
| 803 |
+
release_envelope.manifest.canonical_payload() != manifest.canonical_payload()
|
| 804 |
+
or dict(release_envelope.manifest.signatures) != dict(manifest.signatures)
|
| 805 |
+
or dict(release_envelope.manifest.pq_signatures) != dict(manifest.pq_signatures)
|
| 806 |
+
):
|
| 807 |
+
return False, "release_envelope_manifest_mismatch", None
|
| 808 |
+
eff_parent_sha = (expected_parent_bundle_sha256 or self.current_bundle_sha256 or "").strip().lower()
|
| 809 |
+
if not eff_parent_sha:
|
| 810 |
+
return False, "release_envelope_missing_expected_parent_bundle_sha256", None
|
| 811 |
+
if release_envelope.parent_bundle_sha256.strip().lower() != eff_parent_sha:
|
| 812 |
+
return False, "release_envelope_parent_bundle_mismatch", None
|
| 813 |
+
eff_suite_sha = (expected_canary_suite_sha256 or self.canary_gate.suite_sha256()).strip().lower()
|
| 814 |
+
if release_envelope.canary_suite_sha256.strip().lower() != eff_suite_sha:
|
| 815 |
+
return False, "release_envelope_canary_suite_mismatch", None
|
| 816 |
+
eff_eval_signers = (
|
| 817 |
+
dict(trusted_evaluator_signers)
|
| 818 |
+
if trusted_evaluator_signers is not None
|
| 819 |
+
else dict(self.trusted_evaluator_signers)
|
| 820 |
+
)
|
| 821 |
+
eff_contrib_signers = (
|
| 822 |
+
dict(trusted_contributor_signers)
|
| 823 |
+
if trusted_contributor_signers is not None
|
| 824 |
+
else dict(self.trusted_contributor_signers)
|
| 825 |
+
)
|
| 826 |
+
env_ok, env_reason = AcyclicProvenanceVerifier.verify_envelope(
|
| 827 |
+
release_envelope,
|
| 828 |
+
trusted_release_signers=self.trusted_signers,
|
| 829 |
+
trusted_evaluator_signers=eff_eval_signers,
|
| 830 |
+
threshold_m=self.threshold_m,
|
| 831 |
+
min_evaluator_asns=max(self.min_distinct_asns, manifest.min_asn_quorum),
|
| 832 |
+
current_seq=self.current_seq,
|
| 833 |
+
bootstrap_checkpoint_seq=bootstrap_checkpoint_seq,
|
| 834 |
+
require_pq=require_pq_flag,
|
| 835 |
+
asn_verifier=asn_verifier,
|
| 836 |
+
trusted_release_pq_signers=eff_pq_signers,
|
| 837 |
+
trusted_contributor_signers=eff_contrib_signers,
|
| 838 |
+
)
|
| 839 |
+
if not env_ok:
|
| 840 |
+
return False, f"acyclic_provenance_failed:{env_reason}", None
|
| 841 |
+
if authenticated_baseline is None and release_envelope.authenticated_baseline is not None:
|
| 842 |
+
baseline_metrics = release_envelope.authenticated_baseline.to_metrics_dict()
|
| 843 |
+
|
| 844 |
+
# 3. Verify K-of-N distinct-ASN Sybil quorum
|
| 845 |
if not q_ok:
|
| 846 |
return False, f"sybil_quorum_failed:{q_reason}", None
|
| 847 |
|
|
|
|
| 858 |
if not canary_report.passed:
|
| 859 |
return False, f"golden_canary_gate_rejected:{canary_report.reason}", canary_report
|
| 860 |
|
| 861 |
+
# All checks passed! Advance monotonic sequence and active bundle digest
|
| 862 |
self.current_seq = manifest.seq
|
| 863 |
+
self.current_bundle_sha256 = actual_sha.lower()
|
| 864 |
return True, "update_staged_and_verified", canary_report
|
| 865 |
+
|
| 866 |
+
def dispatch_contribution_udp(
|
| 867 |
+
self,
|
| 868 |
+
signed_bundle: Optional[Mapping[str, Any]] = None,
|
| 869 |
+
*,
|
| 870 |
+
delta_w: Optional[Sequence[float]] = None,
|
| 871 |
+
delta_b: Optional[Sequence[float]] = None,
|
| 872 |
+
reliability_bins: Optional[Sequence[Any]] = None,
|
| 873 |
+
n_samples: int = 25,
|
| 874 |
+
epoch: Optional[Union[int, float]] = None,
|
| 875 |
+
seq: Optional[int] = None,
|
| 876 |
+
model_tier: str = "ultra",
|
| 877 |
+
salt: bytes = b"armiger-rsi-delta",
|
| 878 |
+
rotate_key: Optional[bool] = None,
|
| 879 |
+
dht_nodes: Optional[Sequence[Tuple[str, int]]] = None,
|
| 880 |
+
collector_nodes: Optional[Sequence[Tuple[str, int]]] = None,
|
| 881 |
+
discover_collectors: bool = True,
|
| 882 |
+
release_pubkey_hex: Optional[str] = None,
|
| 883 |
+
release_salt: Union[bytes, str, None] = None,
|
| 884 |
+
timeout: float = 2.5,
|
| 885 |
+
max_dht_puts: int = 8,
|
| 886 |
+
) -> Dict[str, Any]:
|
| 887 |
+
"""Publish a signed `< 1,000`-byte BEP-44 delta item over outbound UDP to Mainline DHT
|
| 888 |
+
(`target = SHA-1(ephemeral_pubkey || "armiger-rsi-delta")`) and directly to collector
|
| 889 |
+
DHT nodes (`:6881/udp`) discovered via the release pointer.
|
| 890 |
+
|
| 891 |
+
Strictly sends 0 packets and opens 0 sockets when `opt_in_sharing=False`.
|
| 892 |
+
"""
|
| 893 |
+
if not self.opt_in_sharing or self.mode != DHTRSIMode.ANONYMOUS_CONTRIBUTE:
|
| 894 |
+
self._outbound_packets_blocked_opt_out += 1
|
| 895 |
+
return {
|
| 896 |
+
"dispatched": False,
|
| 897 |
+
"reason": "opt_in_sharing_disabled",
|
| 898 |
+
"packets_sent": 0,
|
| 899 |
+
"mainline_puts_ok": 0,
|
| 900 |
+
"collector_puts_ok": 0,
|
| 901 |
+
}
|
| 902 |
+
|
| 903 |
+
if signed_bundle is None:
|
| 904 |
+
bundle = self.export_signed_delta_bundle(
|
| 905 |
+
delta_w=delta_w,
|
| 906 |
+
delta_b=delta_b,
|
| 907 |
+
reliability_bins=reliability_bins,
|
| 908 |
+
n_samples=n_samples,
|
| 909 |
+
epoch=epoch,
|
| 910 |
+
seq=seq,
|
| 911 |
+
model_tier=model_tier,
|
| 912 |
+
salt=salt,
|
| 913 |
+
rotate_key=rotate_key,
|
| 914 |
+
)
|
| 915 |
+
if bundle is None:
|
| 916 |
+
return {
|
| 917 |
+
"dispatched": False,
|
| 918 |
+
"reason": "opt_in_sharing_disabled",
|
| 919 |
+
"packets_sent": 0,
|
| 920 |
+
"mainline_puts_ok": 0,
|
| 921 |
+
"collector_puts_ok": 0,
|
| 922 |
+
}
|
| 923 |
+
else:
|
| 924 |
+
bundle = dict(signed_bundle)
|
| 925 |
+
|
| 926 |
+
v_raw = bundle["v"]
|
| 927 |
+
v_bytes = v_raw.encode("utf-8") if isinstance(v_raw, str) else bytes(v_raw)
|
| 928 |
+
seq_int = int(bundle["seq"])
|
| 929 |
+
salt_raw = bundle.get("salt", b"armiger-rsi-delta")
|
| 930 |
+
salt_bytes = salt_raw.encode("utf-8") if isinstance(salt_raw, str) else bytes(salt_raw)
|
| 931 |
+
k_raw = bundle.get("k") if bundle.get("k") is not None else bundle.get("pubkey_hex")
|
| 932 |
+
pub_bytes = bytes.fromhex(k_raw.strip()) if isinstance(k_raw, str) else bytes(k_raw)
|
| 933 |
+
sig_raw = bundle.get("sig") if bundle.get("sig") is not None else bundle.get("sig_hex")
|
| 934 |
+
sig_bytes = bytes.fromhex(sig_raw.strip()) if isinstance(sig_raw, str) else bytes(sig_raw)
|
| 935 |
+
|
| 936 |
+
# Pre-transmission Canary Firewall & Ed25519 signature verification
|
| 937 |
+
self.anonymity_guard.inspect_wire_payload(v_bytes)
|
| 938 |
+
if len(v_bytes) > 1000:
|
| 939 |
+
raise AnonymityViolationError(f"bep44_value_exceeds_1000_bytes:{len(v_bytes)}")
|
| 940 |
+
vk = ed25519.Ed25519PublicKey.from_public_bytes(pub_bytes)
|
| 941 |
+
vk.verify(sig_bytes, _bep44_canonical_sign_buffer(v=v_bytes, seq=seq_int, salt=salt_bytes))
|
| 942 |
+
|
| 943 |
+
import socket as _socket
|
| 944 |
+
try:
|
| 945 |
+
from verify_dht import (
|
| 946 |
+
DEFAULT_RELEASE_PUBKEY_HEX,
|
| 947 |
+
DEFAULT_RELEASE_SALT,
|
| 948 |
+
_decode_compact_nodes_with_id,
|
| 949 |
+
_xor_distance,
|
| 950 |
+
bdecode_krpc,
|
| 951 |
+
bencode,
|
| 952 |
+
generate_bep42_node_id,
|
| 953 |
+
parse_bootstrap_nodes,
|
| 954 |
+
verify_dht_release,
|
| 955 |
+
)
|
| 956 |
+
except ImportError:
|
| 957 |
+
from verify_dht import (
|
| 958 |
+
DEFAULT_RELEASE_PUBKEY_HEX,
|
| 959 |
+
DEFAULT_RELEASE_SALT,
|
| 960 |
+
_decode_compact_nodes_with_id,
|
| 961 |
+
_xor_distance,
|
| 962 |
+
bdecode_krpc,
|
| 963 |
+
bencode,
|
| 964 |
+
generate_bep42_node_id,
|
| 965 |
+
parse_bootstrap_nodes,
|
| 966 |
+
verify_dht_release,
|
| 967 |
+
)
|
| 968 |
+
|
| 969 |
+
target_20 = hashlib.sha1(pub_bytes + salt_bytes).digest()
|
| 970 |
+
eff_dht_nodes: List[Tuple[str, int]] = (
|
| 971 |
+
list(dht_nodes) if dht_nodes is not None else parse_bootstrap_nodes()
|
| 972 |
+
)
|
| 973 |
+
eff_collectors: List[Tuple[str, int]] = list(collector_nodes or ())
|
| 974 |
+
env_collectors = os.environ.get("ARMIGER_DHT_COLLECTOR_NODES", "").strip()
|
| 975 |
+
if env_collectors:
|
| 976 |
+
for c_addr in parse_bootstrap_nodes(env_collectors):
|
| 977 |
+
if c_addr not in eff_collectors:
|
| 978 |
+
eff_collectors.append(c_addr)
|
| 979 |
+
|
| 980 |
+
if discover_collectors and eff_dht_nodes:
|
| 981 |
+
try:
|
| 982 |
+
ptr_rec = verify_dht_release(
|
| 983 |
+
pubkey_hex=release_pubkey_hex or DEFAULT_RELEASE_PUBKEY_HEX,
|
| 984 |
+
salt=release_salt or DEFAULT_RELEASE_SALT,
|
| 985 |
+
dht_nodes=eff_dht_nodes,
|
| 986 |
+
pointer_only=True,
|
| 987 |
+
timeout=timeout,
|
| 988 |
+
max_dht_queries=16,
|
| 989 |
+
)
|
| 990 |
+
for c_item in ptr_rec.get("collector_nodes", []) or []:
|
| 991 |
+
if isinstance(c_item, (list, tuple)) and len(c_item) == 2:
|
| 992 |
+
c_tup = (str(c_item[0]), int(c_item[1]))
|
| 993 |
+
if c_tup not in eff_collectors:
|
| 994 |
+
eff_collectors.append(c_tup)
|
| 995 |
+
except Exception:
|
| 996 |
+
pass
|
| 997 |
+
|
| 998 |
+
packets_sent = 0
|
| 999 |
+
mainline_puts_ok = 0
|
| 1000 |
+
collector_puts_ok = 0
|
| 1001 |
+
errors: List[str] = []
|
| 1002 |
+
put_attempted_addrs: Set[Tuple[str, int]] = set()
|
| 1003 |
+
|
| 1004 |
+
def _single_socket_get_and_put(addr: Tuple[str, int]) -> Tuple[bool, List[Tuple[bytes, Tuple[str, int]]]]:
|
| 1005 |
+
nonlocal packets_sent
|
| 1006 |
+
host_str, port_int = str(addr[0]).strip(), int(addr[1])
|
| 1007 |
+
try:
|
| 1008 |
+
resolved_ip = _socket.gethostbyname(host_str)
|
| 1009 |
+
except Exception as exc:
|
| 1010 |
+
errors.append(f"{host_str}:{port_int}:resolve:{exc}")
|
| 1011 |
+
return False, []
|
| 1012 |
+
norm_addr = (resolved_ip, port_int)
|
| 1013 |
+
if norm_addr in put_attempted_addrs:
|
| 1014 |
+
return False, []
|
| 1015 |
+
sock = _socket.socket(_socket.AF_INET, _socket.SOCK_DGRAM)
|
| 1016 |
+
refs: List[Tuple[bytes, Tuple[str, int]]] = []
|
| 1017 |
+
try:
|
| 1018 |
+
sock.settimeout(max(0.2, float(timeout)))
|
| 1019 |
+
sock.connect(norm_addr)
|
| 1020 |
+
local_ip = str(sock.getsockname()[0]) or "127.0.0.1"
|
| 1021 |
+
node_id = generate_bep42_node_id(local_ip)
|
| 1022 |
+
q_get = {
|
| 1023 |
+
b"t": b"cg01",
|
| 1024 |
+
b"y": b"q",
|
| 1025 |
+
b"q": b"get",
|
| 1026 |
+
b"a": {b"id": node_id, b"target": target_20},
|
| 1027 |
+
}
|
| 1028 |
+
sock.send(bencode(q_get))
|
| 1029 |
+
packets_sent += 1
|
| 1030 |
+
self._outbound_udp_packets_sent += 1
|
| 1031 |
+
raw_get = sock.recv(8192)
|
| 1032 |
+
msg_get = bdecode_krpc(raw_get)
|
| 1033 |
+
if not isinstance(msg_get, dict) or msg_get.get(b"y") != b"r":
|
| 1034 |
+
return False, refs
|
| 1035 |
+
r_body = msg_get.get(b"r")
|
| 1036 |
+
if not isinstance(r_body, dict):
|
| 1037 |
+
return False, refs
|
| 1038 |
+
refs.extend(_decode_compact_nodes_with_id(r_body.get(b"nodes"), None))
|
| 1039 |
+
for entry in r_body.get(b"nodes_list", []) or []:
|
| 1040 |
+
if (
|
| 1041 |
+
isinstance(entry, list)
|
| 1042 |
+
and len(entry) == 3
|
| 1043 |
+
and isinstance(entry[0], (bytes, bytearray))
|
| 1044 |
+
and len(entry[0]) == 20
|
| 1045 |
+
and isinstance(entry[1], (bytes, bytearray))
|
| 1046 |
+
):
|
| 1047 |
+
refs.append((bytes(entry[0]), ( bytes(entry[1]).decode("ascii", errors="ignore"), int(entry[2]) )))
|
| 1048 |
+
tok = r_body.get(b"token")
|
| 1049 |
+
if not isinstance(tok, (bytes, bytearray)) or not tok:
|
| 1050 |
+
return False, refs
|
| 1051 |
+
put_attempted_addrs.add(norm_addr)
|
| 1052 |
+
q_put = {
|
| 1053 |
+
b"t": b"cp01",
|
| 1054 |
+
b"y": b"q",
|
| 1055 |
+
b"q": b"put",
|
| 1056 |
+
b"a": {
|
| 1057 |
+
b"id": node_id,
|
| 1058 |
+
b"k": pub_bytes,
|
| 1059 |
+
b"salt": salt_bytes,
|
| 1060 |
+
b"seq": int(seq_int),
|
| 1061 |
+
b"sig": sig_bytes,
|
| 1062 |
+
b"token": bytes(tok),
|
| 1063 |
+
b"v": v_bytes,
|
| 1064 |
+
},
|
| 1065 |
+
}
|
| 1066 |
+
sock.send(bencode(q_put))
|
| 1067 |
+
packets_sent += 1
|
| 1068 |
+
self._outbound_udp_packets_sent += 1
|
| 1069 |
+
raw_put = sock.recv(8192)
|
| 1070 |
+
msg_put = bdecode_krpc(raw_put)
|
| 1071 |
+
if isinstance(msg_put, dict) and msg_put.get(b"y") == b"r":
|
| 1072 |
+
self._outbound_udp_puts_succeeded += 1
|
| 1073 |
+
return True, refs
|
| 1074 |
+
err_val = msg_put.get(b"e_detail") or msg_put.get(b"e") if isinstance(msg_put, dict) else "invalid_resp"
|
| 1075 |
+
errors.append(f"{resolved_ip}:{port_int}:put_rejected:{err_val}")
|
| 1076 |
+
return False, refs
|
| 1077 |
+
except Exception as exc:
|
| 1078 |
+
errors.append(f"{resolved_ip}:{port_int}:{exc}")
|
| 1079 |
+
return False, refs
|
| 1080 |
+
finally:
|
| 1081 |
+
sock.close()
|
| 1082 |
+
|
| 1083 |
+
resolved_collector_set: Set[Tuple[str, int]] = set()
|
| 1084 |
+
for c_host, c_port in eff_collectors:
|
| 1085 |
+
try:
|
| 1086 |
+
c_ip = _socket.gethostbyname(str(c_host).strip())
|
| 1087 |
+
resolved_collector_set.add((c_ip, int(c_port)))
|
| 1088 |
+
except Exception:
|
| 1089 |
+
continue
|
| 1090 |
+
|
| 1091 |
+
# 1. Direct BEP-44 put to discovered/configured collector DHT nodes (:6881/udp)
|
| 1092 |
+
for c_addr in eff_collectors:
|
| 1093 |
+
ok_put, _ = _single_socket_get_and_put(c_addr)
|
| 1094 |
+
if ok_put:
|
| 1095 |
+
collector_puts_ok += 1
|
| 1096 |
+
|
| 1097 |
+
# 2. Iterative BEP-44 lookup + put to Mainline DHT nodes
|
| 1098 |
+
frontier: List[Tuple[int, Tuple[str, int]]] = [
|
| 1099 |
+
(_xor_distance(None, target_20), n_addr) for n_addr in eff_dht_nodes
|
| 1100 |
+
]
|
| 1101 |
+
visited_walk: Set[Tuple[str, int]] = set()
|
| 1102 |
+
while frontier and len(visited_walk) < 16 and mainline_puts_ok < max_dht_puts:
|
| 1103 |
+
frontier.sort(key=lambda item: item[0])
|
| 1104 |
+
_, cur_addr = frontier.pop(0)
|
| 1105 |
+
try:
|
| 1106 |
+
cur_ip = _socket.gethostbyname(str(cur_addr[0]).strip())
|
| 1107 |
+
norm_cur = (cur_ip, int(cur_addr[1]))
|
| 1108 |
+
except Exception:
|
| 1109 |
+
continue
|
| 1110 |
+
if norm_cur in visited_walk:
|
| 1111 |
+
continue
|
| 1112 |
+
visited_walk.add(norm_cur)
|
| 1113 |
+
if norm_cur in put_attempted_addrs:
|
| 1114 |
+
if norm_cur in resolved_collector_set and collector_puts_ok > 0:
|
| 1115 |
+
mainline_puts_ok += 1
|
| 1116 |
+
continue
|
| 1117 |
+
ok_put, refs = _single_socket_get_and_put(norm_cur)
|
| 1118 |
+
if ok_put:
|
| 1119 |
+
mainline_puts_ok += 1
|
| 1120 |
+
if norm_cur in resolved_collector_set:
|
| 1121 |
+
collector_puts_ok += 1
|
| 1122 |
+
for ref_nid, ref_addr in refs:
|
| 1123 |
+
if ref_addr not in visited_walk:
|
| 1124 |
+
frontier.append((_xor_distance(ref_nid, target_20), ref_addr))
|
| 1125 |
+
|
| 1126 |
+
return {
|
| 1127 |
+
"dispatched": (mainline_puts_ok + collector_puts_ok) > 0,
|
| 1128 |
+
"target_hex": target_20.hex(),
|
| 1129 |
+
"pubkey_hex": pub_bytes.hex(),
|
| 1130 |
+
"seq": seq_int,
|
| 1131 |
+
"payload_bytes": len(v_bytes),
|
| 1132 |
+
"packets_sent": packets_sent,
|
| 1133 |
+
"mainline_puts_ok": mainline_puts_ok,
|
| 1134 |
+
"collector_puts_ok": collector_puts_ok,
|
| 1135 |
+
"collector_nodes": [list(c) for c in eff_collectors],
|
| 1136 |
+
"errors": errors,
|
| 1137 |
+
}
|
armiger_inference.py
CHANGED
|
@@ -130,7 +130,7 @@ def _extract_token_hashes(text: str) -> frozenset[str]:
|
|
| 130 |
|
| 131 |
def _get_vocab_ext_map(vocab_path: Optional[Path] = None) -> Dict[str, int]:
|
| 132 |
global _VOCAB_EXT_MAP
|
| 133 |
-
if _VOCAB_EXT_MAP is not None and len(_VOCAB_EXT_MAP) > 64:
|
| 134 |
return _VOCAB_EXT_MAP
|
| 135 |
vpath = vocab_path if (vocab_path and vocab_path.exists()) else Path(__file__).resolve().with_name("vocab.json")
|
| 136 |
if not vpath.exists() and os.environ.get("ARMIGER_MODEL_DIR"):
|
|
@@ -469,7 +469,11 @@ class ArmigerStandaloneONNX:
|
|
| 469 |
) -> None:
|
| 470 |
import onnxruntime as ort
|
| 471 |
|
| 472 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 473 |
if onnx_path is None:
|
| 474 |
for cand in ("armiger.onnx", "armiger.int8.onnx", "armiger_ultra.fp32.onnx"):
|
| 475 |
if (base / cand).exists():
|
|
@@ -485,14 +489,28 @@ class ArmigerStandaloneONNX:
|
|
| 485 |
_get_vocab_ext_map(self.vocab_path)
|
| 486 |
self.calibration_path = Path(calibration_path) if calibration_path else (base / "calibration.json")
|
| 487 |
self.session = ort.InferenceSession(str(self.onnx_path), providers=["CPUExecutionProvider"])
|
| 488 |
-
self.head_names = [o.name for o in self.session.get_outputs()]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 489 |
|
| 490 |
@staticmethod
|
| 491 |
def _apply_v3_post_neural_guard(row: Mapping[str, Any], item_res: Dict[str, Any]) -> Dict[str, Any]:
|
| 492 |
return _apply_v3_post_neural_guard(row, item_res)
|
| 493 |
|
| 494 |
def predict_batch(self, rows: Sequence[Mapping[str, Any]]) -> List[Dict[str, Any]]:
|
| 495 |
-
feed =
|
| 496 |
raw_outs = self.session.run(self.head_names, feed)
|
| 497 |
logits_by_head = dict(zip(self.head_names, raw_outs))
|
| 498 |
results: List[Dict[str, Any]] = []
|
|
@@ -507,9 +525,9 @@ class ArmigerStandaloneONNX:
|
|
| 507 |
probs_2d = exps / np.maximum(1e-12, np.sum(exps, axis=-1, keepdims=True))
|
| 508 |
raw_nm = str(rows[i].get("name") or rows[i].get("title") or "")
|
| 509 |
try:
|
| 510 |
-
from armiger.byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans
|
| 511 |
-
except ImportError:
|
| 512 |
from byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans
|
|
|
|
|
|
|
| 513 |
raw_b = raw_nm.encode("utf-8", errors="replace")[: vec.shape[0]]
|
| 514 |
tag_ids = [int(x) for x in decode_bio_logits_viterbi(vec, raw_b).tolist()]
|
| 515 |
classes = NEURAL_HEAD_CLASSES.get(
|
|
|
|
| 130 |
|
| 131 |
def _get_vocab_ext_map(vocab_path: Optional[Path] = None) -> Dict[str, int]:
|
| 132 |
global _VOCAB_EXT_MAP
|
| 133 |
+
if vocab_path is None and _VOCAB_EXT_MAP is not None and len(_VOCAB_EXT_MAP) > 64:
|
| 134 |
return _VOCAB_EXT_MAP
|
| 135 |
vpath = vocab_path if (vocab_path and vocab_path.exists()) else Path(__file__).resolve().with_name("vocab.json")
|
| 136 |
if not vpath.exists() and os.environ.get("ARMIGER_MODEL_DIR"):
|
|
|
|
| 469 |
) -> None:
|
| 470 |
import onnxruntime as ort
|
| 471 |
|
| 472 |
+
try:
|
| 473 |
+
from artifacts import model_directory
|
| 474 |
+
base = model_directory()
|
| 475 |
+
except ImportError:
|
| 476 |
+
base = Path(__file__).resolve().parent
|
| 477 |
if onnx_path is None:
|
| 478 |
for cand in ("armiger.onnx", "armiger.int8.onnx", "armiger_ultra.fp32.onnx"):
|
| 479 |
if (base / cand).exists():
|
|
|
|
| 489 |
_get_vocab_ext_map(self.vocab_path)
|
| 490 |
self.calibration_path = Path(calibration_path) if calibration_path else (base / "calibration.json")
|
| 491 |
self.session = ort.InferenceSession(str(self.onnx_path), providers=["CPUExecutionProvider"])
|
| 492 |
+
self.head_names = [o.name for o in self.session.get_outputs() if o.name != "bio64"]
|
| 493 |
+
self._batch_fn = build_onnx_batch_inputs
|
| 494 |
+
try:
|
| 495 |
+
for inp in self.session.get_inputs():
|
| 496 |
+
if getattr(inp, "name", "") == "stats":
|
| 497 |
+
sh = getattr(inp, "shape", None)
|
| 498 |
+
if isinstance(sh, (list, tuple)) and len(sh) >= 2 and type(sh[1]) is int and sh[1] == 41:
|
| 499 |
+
try:
|
| 500 |
+
from r13_features import build_onnx_batch_inputs as _r13_batch_fn
|
| 501 |
+
except ImportError:
|
| 502 |
+
from r13_features import build_onnx_batch_inputs as _r13_batch_fn # type: ignore
|
| 503 |
+
self._batch_fn = _r13_batch_fn
|
| 504 |
+
break
|
| 505 |
+
except Exception:
|
| 506 |
+
pass
|
| 507 |
|
| 508 |
@staticmethod
|
| 509 |
def _apply_v3_post_neural_guard(row: Mapping[str, Any], item_res: Dict[str, Any]) -> Dict[str, Any]:
|
| 510 |
return _apply_v3_post_neural_guard(row, item_res)
|
| 511 |
|
| 512 |
def predict_batch(self, rows: Sequence[Mapping[str, Any]]) -> List[Dict[str, Any]]:
|
| 513 |
+
feed = self._batch_fn(rows)
|
| 514 |
raw_outs = self.session.run(self.head_names, feed)
|
| 515 |
logits_by_head = dict(zip(self.head_names, raw_outs))
|
| 516 |
results: List[Dict[str, Any]] = []
|
|
|
|
| 525 |
probs_2d = exps / np.maximum(1e-12, np.sum(exps, axis=-1, keepdims=True))
|
| 526 |
raw_nm = str(rows[i].get("name") or rows[i].get("title") or "")
|
| 527 |
try:
|
|
|
|
|
|
|
| 528 |
from byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans
|
| 529 |
+
except ImportError:
|
| 530 |
+
from byte_span_extractor import decode_bio_logits_viterbi, decode_bio_tags_to_spans # type: ignore
|
| 531 |
raw_b = raw_nm.encode("utf-8", errors="replace")[: vec.shape[0]]
|
| 532 |
tag_ids = [int(x) for x in decode_bio_logits_viterbi(vec, raw_b).tolist()]
|
| 533 |
classes = NEURAL_HEAD_CLASSES.get(
|
assets/armiger_overview.png
ADDED
|
Git LFS Details
|
calibration.json
CHANGED
|
@@ -1,67 +1,84 @@
|
|
| 1 |
{
|
| 2 |
-
"
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
"lang": {
|
| 11 |
-
"temperature_T": 0.7,
|
| 12 |
-
"cal_temp_scaled": {
|
| 13 |
-
"n": 368,
|
| 14 |
-
"fitted_on": "snap_wave13_r5_20261021T115351Z_bfb9dfb67b41_val_guarded"
|
| 15 |
-
}
|
| 16 |
},
|
| 17 |
-
"
|
| 18 |
-
|
| 19 |
-
"
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
|
|
|
|
|
|
|
|
|
| 23 |
},
|
| 24 |
-
"
|
| 25 |
-
"
|
| 26 |
-
"
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
|
|
|
|
|
|
| 30 |
},
|
| 31 |
"english_subs": {
|
| 32 |
-
"
|
| 33 |
-
"
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
|
|
|
|
|
|
| 37 |
},
|
| 38 |
-
"
|
| 39 |
-
"
|
| 40 |
-
"
|
| 41 |
-
"n": 243,
|
| 42 |
-
"fitted_on": "snap_wave13_r5_20261021T115351Z_bfb9dfb67b41_val_guarded"
|
| 43 |
-
}
|
| 44 |
},
|
| 45 |
-
"
|
| 46 |
-
"
|
| 47 |
-
"
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
|
|
|
|
|
|
| 51 |
},
|
| 52 |
-
"
|
| 53 |
-
|
| 54 |
-
"
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
}
|
| 59 |
},
|
| 60 |
"menon_logit_adjustment_stop_fit": {
|
| 61 |
-
"
|
| 62 |
"tau": 0.0
|
| 63 |
},
|
| 64 |
-
"
|
| 65 |
"tau": 0.0
|
| 66 |
},
|
| 67 |
"E100000_TC_v3": {
|
|
@@ -83,106 +100,2676 @@
|
|
| 83 |
"tau": 0.0
|
| 84 |
}
|
| 85 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
"mondrian_conformal_thresholds": {
|
| 87 |
"owner_scope": {
|
| 88 |
"alpha": 0.02,
|
| 89 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
}
|
| 91 |
},
|
| 92 |
-
"calibration_version": "v3.0-gold-mit3-final-guarded",
|
| 93 |
"temperature_audit": {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
"category": {
|
| 95 |
-
"
|
| 96 |
-
"
|
| 97 |
-
"
|
| 98 |
-
"
|
| 99 |
-
"
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
},
|
| 104 |
"lang": {
|
| 105 |
-
"
|
| 106 |
-
"
|
| 107 |
-
"
|
| 108 |
-
"
|
| 109 |
-
"
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
},
|
| 114 |
"owner_scope": {
|
| 115 |
-
"
|
| 116 |
-
"
|
| 117 |
-
"
|
| 118 |
-
"
|
| 119 |
-
"
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 123 |
},
|
| 124 |
"title_language": {
|
| 125 |
-
"
|
| 126 |
-
"
|
| 127 |
-
"
|
| 128 |
-
"
|
| 129 |
-
"
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
},
|
| 134 |
-
"
|
| 135 |
-
"
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
"
|
| 140 |
-
"ece5_at_T1_pct": 9.9213,
|
| 141 |
-
"ece5_guarded_pct": 2.5349,
|
| 142 |
-
"acc_pct": 96.3504
|
| 143 |
},
|
| 144 |
-
"
|
| 145 |
-
"
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
"
|
| 150 |
-
"ece5_at_T1_pct": 3.8909,
|
| 151 |
-
"ece5_guarded_pct": 2.9506,
|
| 152 |
-
"acc_pct": 84.7737
|
| 153 |
},
|
| 154 |
-
"
|
| 155 |
-
"
|
| 156 |
-
|
|
|
|
|
|
|
|
|
|
| 157 |
},
|
| 158 |
-
"junk": {
|
| 159 |
-
"inherited": true,
|
| 160 |
-
"reason": "no Gold-val label support; retained wave5a fit"
|
| 161 |
-
}
|
| 162 |
-
},
|
| 163 |
-
"mondrian_thresholds": {
|
| 164 |
-
"adult": 0.5451,
|
| 165 |
-
"junk": 0.101,
|
| 166 |
-
"owner_scope": 0.477
|
| 167 |
-
},
|
| 168 |
-
"mondrian_audit": {
|
| 169 |
"owner_scope": {
|
| 170 |
-
"
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
"
|
| 175 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
}
|
| 177 |
-
}
|
| 178 |
-
"fitted_snapshot_id": "snap_wave13_r5_20261021T115351Z_bfb9dfb67b41",
|
| 179 |
-
"calibrated_onnx_fp32_sha256": "6459a20d7bf45d46d59efbcb50ffb8a519bacda14ae36403dcc510eceaa38f44",
|
| 180 |
-
"mondrian_fitted_calibration_split": {
|
| 181 |
-
"adult": 0.799935,
|
| 182 |
-
"junk": 0.259358,
|
| 183 |
-
"owner_scope": 0.944382,
|
| 184 |
-
"method": "split-conformal FPR quantile on packed calibration split (512), temperature-scaled; NOT shipped as gates (recall cost on confirmation) - see mondrian_refit_report"
|
| 185 |
-
},
|
| 186 |
-
"mondrian_provenance": "Inherited production gates retained for v3.0.0 (higher adult/owner recall on confirmation splits than calibration-fitted FPR gates; fitted alternates recorded for operators).",
|
| 187 |
-
"calibrated_onnx_int8_sha256": "a0dab886be6252739d4c89ce01d65d8d2135204beaec6b00159d038cf420f329"
|
| 188 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"calibrated_onnx_fp32_sha256": "6f94aef676e5bec4968378437769cfdc768fefcc6a21854b8319b33456ebd2ec",
|
| 3 |
+
"calibration_version": "v3.0-gold-r13_3-val-only",
|
| 4 |
+
"feature_stats_dim": 41,
|
| 5 |
+
"fitted_snapshot_id": "snap_r13_20261024T140340Z_064c8e6ccb1f",
|
| 6 |
+
"fp32_vs_int8_val_parity": {
|
| 7 |
+
"adult": {
|
| 8 |
+
"fp32_int8_agreement_pct": 100.0,
|
| 9 |
+
"meets_99_5pct_gate": true
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
},
|
| 11 |
+
"all_heads_min_agreement_pct": 99.57,
|
| 12 |
+
"anime": {
|
| 13 |
+
"bal_acc_drift_pct": 0.11,
|
| 14 |
+
"fp32_int8_agreement_pct": 99.93,
|
| 15 |
+
"fp32_val_bal_acc": 77.67,
|
| 16 |
+
"fp32_val_macro_f1": 75.02,
|
| 17 |
+
"int8_val_bal_acc": 77.78,
|
| 18 |
+
"int8_val_macro_f1": 75.18,
|
| 19 |
+
"meets_99_5pct_gate": true
|
| 20 |
},
|
| 21 |
+
"category": {
|
| 22 |
+
"bal_acc_drift_pct": 0.34,
|
| 23 |
+
"fp32_int8_agreement_pct": 99.57,
|
| 24 |
+
"fp32_val_bal_acc": 85.15,
|
| 25 |
+
"fp32_val_macro_f1": 84.97,
|
| 26 |
+
"int8_val_bal_acc": 84.81,
|
| 27 |
+
"int8_val_macro_f1": 84.75,
|
| 28 |
+
"meets_99_5pct_gate": true
|
| 29 |
},
|
| 30 |
"english_subs": {
|
| 31 |
+
"bal_acc_drift_pct": 0.0,
|
| 32 |
+
"fp32_int8_agreement_pct": 99.57,
|
| 33 |
+
"fp32_val_bal_acc": 97.65,
|
| 34 |
+
"fp32_val_macro_f1": 97.57,
|
| 35 |
+
"int8_val_bal_acc": 97.65,
|
| 36 |
+
"int8_val_macro_f1": 97.57,
|
| 37 |
+
"meets_99_5pct_gate": true
|
| 38 |
},
|
| 39 |
+
"junk": {
|
| 40 |
+
"fp32_int8_agreement_pct": 100.0,
|
| 41 |
+
"meets_99_5pct_gate": true
|
|
|
|
|
|
|
|
|
|
| 42 |
},
|
| 43 |
+
"lang": {
|
| 44 |
+
"bal_acc_drift_pct": 0.0,
|
| 45 |
+
"fp32_int8_agreement_pct": 99.79,
|
| 46 |
+
"fp32_val_bal_acc": 73.68,
|
| 47 |
+
"fp32_val_macro_f1": 74.78,
|
| 48 |
+
"int8_val_bal_acc": 73.68,
|
| 49 |
+
"int8_val_macro_f1": 74.78,
|
| 50 |
+
"meets_99_5pct_gate": true
|
| 51 |
},
|
| 52 |
+
"meets_99_5pct_gate": true,
|
| 53 |
+
"owner_scope": {
|
| 54 |
+
"bal_acc_drift_pct": 0.0,
|
| 55 |
+
"fp32_int8_agreement_pct": 100.0,
|
| 56 |
+
"fp32_val_bal_acc": 96.32,
|
| 57 |
+
"fp32_val_macro_f1": 96.32,
|
| 58 |
+
"int8_val_bal_acc": 96.32,
|
| 59 |
+
"int8_val_macro_f1": 96.32,
|
| 60 |
+
"meets_99_5pct_gate": true
|
| 61 |
+
},
|
| 62 |
+
"span_bio": {
|
| 63 |
+
"fp32_int8_agreement_pct": 99.7,
|
| 64 |
+
"fp32_val_byte_acc": 67.93,
|
| 65 |
+
"int8_val_byte_acc": 67.95
|
| 66 |
+
},
|
| 67 |
+
"title_language": {
|
| 68 |
+
"bal_acc_drift_pct": 0.0,
|
| 69 |
+
"fp32_int8_agreement_pct": 99.86,
|
| 70 |
+
"fp32_val_bal_acc": 83.33,
|
| 71 |
+
"fp32_val_macro_f1": 89.5,
|
| 72 |
+
"int8_val_bal_acc": 83.33,
|
| 73 |
+
"int8_val_macro_f1": 89.5,
|
| 74 |
+
"meets_99_5pct_gate": true
|
| 75 |
}
|
| 76 |
},
|
| 77 |
"menon_logit_adjustment_stop_fit": {
|
| 78 |
+
"D100000": {
|
| 79 |
"tau": 0.0
|
| 80 |
},
|
| 81 |
+
"D80000": {
|
| 82 |
"tau": 0.0
|
| 83 |
},
|
| 84 |
"E100000_TC_v3": {
|
|
|
|
| 100 |
"tau": 0.0
|
| 101 |
}
|
| 102 |
},
|
| 103 |
+
"mondrian_audit": {
|
| 104 |
+
"owner_scope": {
|
| 105 |
+
"alpha": 0.02,
|
| 106 |
+
"method": "split-conformal FPR quantile ceil((n+1)(1-alpha)) on val negatives, 2-fold temperature-scaled probs",
|
| 107 |
+
"n_neg": 625,
|
| 108 |
+
"n_pos": 786,
|
| 109 |
+
"val_fpr_at_thr": 0.0176,
|
| 110 |
+
"val_recall_at_thr": 0.975827
|
| 111 |
+
}
|
| 112 |
+
},
|
| 113 |
"mondrian_conformal_thresholds": {
|
| 114 |
"owner_scope": {
|
| 115 |
"alpha": 0.02,
|
| 116 |
+
"threshold": 0.280814
|
| 117 |
+
}
|
| 118 |
+
},
|
| 119 |
+
"mondrian_thresholds": {
|
| 120 |
+
"owner_scope": 0.280814
|
| 121 |
+
},
|
| 122 |
+
"owner_scope_r12": {
|
| 123 |
+
"counts": {
|
| 124 |
+
"audiobook": 61,
|
| 125 |
+
"comic": 149,
|
| 126 |
+
"course": 22,
|
| 127 |
+
"ebook": 75,
|
| 128 |
+
"game": 157,
|
| 129 |
+
"movie": 377,
|
| 130 |
+
"music": 184,
|
| 131 |
+
"software": 118,
|
| 132 |
+
"tv_show": 267,
|
| 133 |
+
"unknown": 0,
|
| 134 |
+
"xxx": 1
|
| 135 |
+
},
|
| 136 |
+
"excluded_categories": [
|
| 137 |
+
"music",
|
| 138 |
+
"comic",
|
| 139 |
+
"game",
|
| 140 |
+
"audiobook",
|
| 141 |
+
"software",
|
| 142 |
+
"ebook",
|
| 143 |
+
"course"
|
| 144 |
+
],
|
| 145 |
+
"feature_stats_dim": 41,
|
| 146 |
+
"fitting": "two-fold group cross-fit audit; full validation refit",
|
| 147 |
+
"folds": [
|
| 148 |
+
{
|
| 149 |
+
"balanced_error": 0.03196061087255235,
|
| 150 |
+
"counts": {
|
| 151 |
+
"audiobook": 32,
|
| 152 |
+
"comic": 82,
|
| 153 |
+
"course": 13,
|
| 154 |
+
"ebook": 41,
|
| 155 |
+
"game": 79,
|
| 156 |
+
"movie": 172,
|
| 157 |
+
"music": 87,
|
| 158 |
+
"software": 65,
|
| 159 |
+
"tv_show": 133,
|
| 160 |
+
"unknown": 0,
|
| 161 |
+
"xxx": 1
|
| 162 |
+
},
|
| 163 |
+
"heldout_fold": 0,
|
| 164 |
+
"holdout_groups": [
|
| 165 |
+
"r10-comp-00005",
|
| 166 |
+
"r10-comp-00049",
|
| 167 |
+
"r10-comp-00059",
|
| 168 |
+
"r10-comp-00067",
|
| 169 |
+
"r10-comp-00084",
|
| 170 |
+
"r10-comp-00092",
|
| 171 |
+
"r10-comp-00122",
|
| 172 |
+
"r10-comp-00176",
|
| 173 |
+
"r10-comp-00204",
|
| 174 |
+
"r10-comp-00221",
|
| 175 |
+
"r10-comp-00236",
|
| 176 |
+
"r10-comp-00238",
|
| 177 |
+
"r10-comp-00240",
|
| 178 |
+
"r10-comp-00269",
|
| 179 |
+
"r10-comp-00311",
|
| 180 |
+
"r10-comp-00343",
|
| 181 |
+
"r10-comp-00371",
|
| 182 |
+
"r10-comp-00379",
|
| 183 |
+
"r10-comp-00384",
|
| 184 |
+
"r10-comp-00393",
|
| 185 |
+
"r10-comp-00450",
|
| 186 |
+
"r10-comp-00469",
|
| 187 |
+
"r10-comp-00542",
|
| 188 |
+
"r10-comp-00553",
|
| 189 |
+
"r10-comp-00558",
|
| 190 |
+
"r10-comp-00560",
|
| 191 |
+
"r10-comp-00563",
|
| 192 |
+
"r10-comp-00610",
|
| 193 |
+
"r10-comp-00614",
|
| 194 |
+
"r10-comp-00619",
|
| 195 |
+
"r10-comp-00626",
|
| 196 |
+
"r10-comp-00646",
|
| 197 |
+
"r10-comp-00675",
|
| 198 |
+
"r10-comp-00705",
|
| 199 |
+
"r10-comp-00732",
|
| 200 |
+
"r10-comp-00749",
|
| 201 |
+
"r10-comp-00768",
|
| 202 |
+
"r10-comp-00769",
|
| 203 |
+
"r10-comp-00777",
|
| 204 |
+
"r10-comp-00817",
|
| 205 |
+
"r10-comp-00848",
|
| 206 |
+
"r10-comp-00874",
|
| 207 |
+
"r10-comp-00892",
|
| 208 |
+
"r10-comp-01011",
|
| 209 |
+
"r10-comp-01014",
|
| 210 |
+
"r10-comp-01039",
|
| 211 |
+
"r10-comp-01047",
|
| 212 |
+
"r10-comp-01136",
|
| 213 |
+
"r10-comp-01277",
|
| 214 |
+
"r10-comp-01309",
|
| 215 |
+
"r10-comp-01320",
|
| 216 |
+
"r10-comp-01341",
|
| 217 |
+
"r10-comp-01353",
|
| 218 |
+
"r10-comp-01356",
|
| 219 |
+
"r10-comp-01405",
|
| 220 |
+
"r10-comp-01441",
|
| 221 |
+
"r10-comp-01445",
|
| 222 |
+
"r10-comp-01447",
|
| 223 |
+
"r10-comp-01485",
|
| 224 |
+
"r10-comp-01566",
|
| 225 |
+
"r10-comp-01622",
|
| 226 |
+
"r10-comp-01626",
|
| 227 |
+
"r10-comp-01647",
|
| 228 |
+
"r10-comp-01693",
|
| 229 |
+
"r10-comp-01725",
|
| 230 |
+
"r10-comp-01751",
|
| 231 |
+
"r10-comp-01760",
|
| 232 |
+
"r10-comp-01795",
|
| 233 |
+
"r10-comp-01808",
|
| 234 |
+
"r10-comp-01816",
|
| 235 |
+
"r10-comp-01826",
|
| 236 |
+
"r10-comp-01827",
|
| 237 |
+
"r10-comp-01860",
|
| 238 |
+
"r10-comp-01893",
|
| 239 |
+
"r10-comp-01907",
|
| 240 |
+
"r10-comp-01941",
|
| 241 |
+
"r10-comp-01972",
|
| 242 |
+
"r10-comp-01981",
|
| 243 |
+
"r10-comp-01999",
|
| 244 |
+
"r10-comp-02018",
|
| 245 |
+
"r10-comp-02137",
|
| 246 |
+
"r10-comp-02146",
|
| 247 |
+
"r10-comp-02206",
|
| 248 |
+
"r10-comp-02236",
|
| 249 |
+
"r10-comp-02302",
|
| 250 |
+
"r10-comp-02305",
|
| 251 |
+
"r10-comp-02341",
|
| 252 |
+
"r10-comp-02361",
|
| 253 |
+
"r10-comp-02373",
|
| 254 |
+
"r10-comp-02396",
|
| 255 |
+
"r10-comp-02421",
|
| 256 |
+
"r10-comp-02448",
|
| 257 |
+
"r10-comp-02497",
|
| 258 |
+
"r10-comp-02548",
|
| 259 |
+
"r10-comp-02553",
|
| 260 |
+
"r10-comp-02591",
|
| 261 |
+
"r10-comp-02593",
|
| 262 |
+
"r10-comp-02600",
|
| 263 |
+
"r10-comp-02622",
|
| 264 |
+
"r10-comp-02640",
|
| 265 |
+
"r10-comp-02654",
|
| 266 |
+
"r10-comp-02698",
|
| 267 |
+
"r10-comp-02764",
|
| 268 |
+
"r10-comp-02824",
|
| 269 |
+
"r10-comp-02839",
|
| 270 |
+
"r10-comp-02930",
|
| 271 |
+
"r10-comp-02983",
|
| 272 |
+
"r10-comp-02990",
|
| 273 |
+
"r10-comp-03005",
|
| 274 |
+
"r10-comp-03040",
|
| 275 |
+
"r10-comp-03133",
|
| 276 |
+
"r10-comp-03188",
|
| 277 |
+
"r10-comp-03229",
|
| 278 |
+
"r10-comp-03239",
|
| 279 |
+
"r10-comp-03260",
|
| 280 |
+
"r10-comp-03265",
|
| 281 |
+
"r10-comp-03274",
|
| 282 |
+
"r10-comp-03282",
|
| 283 |
+
"r10-comp-03298",
|
| 284 |
+
"r10-comp-03300",
|
| 285 |
+
"r10-comp-03317",
|
| 286 |
+
"r10-comp-03384",
|
| 287 |
+
"r10-comp-03436",
|
| 288 |
+
"r10-comp-03535",
|
| 289 |
+
"r10-comp-03565",
|
| 290 |
+
"r10-comp-03586",
|
| 291 |
+
"r10-comp-03590",
|
| 292 |
+
"r10-comp-03597",
|
| 293 |
+
"r10-comp-03603",
|
| 294 |
+
"r10-comp-03633",
|
| 295 |
+
"r10-comp-03648",
|
| 296 |
+
"r10-comp-03649",
|
| 297 |
+
"r10-comp-03681",
|
| 298 |
+
"r10-comp-03711",
|
| 299 |
+
"r10-comp-03823",
|
| 300 |
+
"r10-comp-03838",
|
| 301 |
+
"r10-comp-03846",
|
| 302 |
+
"r10-comp-03847",
|
| 303 |
+
"r10-comp-03857",
|
| 304 |
+
"r10-comp-03859",
|
| 305 |
+
"r10-comp-03877",
|
| 306 |
+
"r10-comp-03892",
|
| 307 |
+
"r10-comp-03894",
|
| 308 |
+
"r10-comp-03916",
|
| 309 |
+
"r10-comp-03924",
|
| 310 |
+
"r10-comp-03942",
|
| 311 |
+
"r10-comp-03957",
|
| 312 |
+
"r10-comp-03963",
|
| 313 |
+
"r10-comp-04000",
|
| 314 |
+
"r10-comp-04029",
|
| 315 |
+
"r10-comp-04049",
|
| 316 |
+
"r10-comp-04114",
|
| 317 |
+
"r10-comp-04119",
|
| 318 |
+
"r10-comp-04139",
|
| 319 |
+
"r10-comp-04197",
|
| 320 |
+
"r10-comp-04238",
|
| 321 |
+
"r10-comp-04251",
|
| 322 |
+
"r10-comp-04270",
|
| 323 |
+
"r10-comp-04271",
|
| 324 |
+
"r10-comp-04278",
|
| 325 |
+
"r10-comp-04286",
|
| 326 |
+
"r10-comp-04297",
|
| 327 |
+
"r10-comp-04322",
|
| 328 |
+
"r10-comp-04340",
|
| 329 |
+
"r10-comp-04342",
|
| 330 |
+
"r10-comp-04349",
|
| 331 |
+
"r10-comp-04365",
|
| 332 |
+
"r10-comp-04385",
|
| 333 |
+
"r10-comp-04393",
|
| 334 |
+
"r10-comp-04438",
|
| 335 |
+
"r10-comp-04450",
|
| 336 |
+
"r10-comp-04494",
|
| 337 |
+
"r10-comp-04505",
|
| 338 |
+
"r10-comp-04534",
|
| 339 |
+
"r10-comp-04544",
|
| 340 |
+
"r10-comp-04555",
|
| 341 |
+
"r10-comp-04594",
|
| 342 |
+
"r10-comp-04596",
|
| 343 |
+
"r10-comp-04605",
|
| 344 |
+
"r10-comp-04637",
|
| 345 |
+
"r10-comp-04656",
|
| 346 |
+
"r10-comp-04657",
|
| 347 |
+
"r10-comp-04660",
|
| 348 |
+
"r10-comp-04671",
|
| 349 |
+
"r10-comp-04701",
|
| 350 |
+
"r10-comp-04717",
|
| 351 |
+
"r10-comp-04725",
|
| 352 |
+
"r10-comp-04740",
|
| 353 |
+
"r10-comp-04742",
|
| 354 |
+
"r10-comp-04753",
|
| 355 |
+
"r10-comp-04755",
|
| 356 |
+
"r10-comp-04762",
|
| 357 |
+
"r10-comp-04772",
|
| 358 |
+
"r10-comp-04776",
|
| 359 |
+
"r10-comp-04783",
|
| 360 |
+
"r10-comp-04796",
|
| 361 |
+
"r10-comp-04812",
|
| 362 |
+
"r10-comp-04827",
|
| 363 |
+
"r10-comp-04845",
|
| 364 |
+
"r10-comp-04871",
|
| 365 |
+
"r10-comp-04876",
|
| 366 |
+
"r10-comp-04879",
|
| 367 |
+
"r10-comp-04887",
|
| 368 |
+
"r10-comp-04898",
|
| 369 |
+
"r10-comp-04913",
|
| 370 |
+
"r10-comp-04919",
|
| 371 |
+
"r10-comp-04956",
|
| 372 |
+
"r10-comp-04966",
|
| 373 |
+
"r10-comp-04986",
|
| 374 |
+
"r10-comp-04990",
|
| 375 |
+
"r10-comp-05023",
|
| 376 |
+
"r10-comp-05027",
|
| 377 |
+
"r10-comp-05029",
|
| 378 |
+
"r10-comp-05038",
|
| 379 |
+
"r10-comp-05061",
|
| 380 |
+
"r10-comp-05070",
|
| 381 |
+
"r10-comp-05080",
|
| 382 |
+
"r10-comp-05081",
|
| 383 |
+
"r10-comp-05083",
|
| 384 |
+
"r10-comp-05099",
|
| 385 |
+
"r10-comp-05106",
|
| 386 |
+
"r10-comp-05111",
|
| 387 |
+
"r10-comp-05151",
|
| 388 |
+
"r10-comp-05157",
|
| 389 |
+
"r10-comp-05162",
|
| 390 |
+
"r10-comp-05168",
|
| 391 |
+
"r10-comp-05178",
|
| 392 |
+
"r10-comp-05218",
|
| 393 |
+
"r10-comp-05224",
|
| 394 |
+
"r10-comp-05234",
|
| 395 |
+
"r10-comp-05276",
|
| 396 |
+
"r10-comp-05309",
|
| 397 |
+
"r10-comp-05310",
|
| 398 |
+
"r10-comp-05316",
|
| 399 |
+
"r10-comp-05337",
|
| 400 |
+
"r10-comp-05349",
|
| 401 |
+
"r10-comp-05350",
|
| 402 |
+
"r10-comp-05352",
|
| 403 |
+
"r10-comp-05356",
|
| 404 |
+
"r10-comp-05366",
|
| 405 |
+
"r10-comp-05369",
|
| 406 |
+
"r10-comp-05370",
|
| 407 |
+
"r10-comp-05381",
|
| 408 |
+
"r10-comp-05383",
|
| 409 |
+
"r10-comp-05384",
|
| 410 |
+
"r10-comp-05389",
|
| 411 |
+
"r10-comp-05390",
|
| 412 |
+
"r10-comp-05427",
|
| 413 |
+
"r10-comp-05440",
|
| 414 |
+
"r10-comp-05452",
|
| 415 |
+
"r10-comp-05465",
|
| 416 |
+
"r10-comp-05466",
|
| 417 |
+
"r10-comp-05481",
|
| 418 |
+
"r10-comp-05486",
|
| 419 |
+
"r10-comp-05490",
|
| 420 |
+
"r10-comp-05539",
|
| 421 |
+
"r10-comp-05542",
|
| 422 |
+
"r10-comp-05545",
|
| 423 |
+
"r10-comp-05547",
|
| 424 |
+
"r10-comp-05553",
|
| 425 |
+
"r10-comp-05606",
|
| 426 |
+
"r10-comp-05621",
|
| 427 |
+
"r10-comp-05630",
|
| 428 |
+
"r10-comp-05653",
|
| 429 |
+
"r10-comp-05668",
|
| 430 |
+
"r10-comp-05682",
|
| 431 |
+
"r10-comp-05685",
|
| 432 |
+
"r10-comp-05688",
|
| 433 |
+
"r10-comp-05691",
|
| 434 |
+
"r10-comp-05711",
|
| 435 |
+
"r10-comp-05723",
|
| 436 |
+
"r10-comp-05725",
|
| 437 |
+
"r10-comp-05737",
|
| 438 |
+
"r10-comp-05744",
|
| 439 |
+
"r10-comp-05751",
|
| 440 |
+
"r10-comp-05755",
|
| 441 |
+
"r10-comp-05783",
|
| 442 |
+
"r10-comp-05789",
|
| 443 |
+
"r10-comp-05802",
|
| 444 |
+
"r10-comp-05849",
|
| 445 |
+
"r10-comp-05877",
|
| 446 |
+
"r10-comp-05880",
|
| 447 |
+
"r10-comp-05892",
|
| 448 |
+
"r10-comp-05908",
|
| 449 |
+
"r10-comp-05928",
|
| 450 |
+
"r10-comp-05957",
|
| 451 |
+
"r10-comp-05958",
|
| 452 |
+
"r10-comp-05980",
|
| 453 |
+
"r10-comp-06003",
|
| 454 |
+
"r10-comp-06019",
|
| 455 |
+
"r10-comp-06024",
|
| 456 |
+
"r10-comp-06032",
|
| 457 |
+
"r10-comp-06051",
|
| 458 |
+
"r10-comp-06057",
|
| 459 |
+
"r10-comp-06079",
|
| 460 |
+
"r10-comp-06089",
|
| 461 |
+
"r10-comp-06109",
|
| 462 |
+
"r10-comp-06139",
|
| 463 |
+
"r10-comp-06140",
|
| 464 |
+
"r10-comp-06151",
|
| 465 |
+
"r10-comp-06194",
|
| 466 |
+
"r10-comp-06196",
|
| 467 |
+
"r10-comp-06216",
|
| 468 |
+
"r10-comp-06230",
|
| 469 |
+
"r10-comp-06231",
|
| 470 |
+
"r10-comp-06286",
|
| 471 |
+
"r10-comp-06293",
|
| 472 |
+
"r10-comp-06306",
|
| 473 |
+
"r10-comp-06330",
|
| 474 |
+
"r10-comp-06354",
|
| 475 |
+
"r10-comp-06366",
|
| 476 |
+
"r10-comp-06391",
|
| 477 |
+
"r10-comp-06412",
|
| 478 |
+
"r10-comp-06415",
|
| 479 |
+
"r10-comp-06420",
|
| 480 |
+
"r10-comp-06423",
|
| 481 |
+
"r10-comp-06451",
|
| 482 |
+
"r10-comp-06457",
|
| 483 |
+
"r10-comp-06467",
|
| 484 |
+
"r10-comp-06498",
|
| 485 |
+
"r10-comp-06505",
|
| 486 |
+
"r10-comp-06523",
|
| 487 |
+
"r10-comp-06524",
|
| 488 |
+
"r10-comp-06545",
|
| 489 |
+
"r10-comp-06579",
|
| 490 |
+
"r10-comp-06590",
|
| 491 |
+
"r10-comp-06595",
|
| 492 |
+
"r10-comp-06638",
|
| 493 |
+
"r10-comp-06654",
|
| 494 |
+
"r10-comp-06676",
|
| 495 |
+
"r10-comp-06691",
|
| 496 |
+
"r10-comp-06702",
|
| 497 |
+
"r10-comp-06711",
|
| 498 |
+
"r10-comp-06721",
|
| 499 |
+
"r10-comp-06744",
|
| 500 |
+
"r10-comp-06779",
|
| 501 |
+
"r10-comp-06785",
|
| 502 |
+
"r10-comp-06802",
|
| 503 |
+
"r10-comp-06815",
|
| 504 |
+
"r10-comp-06843",
|
| 505 |
+
"r10-comp-06863",
|
| 506 |
+
"r10-comp-06874",
|
| 507 |
+
"r10-comp-06885",
|
| 508 |
+
"r10-comp-06886",
|
| 509 |
+
"r10-comp-06911",
|
| 510 |
+
"r10-comp-06918",
|
| 511 |
+
"r10-comp-06927",
|
| 512 |
+
"r10-comp-06930",
|
| 513 |
+
"r10-comp-06931",
|
| 514 |
+
"r10-comp-06943",
|
| 515 |
+
"r10-comp-06959",
|
| 516 |
+
"r10-comp-06962",
|
| 517 |
+
"r10-comp-06971",
|
| 518 |
+
"r10-comp-06995",
|
| 519 |
+
"r10-comp-07009",
|
| 520 |
+
"r10-comp-07010",
|
| 521 |
+
"r10-comp-07014",
|
| 522 |
+
"r10-comp-07017",
|
| 523 |
+
"r10-comp-07018",
|
| 524 |
+
"r10-comp-07024",
|
| 525 |
+
"r10-comp-07025",
|
| 526 |
+
"r10-comp-07037",
|
| 527 |
+
"r10-comp-07069",
|
| 528 |
+
"r10-comp-07070",
|
| 529 |
+
"r10-comp-07076",
|
| 530 |
+
"r10-comp-07092",
|
| 531 |
+
"r10-comp-07097",
|
| 532 |
+
"r10-comp-07105",
|
| 533 |
+
"r10-comp-07110",
|
| 534 |
+
"r10-comp-07123",
|
| 535 |
+
"r10-comp-07127",
|
| 536 |
+
"r10-comp-07148",
|
| 537 |
+
"r10-comp-07153",
|
| 538 |
+
"r10-comp-07196",
|
| 539 |
+
"r10-comp-07224",
|
| 540 |
+
"r10-comp-07229",
|
| 541 |
+
"r10-comp-07239",
|
| 542 |
+
"r10-comp-07256",
|
| 543 |
+
"r10-comp-07268",
|
| 544 |
+
"r10-comp-07271",
|
| 545 |
+
"r10-comp-07284",
|
| 546 |
+
"r10-comp-07288",
|
| 547 |
+
"r10-comp-07293",
|
| 548 |
+
"r10-comp-07308",
|
| 549 |
+
"r10-comp-07313",
|
| 550 |
+
"r10-comp-07334",
|
| 551 |
+
"r10-comp-07347",
|
| 552 |
+
"r10-comp-07353",
|
| 553 |
+
"r10-comp-07365",
|
| 554 |
+
"r10-comp-07377",
|
| 555 |
+
"r10-comp-07387",
|
| 556 |
+
"r10-comp-07391",
|
| 557 |
+
"r10-comp-07404",
|
| 558 |
+
"r10-comp-07408",
|
| 559 |
+
"r10-comp-07410",
|
| 560 |
+
"r10-comp-07420",
|
| 561 |
+
"r10-comp-07423",
|
| 562 |
+
"r10-comp-07435",
|
| 563 |
+
"r10-comp-07441",
|
| 564 |
+
"r10-comp-07451",
|
| 565 |
+
"r10-comp-07453",
|
| 566 |
+
"r10-comp-07458",
|
| 567 |
+
"r10-comp-07466",
|
| 568 |
+
"r10-comp-07475",
|
| 569 |
+
"r10-comp-07487",
|
| 570 |
+
"r10-comp-07511",
|
| 571 |
+
"r10-comp-07517",
|
| 572 |
+
"r10-comp-07522",
|
| 573 |
+
"r10-comp-07530",
|
| 574 |
+
"r10-comp-07531",
|
| 575 |
+
"r10-comp-07533",
|
| 576 |
+
"r10-comp-07537",
|
| 577 |
+
"r10-comp-07539",
|
| 578 |
+
"r10-comp-07540",
|
| 579 |
+
"r10-comp-07548",
|
| 580 |
+
"r10-comp-07579",
|
| 581 |
+
"r10-comp-07585",
|
| 582 |
+
"r10-comp-07631",
|
| 583 |
+
"r10-comp-07641",
|
| 584 |
+
"r10-comp-07654",
|
| 585 |
+
"r10-comp-07657",
|
| 586 |
+
"r10-comp-07658",
|
| 587 |
+
"r10-comp-07660",
|
| 588 |
+
"r10-comp-07669",
|
| 589 |
+
"r10-comp-07674",
|
| 590 |
+
"r10-comp-07687",
|
| 591 |
+
"r10-comp-07750",
|
| 592 |
+
"r10-comp-07755",
|
| 593 |
+
"r10-comp-07767",
|
| 594 |
+
"r10-comp-07785",
|
| 595 |
+
"r10-comp-07786",
|
| 596 |
+
"r10-comp-07789",
|
| 597 |
+
"r10-comp-07801",
|
| 598 |
+
"r10-comp-07819",
|
| 599 |
+
"r10-comp-07821",
|
| 600 |
+
"r10-comp-07824",
|
| 601 |
+
"r10-comp-07827",
|
| 602 |
+
"r10-comp-07869",
|
| 603 |
+
"r10-comp-07884",
|
| 604 |
+
"r10-comp-07886",
|
| 605 |
+
"r10-comp-07887",
|
| 606 |
+
"r10-comp-07889",
|
| 607 |
+
"r10-comp-07895",
|
| 608 |
+
"r10-comp-07914",
|
| 609 |
+
"r10-comp-07933",
|
| 610 |
+
"r10-comp-07944",
|
| 611 |
+
"r10-comp-07972",
|
| 612 |
+
"r10-comp-07979",
|
| 613 |
+
"r10-comp-08007",
|
| 614 |
+
"r10-comp-08019",
|
| 615 |
+
"r10-comp-08026",
|
| 616 |
+
"r10-comp-08042",
|
| 617 |
+
"r10-comp-08048",
|
| 618 |
+
"r10-comp-08049",
|
| 619 |
+
"r10-comp-08050",
|
| 620 |
+
"r10-comp-08057",
|
| 621 |
+
"r10-comp-08068",
|
| 622 |
+
"r10-comp-08069",
|
| 623 |
+
"r10-comp-08070",
|
| 624 |
+
"r10-comp-08082",
|
| 625 |
+
"r10-comp-08087",
|
| 626 |
+
"r10-comp-08088",
|
| 627 |
+
"r10-comp-08117",
|
| 628 |
+
"r10-comp-08124",
|
| 629 |
+
"r10-comp-08126",
|
| 630 |
+
"r10-comp-08129",
|
| 631 |
+
"r10-comp-08132",
|
| 632 |
+
"r10-comp-08148",
|
| 633 |
+
"r10-comp-08163",
|
| 634 |
+
"r10-comp-08171",
|
| 635 |
+
"r10-comp-08189",
|
| 636 |
+
"r10-comp-08190",
|
| 637 |
+
"r10-comp-08196",
|
| 638 |
+
"r10-comp-08202",
|
| 639 |
+
"r10-comp-08212",
|
| 640 |
+
"r10-comp-08248",
|
| 641 |
+
"r10-comp-08253",
|
| 642 |
+
"r10-comp-08254",
|
| 643 |
+
"r10-comp-08257",
|
| 644 |
+
"r10-comp-08263",
|
| 645 |
+
"r10-comp-08274",
|
| 646 |
+
"r10-comp-08293",
|
| 647 |
+
"r10-comp-08294",
|
| 648 |
+
"r10-comp-08298",
|
| 649 |
+
"r10-comp-08301",
|
| 650 |
+
"r10-comp-08302",
|
| 651 |
+
"r10-comp-08333",
|
| 652 |
+
"r10-comp-08337",
|
| 653 |
+
"r10-comp-08356",
|
| 654 |
+
"r10-comp-08367",
|
| 655 |
+
"r10-comp-08378",
|
| 656 |
+
"r10-comp-08379",
|
| 657 |
+
"r10-comp-08383",
|
| 658 |
+
"r10-comp-08400",
|
| 659 |
+
"r10-comp-08408",
|
| 660 |
+
"r10-comp-08419",
|
| 661 |
+
"r10-comp-08433",
|
| 662 |
+
"r10-comp-08470",
|
| 663 |
+
"r10-comp-08477",
|
| 664 |
+
"r10-comp-08490",
|
| 665 |
+
"r10-comp-08493",
|
| 666 |
+
"r10-comp-08507",
|
| 667 |
+
"r10-comp-08515",
|
| 668 |
+
"r10-comp-08530",
|
| 669 |
+
"r10-comp-08538",
|
| 670 |
+
"r10-comp-08539",
|
| 671 |
+
"r10-comp-08556",
|
| 672 |
+
"r10-comp-08569",
|
| 673 |
+
"r10-comp-08590",
|
| 674 |
+
"r10-comp-08598",
|
| 675 |
+
"r10-comp-08634",
|
| 676 |
+
"r10-comp-08657",
|
| 677 |
+
"r10-comp-08674",
|
| 678 |
+
"r10-comp-08684",
|
| 679 |
+
"r10-comp-08686",
|
| 680 |
+
"r10-comp-08690",
|
| 681 |
+
"r10-comp-08695",
|
| 682 |
+
"r10-comp-08702",
|
| 683 |
+
"r10-comp-08706",
|
| 684 |
+
"r10-comp-08721",
|
| 685 |
+
"r10-comp-08742",
|
| 686 |
+
"r10-comp-08776",
|
| 687 |
+
"r10-comp-08795",
|
| 688 |
+
"r10-comp-08803",
|
| 689 |
+
"r10-comp-08806",
|
| 690 |
+
"r10-comp-08816",
|
| 691 |
+
"r10-comp-08817",
|
| 692 |
+
"r10-comp-08824",
|
| 693 |
+
"r10-comp-08835",
|
| 694 |
+
"r10-comp-08841",
|
| 695 |
+
"r10-comp-08843",
|
| 696 |
+
"r10-comp-08855",
|
| 697 |
+
"r10-comp-08870",
|
| 698 |
+
"r10-comp-08893",
|
| 699 |
+
"r10-comp-08909",
|
| 700 |
+
"r10-comp-08926",
|
| 701 |
+
"r10-comp-08932",
|
| 702 |
+
"r10-comp-08941",
|
| 703 |
+
"r10-comp-08946",
|
| 704 |
+
"r10-comp-08956",
|
| 705 |
+
"r10-comp-08972",
|
| 706 |
+
"r10-comp-08985",
|
| 707 |
+
"r10-comp-09006",
|
| 708 |
+
"r10-comp-09031",
|
| 709 |
+
"r10-comp-09041",
|
| 710 |
+
"r10-comp-09044",
|
| 711 |
+
"r10-comp-09051",
|
| 712 |
+
"r10-comp-09063",
|
| 713 |
+
"r10-comp-09066",
|
| 714 |
+
"r10-comp-09085",
|
| 715 |
+
"r10-comp-09086",
|
| 716 |
+
"r10-comp-09087",
|
| 717 |
+
"r10-comp-09092",
|
| 718 |
+
"r10-comp-09100",
|
| 719 |
+
"r10-comp-09137",
|
| 720 |
+
"r10-comp-09187",
|
| 721 |
+
"r10-comp-09189",
|
| 722 |
+
"r10-comp-09192",
|
| 723 |
+
"r10-comp-09196",
|
| 724 |
+
"r10-comp-09199",
|
| 725 |
+
"r10-comp-09207",
|
| 726 |
+
"r10-comp-09210",
|
| 727 |
+
"r10-comp-09222",
|
| 728 |
+
"r10-comp-09229",
|
| 729 |
+
"r10-comp-09240",
|
| 730 |
+
"r10-comp-09242",
|
| 731 |
+
"r10-comp-09249",
|
| 732 |
+
"r10-comp-09255",
|
| 733 |
+
"r10-comp-09258",
|
| 734 |
+
"r10-comp-09272",
|
| 735 |
+
"r10-comp-09273",
|
| 736 |
+
"r10-comp-09291",
|
| 737 |
+
"r10-comp-09292",
|
| 738 |
+
"r10-comp-09326",
|
| 739 |
+
"r10-comp-09329"
|
| 740 |
+
],
|
| 741 |
+
"n_holdout": 706,
|
| 742 |
+
"n_train": 705,
|
| 743 |
+
"raw_thresholds": {
|
| 744 |
+
"audiobook": 0.364,
|
| 745 |
+
"comic": 0.364,
|
| 746 |
+
"course": 0.364,
|
| 747 |
+
"ebook": 0.364,
|
| 748 |
+
"game": 0.364,
|
| 749 |
+
"movie": 0.364,
|
| 750 |
+
"music": 0.364,
|
| 751 |
+
"software": 0.364,
|
| 752 |
+
"tv_show": 0.364,
|
| 753 |
+
"unknown": 0.364,
|
| 754 |
+
"xxx": 0.364
|
| 755 |
+
},
|
| 756 |
+
"tau_global": 0.364,
|
| 757 |
+
"thresholds": {
|
| 758 |
+
"audiobook": 0.364,
|
| 759 |
+
"comic": 0.36399999999999993,
|
| 760 |
+
"course": 0.364,
|
| 761 |
+
"ebook": 0.364,
|
| 762 |
+
"game": 0.364,
|
| 763 |
+
"movie": 0.364,
|
| 764 |
+
"music": 0.364,
|
| 765 |
+
"software": 0.364,
|
| 766 |
+
"tv_show": 0.364,
|
| 767 |
+
"unknown": 0.364,
|
| 768 |
+
"xxx": 0.364
|
| 769 |
+
},
|
| 770 |
+
"train_groups": [
|
| 771 |
+
"r10-comp-00018",
|
| 772 |
+
"r10-comp-00028",
|
| 773 |
+
"r10-comp-00030",
|
| 774 |
+
"r10-comp-00037",
|
| 775 |
+
"r10-comp-00044",
|
| 776 |
+
"r10-comp-00054",
|
| 777 |
+
"r10-comp-00068",
|
| 778 |
+
"r10-comp-00069",
|
| 779 |
+
"r10-comp-00087",
|
| 780 |
+
"r10-comp-00099",
|
| 781 |
+
"r10-comp-00102",
|
| 782 |
+
"r10-comp-00127",
|
| 783 |
+
"r10-comp-00134",
|
| 784 |
+
"r10-comp-00137",
|
| 785 |
+
"r10-comp-00144",
|
| 786 |
+
"r10-comp-00146",
|
| 787 |
+
"r10-comp-00169",
|
| 788 |
+
"r10-comp-00189",
|
| 789 |
+
"r10-comp-00207",
|
| 790 |
+
"r10-comp-00208",
|
| 791 |
+
"r10-comp-00213",
|
| 792 |
+
"r10-comp-00233",
|
| 793 |
+
"r10-comp-00268",
|
| 794 |
+
"r10-comp-00280",
|
| 795 |
+
"r10-comp-00315",
|
| 796 |
+
"r10-comp-00320",
|
| 797 |
+
"r10-comp-00335",
|
| 798 |
+
"r10-comp-00341",
|
| 799 |
+
"r10-comp-00344",
|
| 800 |
+
"r10-comp-00367",
|
| 801 |
+
"r10-comp-00387",
|
| 802 |
+
"r10-comp-00396",
|
| 803 |
+
"r10-comp-00399",
|
| 804 |
+
"r10-comp-00405",
|
| 805 |
+
"r10-comp-00409",
|
| 806 |
+
"r10-comp-00429",
|
| 807 |
+
"r10-comp-00431",
|
| 808 |
+
"r10-comp-00433",
|
| 809 |
+
"r10-comp-00464",
|
| 810 |
+
"r10-comp-00473",
|
| 811 |
+
"r10-comp-00477",
|
| 812 |
+
"r10-comp-00491",
|
| 813 |
+
"r10-comp-00533",
|
| 814 |
+
"r10-comp-00543",
|
| 815 |
+
"r10-comp-00549",
|
| 816 |
+
"r10-comp-00559",
|
| 817 |
+
"r10-comp-00564",
|
| 818 |
+
"r10-comp-00568",
|
| 819 |
+
"r10-comp-00605",
|
| 820 |
+
"r10-comp-00620",
|
| 821 |
+
"r10-comp-00684",
|
| 822 |
+
"r10-comp-00703",
|
| 823 |
+
"r10-comp-00711",
|
| 824 |
+
"r10-comp-00723",
|
| 825 |
+
"r10-comp-00745",
|
| 826 |
+
"r10-comp-00776",
|
| 827 |
+
"r10-comp-00783",
|
| 828 |
+
"r10-comp-00802",
|
| 829 |
+
"r10-comp-00811",
|
| 830 |
+
"r10-comp-00838",
|
| 831 |
+
"r10-comp-00887",
|
| 832 |
+
"r10-comp-00936",
|
| 833 |
+
"r10-comp-00950",
|
| 834 |
+
"r10-comp-00970",
|
| 835 |
+
"r10-comp-00990",
|
| 836 |
+
"r10-comp-00992",
|
| 837 |
+
"r10-comp-01000",
|
| 838 |
+
"r10-comp-01009",
|
| 839 |
+
"r10-comp-01019",
|
| 840 |
+
"r10-comp-01025",
|
| 841 |
+
"r10-comp-01029",
|
| 842 |
+
"r10-comp-01106",
|
| 843 |
+
"r10-comp-01114",
|
| 844 |
+
"r10-comp-01123",
|
| 845 |
+
"r10-comp-01141",
|
| 846 |
+
"r10-comp-01147",
|
| 847 |
+
"r10-comp-01154",
|
| 848 |
+
"r10-comp-01190",
|
| 849 |
+
"r10-comp-01307",
|
| 850 |
+
"r10-comp-01338",
|
| 851 |
+
"r10-comp-01429",
|
| 852 |
+
"r10-comp-01440",
|
| 853 |
+
"r10-comp-01483",
|
| 854 |
+
"r10-comp-01556",
|
| 855 |
+
"r10-comp-01559",
|
| 856 |
+
"r10-comp-01595",
|
| 857 |
+
"r10-comp-01601",
|
| 858 |
+
"r10-comp-01637",
|
| 859 |
+
"r10-comp-01667",
|
| 860 |
+
"r10-comp-01675",
|
| 861 |
+
"r10-comp-01735",
|
| 862 |
+
"r10-comp-01756",
|
| 863 |
+
"r10-comp-01831",
|
| 864 |
+
"r10-comp-01926",
|
| 865 |
+
"r10-comp-01949",
|
| 866 |
+
"r10-comp-01961",
|
| 867 |
+
"r10-comp-01963",
|
| 868 |
+
"r10-comp-02022",
|
| 869 |
+
"r10-comp-02069",
|
| 870 |
+
"r10-comp-02111",
|
| 871 |
+
"r10-comp-02112",
|
| 872 |
+
"r10-comp-02196",
|
| 873 |
+
"r10-comp-02198",
|
| 874 |
+
"r10-comp-02222",
|
| 875 |
+
"r10-comp-02289",
|
| 876 |
+
"r10-comp-02292",
|
| 877 |
+
"r10-comp-02310",
|
| 878 |
+
"r10-comp-02336",
|
| 879 |
+
"r10-comp-02411",
|
| 880 |
+
"r10-comp-02494",
|
| 881 |
+
"r10-comp-02557",
|
| 882 |
+
"r10-comp-02561",
|
| 883 |
+
"r10-comp-02576",
|
| 884 |
+
"r10-comp-02602",
|
| 885 |
+
"r10-comp-02699",
|
| 886 |
+
"r10-comp-02760",
|
| 887 |
+
"r10-comp-02794",
|
| 888 |
+
"r10-comp-02875",
|
| 889 |
+
"r10-comp-02880",
|
| 890 |
+
"r10-comp-02999",
|
| 891 |
+
"r10-comp-03234",
|
| 892 |
+
"r10-comp-03241",
|
| 893 |
+
"r10-comp-03251",
|
| 894 |
+
"r10-comp-03253",
|
| 895 |
+
"r10-comp-03267",
|
| 896 |
+
"r10-comp-03269",
|
| 897 |
+
"r10-comp-03272",
|
| 898 |
+
"r10-comp-03304",
|
| 899 |
+
"r10-comp-03318",
|
| 900 |
+
"r10-comp-03328",
|
| 901 |
+
"r10-comp-03348",
|
| 902 |
+
"r10-comp-03446",
|
| 903 |
+
"r10-comp-03453",
|
| 904 |
+
"r10-comp-03457",
|
| 905 |
+
"r10-comp-03493",
|
| 906 |
+
"r10-comp-03526",
|
| 907 |
+
"r10-comp-03538",
|
| 908 |
+
"r10-comp-03570",
|
| 909 |
+
"r10-comp-03585",
|
| 910 |
+
"r10-comp-03612",
|
| 911 |
+
"r10-comp-03635",
|
| 912 |
+
"r10-comp-03754",
|
| 913 |
+
"r10-comp-03766",
|
| 914 |
+
"r10-comp-03858",
|
| 915 |
+
"r10-comp-03881",
|
| 916 |
+
"r10-comp-03884",
|
| 917 |
+
"r10-comp-03885",
|
| 918 |
+
"r10-comp-03886",
|
| 919 |
+
"r10-comp-03887",
|
| 920 |
+
"r10-comp-03915",
|
| 921 |
+
"r10-comp-03929",
|
| 922 |
+
"r10-comp-03948",
|
| 923 |
+
"r10-comp-03950",
|
| 924 |
+
"r10-comp-03959",
|
| 925 |
+
"r10-comp-03993",
|
| 926 |
+
"r10-comp-03996",
|
| 927 |
+
"r10-comp-04016",
|
| 928 |
+
"r10-comp-04023",
|
| 929 |
+
"r10-comp-04025",
|
| 930 |
+
"r10-comp-04033",
|
| 931 |
+
"r10-comp-04034",
|
| 932 |
+
"r10-comp-04036",
|
| 933 |
+
"r10-comp-04047",
|
| 934 |
+
"r10-comp-04056",
|
| 935 |
+
"r10-comp-04073",
|
| 936 |
+
"r10-comp-04084",
|
| 937 |
+
"r10-comp-04088",
|
| 938 |
+
"r10-comp-04101",
|
| 939 |
+
"r10-comp-04108",
|
| 940 |
+
"r10-comp-04110",
|
| 941 |
+
"r10-comp-04120",
|
| 942 |
+
"r10-comp-04124",
|
| 943 |
+
"r10-comp-04130",
|
| 944 |
+
"r10-comp-04136",
|
| 945 |
+
"r10-comp-04151",
|
| 946 |
+
"r10-comp-04179",
|
| 947 |
+
"r10-comp-04182",
|
| 948 |
+
"r10-comp-04210",
|
| 949 |
+
"r10-comp-04211",
|
| 950 |
+
"r10-comp-04253",
|
| 951 |
+
"r10-comp-04287",
|
| 952 |
+
"r10-comp-04289",
|
| 953 |
+
"r10-comp-04292",
|
| 954 |
+
"r10-comp-04318",
|
| 955 |
+
"r10-comp-04330",
|
| 956 |
+
"r10-comp-04344",
|
| 957 |
+
"r10-comp-04396",
|
| 958 |
+
"r10-comp-04397",
|
| 959 |
+
"r10-comp-04442",
|
| 960 |
+
"r10-comp-04445",
|
| 961 |
+
"r10-comp-04476",
|
| 962 |
+
"r10-comp-04478",
|
| 963 |
+
"r10-comp-04479",
|
| 964 |
+
"r10-comp-04485",
|
| 965 |
+
"r10-comp-04521",
|
| 966 |
+
"r10-comp-04531",
|
| 967 |
+
"r10-comp-04532",
|
| 968 |
+
"r10-comp-04540",
|
| 969 |
+
"r10-comp-04552",
|
| 970 |
+
"r10-comp-04563",
|
| 971 |
+
"r10-comp-04570",
|
| 972 |
+
"r10-comp-04682",
|
| 973 |
+
"r10-comp-04704",
|
| 974 |
+
"r10-comp-04706",
|
| 975 |
+
"r10-comp-04709",
|
| 976 |
+
"r10-comp-04714",
|
| 977 |
+
"r10-comp-04716",
|
| 978 |
+
"r10-comp-04721",
|
| 979 |
+
"r10-comp-04738",
|
| 980 |
+
"r10-comp-04741",
|
| 981 |
+
"r10-comp-04798",
|
| 982 |
+
"r10-comp-04826",
|
| 983 |
+
"r10-comp-04834",
|
| 984 |
+
"r10-comp-04855",
|
| 985 |
+
"r10-comp-04863",
|
| 986 |
+
"r10-comp-04874",
|
| 987 |
+
"r10-comp-04883",
|
| 988 |
+
"r10-comp-04893",
|
| 989 |
+
"r10-comp-04901",
|
| 990 |
+
"r10-comp-04911",
|
| 991 |
+
"r10-comp-04912",
|
| 992 |
+
"r10-comp-04918",
|
| 993 |
+
"r10-comp-04935",
|
| 994 |
+
"r10-comp-04962",
|
| 995 |
+
"r10-comp-04971",
|
| 996 |
+
"r10-comp-04995",
|
| 997 |
+
"r10-comp-05006",
|
| 998 |
+
"r10-comp-05010",
|
| 999 |
+
"r10-comp-05014",
|
| 1000 |
+
"r10-comp-05017",
|
| 1001 |
+
"r10-comp-05026",
|
| 1002 |
+
"r10-comp-05075",
|
| 1003 |
+
"r10-comp-05079",
|
| 1004 |
+
"r10-comp-05124",
|
| 1005 |
+
"r10-comp-05126",
|
| 1006 |
+
"r10-comp-05140",
|
| 1007 |
+
"r10-comp-05147",
|
| 1008 |
+
"r10-comp-05156",
|
| 1009 |
+
"r10-comp-05163",
|
| 1010 |
+
"r10-comp-05203",
|
| 1011 |
+
"r10-comp-05205",
|
| 1012 |
+
"r10-comp-05206",
|
| 1013 |
+
"r10-comp-05216",
|
| 1014 |
+
"r10-comp-05237",
|
| 1015 |
+
"r10-comp-05308",
|
| 1016 |
+
"r10-comp-05334",
|
| 1017 |
+
"r10-comp-05336",
|
| 1018 |
+
"r10-comp-05346",
|
| 1019 |
+
"r10-comp-05357",
|
| 1020 |
+
"r10-comp-05382",
|
| 1021 |
+
"r10-comp-05404",
|
| 1022 |
+
"r10-comp-05438",
|
| 1023 |
+
"r10-comp-05444",
|
| 1024 |
+
"r10-comp-05457",
|
| 1025 |
+
"r10-comp-05503",
|
| 1026 |
+
"r10-comp-05511",
|
| 1027 |
+
"r10-comp-05529",
|
| 1028 |
+
"r10-comp-05532",
|
| 1029 |
+
"r10-comp-05537",
|
| 1030 |
+
"r10-comp-05597",
|
| 1031 |
+
"r10-comp-05615",
|
| 1032 |
+
"r10-comp-05639",
|
| 1033 |
+
"r10-comp-05641",
|
| 1034 |
+
"r10-comp-05680",
|
| 1035 |
+
"r10-comp-05718",
|
| 1036 |
+
"r10-comp-05724",
|
| 1037 |
+
"r10-comp-05727",
|
| 1038 |
+
"r10-comp-05728",
|
| 1039 |
+
"r10-comp-05768",
|
| 1040 |
+
"r10-comp-05824",
|
| 1041 |
+
"r10-comp-05825",
|
| 1042 |
+
"r10-comp-05830",
|
| 1043 |
+
"r10-comp-05834",
|
| 1044 |
+
"r10-comp-05851",
|
| 1045 |
+
"r10-comp-05867",
|
| 1046 |
+
"r10-comp-05926",
|
| 1047 |
+
"r10-comp-05951",
|
| 1048 |
+
"r10-comp-05973",
|
| 1049 |
+
"r10-comp-05991",
|
| 1050 |
+
"r10-comp-05993",
|
| 1051 |
+
"r10-comp-06007",
|
| 1052 |
+
"r10-comp-06011",
|
| 1053 |
+
"r10-comp-06015",
|
| 1054 |
+
"r10-comp-06020",
|
| 1055 |
+
"r10-comp-06062",
|
| 1056 |
+
"r10-comp-06065",
|
| 1057 |
+
"r10-comp-06077",
|
| 1058 |
+
"r10-comp-06083",
|
| 1059 |
+
"r10-comp-06091",
|
| 1060 |
+
"r10-comp-06098",
|
| 1061 |
+
"r10-comp-06146",
|
| 1062 |
+
"r10-comp-06156",
|
| 1063 |
+
"r10-comp-06158",
|
| 1064 |
+
"r10-comp-06165",
|
| 1065 |
+
"r10-comp-06172",
|
| 1066 |
+
"r10-comp-06188",
|
| 1067 |
+
"r10-comp-06203",
|
| 1068 |
+
"r10-comp-06206",
|
| 1069 |
+
"r10-comp-06208",
|
| 1070 |
+
"r10-comp-06209",
|
| 1071 |
+
"r10-comp-06259",
|
| 1072 |
+
"r10-comp-06275",
|
| 1073 |
+
"r10-comp-06276",
|
| 1074 |
+
"r10-comp-06295",
|
| 1075 |
+
"r10-comp-06303",
|
| 1076 |
+
"r10-comp-06310",
|
| 1077 |
+
"r10-comp-06319",
|
| 1078 |
+
"r10-comp-06344",
|
| 1079 |
+
"r10-comp-06367",
|
| 1080 |
+
"r10-comp-06404",
|
| 1081 |
+
"r10-comp-06414",
|
| 1082 |
+
"r10-comp-06434",
|
| 1083 |
+
"r10-comp-06441",
|
| 1084 |
+
"r10-comp-06444",
|
| 1085 |
+
"r10-comp-06460",
|
| 1086 |
+
"r10-comp-06463",
|
| 1087 |
+
"r10-comp-06486",
|
| 1088 |
+
"r10-comp-06507",
|
| 1089 |
+
"r10-comp-06520",
|
| 1090 |
+
"r10-comp-06527",
|
| 1091 |
+
"r10-comp-06528",
|
| 1092 |
+
"r10-comp-06536",
|
| 1093 |
+
"r10-comp-06540",
|
| 1094 |
+
"r10-comp-06550",
|
| 1095 |
+
"r10-comp-06556",
|
| 1096 |
+
"r10-comp-06558",
|
| 1097 |
+
"r10-comp-06586",
|
| 1098 |
+
"r10-comp-06594",
|
| 1099 |
+
"r10-comp-06620",
|
| 1100 |
+
"r10-comp-06626",
|
| 1101 |
+
"r10-comp-06627",
|
| 1102 |
+
"r10-comp-06652",
|
| 1103 |
+
"r10-comp-06656",
|
| 1104 |
+
"r10-comp-06663",
|
| 1105 |
+
"r10-comp-06665",
|
| 1106 |
+
"r10-comp-06674",
|
| 1107 |
+
"r10-comp-06683",
|
| 1108 |
+
"r10-comp-06699",
|
| 1109 |
+
"r10-comp-06704",
|
| 1110 |
+
"r10-comp-06707",
|
| 1111 |
+
"r10-comp-06717",
|
| 1112 |
+
"r10-comp-06720",
|
| 1113 |
+
"r10-comp-06729",
|
| 1114 |
+
"r10-comp-06734",
|
| 1115 |
+
"r10-comp-06788",
|
| 1116 |
+
"r10-comp-06822",
|
| 1117 |
+
"r10-comp-06838",
|
| 1118 |
+
"r10-comp-06841",
|
| 1119 |
+
"r10-comp-06846",
|
| 1120 |
+
"r10-comp-06867",
|
| 1121 |
+
"r10-comp-06873",
|
| 1122 |
+
"r10-comp-06878",
|
| 1123 |
+
"r10-comp-06879",
|
| 1124 |
+
"r10-comp-06903",
|
| 1125 |
+
"r10-comp-06909",
|
| 1126 |
+
"r10-comp-06942",
|
| 1127 |
+
"r10-comp-06949",
|
| 1128 |
+
"r10-comp-06982",
|
| 1129 |
+
"r10-comp-06990",
|
| 1130 |
+
"r10-comp-07027",
|
| 1131 |
+
"r10-comp-07033",
|
| 1132 |
+
"r10-comp-07043",
|
| 1133 |
+
"r10-comp-07050",
|
| 1134 |
+
"r10-comp-07056",
|
| 1135 |
+
"r10-comp-07060",
|
| 1136 |
+
"r10-comp-07095",
|
| 1137 |
+
"r10-comp-07100",
|
| 1138 |
+
"r10-comp-07108",
|
| 1139 |
+
"r10-comp-07112",
|
| 1140 |
+
"r10-comp-07116",
|
| 1141 |
+
"r10-comp-07165",
|
| 1142 |
+
"r10-comp-07177",
|
| 1143 |
+
"r10-comp-07202",
|
| 1144 |
+
"r10-comp-07211",
|
| 1145 |
+
"r10-comp-07212",
|
| 1146 |
+
"r10-comp-07217",
|
| 1147 |
+
"r10-comp-07228",
|
| 1148 |
+
"r10-comp-07232",
|
| 1149 |
+
"r10-comp-07241",
|
| 1150 |
+
"r10-comp-07246",
|
| 1151 |
+
"r10-comp-07270",
|
| 1152 |
+
"r10-comp-07276",
|
| 1153 |
+
"r10-comp-07283",
|
| 1154 |
+
"r10-comp-07286",
|
| 1155 |
+
"r10-comp-07291",
|
| 1156 |
+
"r10-comp-07299",
|
| 1157 |
+
"r10-comp-07312",
|
| 1158 |
+
"r10-comp-07322",
|
| 1159 |
+
"r10-comp-07328",
|
| 1160 |
+
"r10-comp-07340",
|
| 1161 |
+
"r10-comp-07357",
|
| 1162 |
+
"r10-comp-07361",
|
| 1163 |
+
"r10-comp-07369",
|
| 1164 |
+
"r10-comp-07381",
|
| 1165 |
+
"r10-comp-07406",
|
| 1166 |
+
"r10-comp-07419",
|
| 1167 |
+
"r10-comp-07434",
|
| 1168 |
+
"r10-comp-07437",
|
| 1169 |
+
"r10-comp-07450",
|
| 1170 |
+
"r10-comp-07465",
|
| 1171 |
+
"r10-comp-07468",
|
| 1172 |
+
"r10-comp-07483",
|
| 1173 |
+
"r10-comp-07499",
|
| 1174 |
+
"r10-comp-07513",
|
| 1175 |
+
"r10-comp-07520",
|
| 1176 |
+
"r10-comp-07532",
|
| 1177 |
+
"r10-comp-07538",
|
| 1178 |
+
"r10-comp-07543",
|
| 1179 |
+
"r10-comp-07547",
|
| 1180 |
+
"r10-comp-07554",
|
| 1181 |
+
"r10-comp-07561",
|
| 1182 |
+
"r10-comp-07564",
|
| 1183 |
+
"r10-comp-07566",
|
| 1184 |
+
"r10-comp-07569",
|
| 1185 |
+
"r10-comp-07584",
|
| 1186 |
+
"r10-comp-07587",
|
| 1187 |
+
"r10-comp-07625",
|
| 1188 |
+
"r10-comp-07642",
|
| 1189 |
+
"r10-comp-07644",
|
| 1190 |
+
"r10-comp-07645",
|
| 1191 |
+
"r10-comp-07675",
|
| 1192 |
+
"r10-comp-07766",
|
| 1193 |
+
"r10-comp-07795",
|
| 1194 |
+
"r10-comp-07796",
|
| 1195 |
+
"r10-comp-07808",
|
| 1196 |
+
"r10-comp-07810",
|
| 1197 |
+
"r10-comp-07837",
|
| 1198 |
+
"r10-comp-07847",
|
| 1199 |
+
"r10-comp-07856",
|
| 1200 |
+
"r10-comp-07859",
|
| 1201 |
+
"r10-comp-07871",
|
| 1202 |
+
"r10-comp-07877",
|
| 1203 |
+
"r10-comp-07878",
|
| 1204 |
+
"r10-comp-07913",
|
| 1205 |
+
"r10-comp-07941",
|
| 1206 |
+
"r10-comp-07960",
|
| 1207 |
+
"r10-comp-07992",
|
| 1208 |
+
"r10-comp-08013",
|
| 1209 |
+
"r10-comp-08023",
|
| 1210 |
+
"r10-comp-08027",
|
| 1211 |
+
"r10-comp-08028",
|
| 1212 |
+
"r10-comp-08031",
|
| 1213 |
+
"r10-comp-08034",
|
| 1214 |
+
"r10-comp-08037",
|
| 1215 |
+
"r10-comp-08039",
|
| 1216 |
+
"r10-comp-08040",
|
| 1217 |
+
"r10-comp-08058",
|
| 1218 |
+
"r10-comp-08065",
|
| 1219 |
+
"r10-comp-08074",
|
| 1220 |
+
"r10-comp-08075",
|
| 1221 |
+
"r10-comp-08079",
|
| 1222 |
+
"r10-comp-08084",
|
| 1223 |
+
"r10-comp-08086",
|
| 1224 |
+
"r10-comp-08093",
|
| 1225 |
+
"r10-comp-08095",
|
| 1226 |
+
"r10-comp-08103",
|
| 1227 |
+
"r10-comp-08104",
|
| 1228 |
+
"r10-comp-08120",
|
| 1229 |
+
"r10-comp-08122",
|
| 1230 |
+
"r10-comp-08138",
|
| 1231 |
+
"r10-comp-08139",
|
| 1232 |
+
"r10-comp-08145",
|
| 1233 |
+
"r10-comp-08152",
|
| 1234 |
+
"r10-comp-08154",
|
| 1235 |
+
"r10-comp-08160",
|
| 1236 |
+
"r10-comp-08162",
|
| 1237 |
+
"r10-comp-08167",
|
| 1238 |
+
"r10-comp-08175",
|
| 1239 |
+
"r10-comp-08192",
|
| 1240 |
+
"r10-comp-08203",
|
| 1241 |
+
"r10-comp-08208",
|
| 1242 |
+
"r10-comp-08214",
|
| 1243 |
+
"r10-comp-08219",
|
| 1244 |
+
"r10-comp-08222",
|
| 1245 |
+
"r10-comp-08230",
|
| 1246 |
+
"r10-comp-08234",
|
| 1247 |
+
"r10-comp-08241",
|
| 1248 |
+
"r10-comp-08255",
|
| 1249 |
+
"r10-comp-08260",
|
| 1250 |
+
"r10-comp-08273",
|
| 1251 |
+
"r10-comp-08275",
|
| 1252 |
+
"r10-comp-08276",
|
| 1253 |
+
"r10-comp-08287",
|
| 1254 |
+
"r10-comp-08289",
|
| 1255 |
+
"r10-comp-08299",
|
| 1256 |
+
"r10-comp-08317",
|
| 1257 |
+
"r10-comp-08360",
|
| 1258 |
+
"r10-comp-08391",
|
| 1259 |
+
"r10-comp-08393",
|
| 1260 |
+
"r10-comp-08394",
|
| 1261 |
+
"r10-comp-08405",
|
| 1262 |
+
"r10-comp-08406",
|
| 1263 |
+
"r10-comp-08409",
|
| 1264 |
+
"r10-comp-08420",
|
| 1265 |
+
"r10-comp-08438",
|
| 1266 |
+
"r10-comp-08440",
|
| 1267 |
+
"r10-comp-08452",
|
| 1268 |
+
"r10-comp-08453",
|
| 1269 |
+
"r10-comp-08459",
|
| 1270 |
+
"r10-comp-08465",
|
| 1271 |
+
"r10-comp-08474",
|
| 1272 |
+
"r10-comp-08505",
|
| 1273 |
+
"r10-comp-08555",
|
| 1274 |
+
"r10-comp-08560",
|
| 1275 |
+
"r10-comp-08573",
|
| 1276 |
+
"r10-comp-08575",
|
| 1277 |
+
"r10-comp-08581",
|
| 1278 |
+
"r10-comp-08587",
|
| 1279 |
+
"r10-comp-08617",
|
| 1280 |
+
"r10-comp-08621",
|
| 1281 |
+
"r10-comp-08626",
|
| 1282 |
+
"r10-comp-08631",
|
| 1283 |
+
"r10-comp-08632",
|
| 1284 |
+
"r10-comp-08639",
|
| 1285 |
+
"r10-comp-08645",
|
| 1286 |
+
"r10-comp-08649",
|
| 1287 |
+
"r10-comp-08668",
|
| 1288 |
+
"r10-comp-08682",
|
| 1289 |
+
"r10-comp-08683",
|
| 1290 |
+
"r10-comp-08685",
|
| 1291 |
+
"r10-comp-08689",
|
| 1292 |
+
"r10-comp-08710",
|
| 1293 |
+
"r10-comp-08719",
|
| 1294 |
+
"r10-comp-08720",
|
| 1295 |
+
"r10-comp-08753",
|
| 1296 |
+
"r10-comp-08754",
|
| 1297 |
+
"r10-comp-08757",
|
| 1298 |
+
"r10-comp-08765",
|
| 1299 |
+
"r10-comp-08766",
|
| 1300 |
+
"r10-comp-08807",
|
| 1301 |
+
"r10-comp-08813",
|
| 1302 |
+
"r10-comp-08819",
|
| 1303 |
+
"r10-comp-08823",
|
| 1304 |
+
"r10-comp-08838",
|
| 1305 |
+
"r10-comp-08852",
|
| 1306 |
+
"r10-comp-08857",
|
| 1307 |
+
"r10-comp-08865",
|
| 1308 |
+
"r10-comp-08884",
|
| 1309 |
+
"r10-comp-08895",
|
| 1310 |
+
"r10-comp-08899",
|
| 1311 |
+
"r10-comp-08900",
|
| 1312 |
+
"r10-comp-08915",
|
| 1313 |
+
"r10-comp-08916",
|
| 1314 |
+
"r10-comp-08920",
|
| 1315 |
+
"r10-comp-08923",
|
| 1316 |
+
"r10-comp-08955",
|
| 1317 |
+
"r10-comp-08970",
|
| 1318 |
+
"r10-comp-08988",
|
| 1319 |
+
"r10-comp-09029",
|
| 1320 |
+
"r10-comp-09067",
|
| 1321 |
+
"r10-comp-09072",
|
| 1322 |
+
"r10-comp-09084",
|
| 1323 |
+
"r10-comp-09095",
|
| 1324 |
+
"r10-comp-09113",
|
| 1325 |
+
"r10-comp-09117",
|
| 1326 |
+
"r10-comp-09130",
|
| 1327 |
+
"r10-comp-09145",
|
| 1328 |
+
"r10-comp-09170",
|
| 1329 |
+
"r10-comp-09173",
|
| 1330 |
+
"r10-comp-09182",
|
| 1331 |
+
"r10-comp-09185",
|
| 1332 |
+
"r10-comp-09186",
|
| 1333 |
+
"r10-comp-09212",
|
| 1334 |
+
"r10-comp-09215",
|
| 1335 |
+
"r10-comp-09233",
|
| 1336 |
+
"r10-comp-09239",
|
| 1337 |
+
"r10-comp-09246",
|
| 1338 |
+
"r10-comp-09252",
|
| 1339 |
+
"r10-comp-09274",
|
| 1340 |
+
"r10-comp-09310",
|
| 1341 |
+
"r10-comp-09312",
|
| 1342 |
+
"r10-comp-09314",
|
| 1343 |
+
"r10-comp-09318",
|
| 1344 |
+
"r10-comp-09327"
|
| 1345 |
+
]
|
| 1346 |
+
},
|
| 1347 |
+
{
|
| 1348 |
+
"balanced_error": 0.0502750811435321,
|
| 1349 |
+
"counts": {
|
| 1350 |
+
"audiobook": 29,
|
| 1351 |
+
"comic": 67,
|
| 1352 |
+
"course": 9,
|
| 1353 |
+
"ebook": 34,
|
| 1354 |
+
"game": 78,
|
| 1355 |
+
"movie": 205,
|
| 1356 |
+
"music": 97,
|
| 1357 |
+
"software": 53,
|
| 1358 |
+
"tv_show": 134,
|
| 1359 |
+
"unknown": 0,
|
| 1360 |
+
"xxx": 0
|
| 1361 |
+
},
|
| 1362 |
+
"heldout_fold": 1,
|
| 1363 |
+
"holdout_groups": [
|
| 1364 |
+
"r10-comp-00018",
|
| 1365 |
+
"r10-comp-00028",
|
| 1366 |
+
"r10-comp-00030",
|
| 1367 |
+
"r10-comp-00037",
|
| 1368 |
+
"r10-comp-00044",
|
| 1369 |
+
"r10-comp-00054",
|
| 1370 |
+
"r10-comp-00068",
|
| 1371 |
+
"r10-comp-00069",
|
| 1372 |
+
"r10-comp-00087",
|
| 1373 |
+
"r10-comp-00099",
|
| 1374 |
+
"r10-comp-00102",
|
| 1375 |
+
"r10-comp-00127",
|
| 1376 |
+
"r10-comp-00134",
|
| 1377 |
+
"r10-comp-00137",
|
| 1378 |
+
"r10-comp-00144",
|
| 1379 |
+
"r10-comp-00146",
|
| 1380 |
+
"r10-comp-00169",
|
| 1381 |
+
"r10-comp-00189",
|
| 1382 |
+
"r10-comp-00207",
|
| 1383 |
+
"r10-comp-00208",
|
| 1384 |
+
"r10-comp-00213",
|
| 1385 |
+
"r10-comp-00233",
|
| 1386 |
+
"r10-comp-00268",
|
| 1387 |
+
"r10-comp-00280",
|
| 1388 |
+
"r10-comp-00315",
|
| 1389 |
+
"r10-comp-00320",
|
| 1390 |
+
"r10-comp-00335",
|
| 1391 |
+
"r10-comp-00341",
|
| 1392 |
+
"r10-comp-00344",
|
| 1393 |
+
"r10-comp-00367",
|
| 1394 |
+
"r10-comp-00387",
|
| 1395 |
+
"r10-comp-00396",
|
| 1396 |
+
"r10-comp-00399",
|
| 1397 |
+
"r10-comp-00405",
|
| 1398 |
+
"r10-comp-00409",
|
| 1399 |
+
"r10-comp-00429",
|
| 1400 |
+
"r10-comp-00431",
|
| 1401 |
+
"r10-comp-00433",
|
| 1402 |
+
"r10-comp-00464",
|
| 1403 |
+
"r10-comp-00473",
|
| 1404 |
+
"r10-comp-00477",
|
| 1405 |
+
"r10-comp-00491",
|
| 1406 |
+
"r10-comp-00533",
|
| 1407 |
+
"r10-comp-00543",
|
| 1408 |
+
"r10-comp-00549",
|
| 1409 |
+
"r10-comp-00559",
|
| 1410 |
+
"r10-comp-00564",
|
| 1411 |
+
"r10-comp-00568",
|
| 1412 |
+
"r10-comp-00605",
|
| 1413 |
+
"r10-comp-00620",
|
| 1414 |
+
"r10-comp-00684",
|
| 1415 |
+
"r10-comp-00703",
|
| 1416 |
+
"r10-comp-00711",
|
| 1417 |
+
"r10-comp-00723",
|
| 1418 |
+
"r10-comp-00745",
|
| 1419 |
+
"r10-comp-00776",
|
| 1420 |
+
"r10-comp-00783",
|
| 1421 |
+
"r10-comp-00802",
|
| 1422 |
+
"r10-comp-00811",
|
| 1423 |
+
"r10-comp-00838",
|
| 1424 |
+
"r10-comp-00887",
|
| 1425 |
+
"r10-comp-00936",
|
| 1426 |
+
"r10-comp-00950",
|
| 1427 |
+
"r10-comp-00970",
|
| 1428 |
+
"r10-comp-00990",
|
| 1429 |
+
"r10-comp-00992",
|
| 1430 |
+
"r10-comp-01000",
|
| 1431 |
+
"r10-comp-01009",
|
| 1432 |
+
"r10-comp-01019",
|
| 1433 |
+
"r10-comp-01025",
|
| 1434 |
+
"r10-comp-01029",
|
| 1435 |
+
"r10-comp-01106",
|
| 1436 |
+
"r10-comp-01114",
|
| 1437 |
+
"r10-comp-01123",
|
| 1438 |
+
"r10-comp-01141",
|
| 1439 |
+
"r10-comp-01147",
|
| 1440 |
+
"r10-comp-01154",
|
| 1441 |
+
"r10-comp-01190",
|
| 1442 |
+
"r10-comp-01307",
|
| 1443 |
+
"r10-comp-01338",
|
| 1444 |
+
"r10-comp-01429",
|
| 1445 |
+
"r10-comp-01440",
|
| 1446 |
+
"r10-comp-01483",
|
| 1447 |
+
"r10-comp-01556",
|
| 1448 |
+
"r10-comp-01559",
|
| 1449 |
+
"r10-comp-01595",
|
| 1450 |
+
"r10-comp-01601",
|
| 1451 |
+
"r10-comp-01637",
|
| 1452 |
+
"r10-comp-01667",
|
| 1453 |
+
"r10-comp-01675",
|
| 1454 |
+
"r10-comp-01735",
|
| 1455 |
+
"r10-comp-01756",
|
| 1456 |
+
"r10-comp-01831",
|
| 1457 |
+
"r10-comp-01926",
|
| 1458 |
+
"r10-comp-01949",
|
| 1459 |
+
"r10-comp-01961",
|
| 1460 |
+
"r10-comp-01963",
|
| 1461 |
+
"r10-comp-02022",
|
| 1462 |
+
"r10-comp-02069",
|
| 1463 |
+
"r10-comp-02111",
|
| 1464 |
+
"r10-comp-02112",
|
| 1465 |
+
"r10-comp-02196",
|
| 1466 |
+
"r10-comp-02198",
|
| 1467 |
+
"r10-comp-02222",
|
| 1468 |
+
"r10-comp-02289",
|
| 1469 |
+
"r10-comp-02292",
|
| 1470 |
+
"r10-comp-02310",
|
| 1471 |
+
"r10-comp-02336",
|
| 1472 |
+
"r10-comp-02411",
|
| 1473 |
+
"r10-comp-02494",
|
| 1474 |
+
"r10-comp-02557",
|
| 1475 |
+
"r10-comp-02561",
|
| 1476 |
+
"r10-comp-02576",
|
| 1477 |
+
"r10-comp-02602",
|
| 1478 |
+
"r10-comp-02699",
|
| 1479 |
+
"r10-comp-02760",
|
| 1480 |
+
"r10-comp-02794",
|
| 1481 |
+
"r10-comp-02875",
|
| 1482 |
+
"r10-comp-02880",
|
| 1483 |
+
"r10-comp-02999",
|
| 1484 |
+
"r10-comp-03234",
|
| 1485 |
+
"r10-comp-03241",
|
| 1486 |
+
"r10-comp-03251",
|
| 1487 |
+
"r10-comp-03253",
|
| 1488 |
+
"r10-comp-03267",
|
| 1489 |
+
"r10-comp-03269",
|
| 1490 |
+
"r10-comp-03272",
|
| 1491 |
+
"r10-comp-03304",
|
| 1492 |
+
"r10-comp-03318",
|
| 1493 |
+
"r10-comp-03328",
|
| 1494 |
+
"r10-comp-03348",
|
| 1495 |
+
"r10-comp-03446",
|
| 1496 |
+
"r10-comp-03453",
|
| 1497 |
+
"r10-comp-03457",
|
| 1498 |
+
"r10-comp-03493",
|
| 1499 |
+
"r10-comp-03526",
|
| 1500 |
+
"r10-comp-03538",
|
| 1501 |
+
"r10-comp-03570",
|
| 1502 |
+
"r10-comp-03585",
|
| 1503 |
+
"r10-comp-03612",
|
| 1504 |
+
"r10-comp-03635",
|
| 1505 |
+
"r10-comp-03754",
|
| 1506 |
+
"r10-comp-03766",
|
| 1507 |
+
"r10-comp-03858",
|
| 1508 |
+
"r10-comp-03881",
|
| 1509 |
+
"r10-comp-03884",
|
| 1510 |
+
"r10-comp-03885",
|
| 1511 |
+
"r10-comp-03886",
|
| 1512 |
+
"r10-comp-03887",
|
| 1513 |
+
"r10-comp-03915",
|
| 1514 |
+
"r10-comp-03929",
|
| 1515 |
+
"r10-comp-03948",
|
| 1516 |
+
"r10-comp-03950",
|
| 1517 |
+
"r10-comp-03959",
|
| 1518 |
+
"r10-comp-03993",
|
| 1519 |
+
"r10-comp-03996",
|
| 1520 |
+
"r10-comp-04016",
|
| 1521 |
+
"r10-comp-04023",
|
| 1522 |
+
"r10-comp-04025",
|
| 1523 |
+
"r10-comp-04033",
|
| 1524 |
+
"r10-comp-04034",
|
| 1525 |
+
"r10-comp-04036",
|
| 1526 |
+
"r10-comp-04047",
|
| 1527 |
+
"r10-comp-04056",
|
| 1528 |
+
"r10-comp-04073",
|
| 1529 |
+
"r10-comp-04084",
|
| 1530 |
+
"r10-comp-04088",
|
| 1531 |
+
"r10-comp-04101",
|
| 1532 |
+
"r10-comp-04108",
|
| 1533 |
+
"r10-comp-04110",
|
| 1534 |
+
"r10-comp-04120",
|
| 1535 |
+
"r10-comp-04124",
|
| 1536 |
+
"r10-comp-04130",
|
| 1537 |
+
"r10-comp-04136",
|
| 1538 |
+
"r10-comp-04151",
|
| 1539 |
+
"r10-comp-04179",
|
| 1540 |
+
"r10-comp-04182",
|
| 1541 |
+
"r10-comp-04210",
|
| 1542 |
+
"r10-comp-04211",
|
| 1543 |
+
"r10-comp-04253",
|
| 1544 |
+
"r10-comp-04287",
|
| 1545 |
+
"r10-comp-04289",
|
| 1546 |
+
"r10-comp-04292",
|
| 1547 |
+
"r10-comp-04318",
|
| 1548 |
+
"r10-comp-04330",
|
| 1549 |
+
"r10-comp-04344",
|
| 1550 |
+
"r10-comp-04396",
|
| 1551 |
+
"r10-comp-04397",
|
| 1552 |
+
"r10-comp-04442",
|
| 1553 |
+
"r10-comp-04445",
|
| 1554 |
+
"r10-comp-04476",
|
| 1555 |
+
"r10-comp-04478",
|
| 1556 |
+
"r10-comp-04479",
|
| 1557 |
+
"r10-comp-04485",
|
| 1558 |
+
"r10-comp-04521",
|
| 1559 |
+
"r10-comp-04531",
|
| 1560 |
+
"r10-comp-04532",
|
| 1561 |
+
"r10-comp-04540",
|
| 1562 |
+
"r10-comp-04552",
|
| 1563 |
+
"r10-comp-04563",
|
| 1564 |
+
"r10-comp-04570",
|
| 1565 |
+
"r10-comp-04682",
|
| 1566 |
+
"r10-comp-04704",
|
| 1567 |
+
"r10-comp-04706",
|
| 1568 |
+
"r10-comp-04709",
|
| 1569 |
+
"r10-comp-04714",
|
| 1570 |
+
"r10-comp-04716",
|
| 1571 |
+
"r10-comp-04721",
|
| 1572 |
+
"r10-comp-04738",
|
| 1573 |
+
"r10-comp-04741",
|
| 1574 |
+
"r10-comp-04798",
|
| 1575 |
+
"r10-comp-04826",
|
| 1576 |
+
"r10-comp-04834",
|
| 1577 |
+
"r10-comp-04855",
|
| 1578 |
+
"r10-comp-04863",
|
| 1579 |
+
"r10-comp-04874",
|
| 1580 |
+
"r10-comp-04883",
|
| 1581 |
+
"r10-comp-04893",
|
| 1582 |
+
"r10-comp-04901",
|
| 1583 |
+
"r10-comp-04911",
|
| 1584 |
+
"r10-comp-04912",
|
| 1585 |
+
"r10-comp-04918",
|
| 1586 |
+
"r10-comp-04935",
|
| 1587 |
+
"r10-comp-04962",
|
| 1588 |
+
"r10-comp-04971",
|
| 1589 |
+
"r10-comp-04995",
|
| 1590 |
+
"r10-comp-05006",
|
| 1591 |
+
"r10-comp-05010",
|
| 1592 |
+
"r10-comp-05014",
|
| 1593 |
+
"r10-comp-05017",
|
| 1594 |
+
"r10-comp-05026",
|
| 1595 |
+
"r10-comp-05075",
|
| 1596 |
+
"r10-comp-05079",
|
| 1597 |
+
"r10-comp-05124",
|
| 1598 |
+
"r10-comp-05126",
|
| 1599 |
+
"r10-comp-05140",
|
| 1600 |
+
"r10-comp-05147",
|
| 1601 |
+
"r10-comp-05156",
|
| 1602 |
+
"r10-comp-05163",
|
| 1603 |
+
"r10-comp-05203",
|
| 1604 |
+
"r10-comp-05205",
|
| 1605 |
+
"r10-comp-05206",
|
| 1606 |
+
"r10-comp-05216",
|
| 1607 |
+
"r10-comp-05237",
|
| 1608 |
+
"r10-comp-05308",
|
| 1609 |
+
"r10-comp-05334",
|
| 1610 |
+
"r10-comp-05336",
|
| 1611 |
+
"r10-comp-05346",
|
| 1612 |
+
"r10-comp-05357",
|
| 1613 |
+
"r10-comp-05382",
|
| 1614 |
+
"r10-comp-05404",
|
| 1615 |
+
"r10-comp-05438",
|
| 1616 |
+
"r10-comp-05444",
|
| 1617 |
+
"r10-comp-05457",
|
| 1618 |
+
"r10-comp-05503",
|
| 1619 |
+
"r10-comp-05511",
|
| 1620 |
+
"r10-comp-05529",
|
| 1621 |
+
"r10-comp-05532",
|
| 1622 |
+
"r10-comp-05537",
|
| 1623 |
+
"r10-comp-05597",
|
| 1624 |
+
"r10-comp-05615",
|
| 1625 |
+
"r10-comp-05639",
|
| 1626 |
+
"r10-comp-05641",
|
| 1627 |
+
"r10-comp-05680",
|
| 1628 |
+
"r10-comp-05718",
|
| 1629 |
+
"r10-comp-05724",
|
| 1630 |
+
"r10-comp-05727",
|
| 1631 |
+
"r10-comp-05728",
|
| 1632 |
+
"r10-comp-05768",
|
| 1633 |
+
"r10-comp-05824",
|
| 1634 |
+
"r10-comp-05825",
|
| 1635 |
+
"r10-comp-05830",
|
| 1636 |
+
"r10-comp-05834",
|
| 1637 |
+
"r10-comp-05851",
|
| 1638 |
+
"r10-comp-05867",
|
| 1639 |
+
"r10-comp-05926",
|
| 1640 |
+
"r10-comp-05951",
|
| 1641 |
+
"r10-comp-05973",
|
| 1642 |
+
"r10-comp-05991",
|
| 1643 |
+
"r10-comp-05993",
|
| 1644 |
+
"r10-comp-06007",
|
| 1645 |
+
"r10-comp-06011",
|
| 1646 |
+
"r10-comp-06015",
|
| 1647 |
+
"r10-comp-06020",
|
| 1648 |
+
"r10-comp-06062",
|
| 1649 |
+
"r10-comp-06065",
|
| 1650 |
+
"r10-comp-06077",
|
| 1651 |
+
"r10-comp-06083",
|
| 1652 |
+
"r10-comp-06091",
|
| 1653 |
+
"r10-comp-06098",
|
| 1654 |
+
"r10-comp-06146",
|
| 1655 |
+
"r10-comp-06156",
|
| 1656 |
+
"r10-comp-06158",
|
| 1657 |
+
"r10-comp-06165",
|
| 1658 |
+
"r10-comp-06172",
|
| 1659 |
+
"r10-comp-06188",
|
| 1660 |
+
"r10-comp-06203",
|
| 1661 |
+
"r10-comp-06206",
|
| 1662 |
+
"r10-comp-06208",
|
| 1663 |
+
"r10-comp-06209",
|
| 1664 |
+
"r10-comp-06259",
|
| 1665 |
+
"r10-comp-06275",
|
| 1666 |
+
"r10-comp-06276",
|
| 1667 |
+
"r10-comp-06295",
|
| 1668 |
+
"r10-comp-06303",
|
| 1669 |
+
"r10-comp-06310",
|
| 1670 |
+
"r10-comp-06319",
|
| 1671 |
+
"r10-comp-06344",
|
| 1672 |
+
"r10-comp-06367",
|
| 1673 |
+
"r10-comp-06404",
|
| 1674 |
+
"r10-comp-06414",
|
| 1675 |
+
"r10-comp-06434",
|
| 1676 |
+
"r10-comp-06441",
|
| 1677 |
+
"r10-comp-06444",
|
| 1678 |
+
"r10-comp-06460",
|
| 1679 |
+
"r10-comp-06463",
|
| 1680 |
+
"r10-comp-06486",
|
| 1681 |
+
"r10-comp-06507",
|
| 1682 |
+
"r10-comp-06520",
|
| 1683 |
+
"r10-comp-06527",
|
| 1684 |
+
"r10-comp-06528",
|
| 1685 |
+
"r10-comp-06536",
|
| 1686 |
+
"r10-comp-06540",
|
| 1687 |
+
"r10-comp-06550",
|
| 1688 |
+
"r10-comp-06556",
|
| 1689 |
+
"r10-comp-06558",
|
| 1690 |
+
"r10-comp-06586",
|
| 1691 |
+
"r10-comp-06594",
|
| 1692 |
+
"r10-comp-06620",
|
| 1693 |
+
"r10-comp-06626",
|
| 1694 |
+
"r10-comp-06627",
|
| 1695 |
+
"r10-comp-06652",
|
| 1696 |
+
"r10-comp-06656",
|
| 1697 |
+
"r10-comp-06663",
|
| 1698 |
+
"r10-comp-06665",
|
| 1699 |
+
"r10-comp-06674",
|
| 1700 |
+
"r10-comp-06683",
|
| 1701 |
+
"r10-comp-06699",
|
| 1702 |
+
"r10-comp-06704",
|
| 1703 |
+
"r10-comp-06707",
|
| 1704 |
+
"r10-comp-06717",
|
| 1705 |
+
"r10-comp-06720",
|
| 1706 |
+
"r10-comp-06729",
|
| 1707 |
+
"r10-comp-06734",
|
| 1708 |
+
"r10-comp-06788",
|
| 1709 |
+
"r10-comp-06822",
|
| 1710 |
+
"r10-comp-06838",
|
| 1711 |
+
"r10-comp-06841",
|
| 1712 |
+
"r10-comp-06846",
|
| 1713 |
+
"r10-comp-06867",
|
| 1714 |
+
"r10-comp-06873",
|
| 1715 |
+
"r10-comp-06878",
|
| 1716 |
+
"r10-comp-06879",
|
| 1717 |
+
"r10-comp-06903",
|
| 1718 |
+
"r10-comp-06909",
|
| 1719 |
+
"r10-comp-06942",
|
| 1720 |
+
"r10-comp-06949",
|
| 1721 |
+
"r10-comp-06982",
|
| 1722 |
+
"r10-comp-06990",
|
| 1723 |
+
"r10-comp-07027",
|
| 1724 |
+
"r10-comp-07033",
|
| 1725 |
+
"r10-comp-07043",
|
| 1726 |
+
"r10-comp-07050",
|
| 1727 |
+
"r10-comp-07056",
|
| 1728 |
+
"r10-comp-07060",
|
| 1729 |
+
"r10-comp-07095",
|
| 1730 |
+
"r10-comp-07100",
|
| 1731 |
+
"r10-comp-07108",
|
| 1732 |
+
"r10-comp-07112",
|
| 1733 |
+
"r10-comp-07116",
|
| 1734 |
+
"r10-comp-07165",
|
| 1735 |
+
"r10-comp-07177",
|
| 1736 |
+
"r10-comp-07202",
|
| 1737 |
+
"r10-comp-07211",
|
| 1738 |
+
"r10-comp-07212",
|
| 1739 |
+
"r10-comp-07217",
|
| 1740 |
+
"r10-comp-07228",
|
| 1741 |
+
"r10-comp-07232",
|
| 1742 |
+
"r10-comp-07241",
|
| 1743 |
+
"r10-comp-07246",
|
| 1744 |
+
"r10-comp-07270",
|
| 1745 |
+
"r10-comp-07276",
|
| 1746 |
+
"r10-comp-07283",
|
| 1747 |
+
"r10-comp-07286",
|
| 1748 |
+
"r10-comp-07291",
|
| 1749 |
+
"r10-comp-07299",
|
| 1750 |
+
"r10-comp-07312",
|
| 1751 |
+
"r10-comp-07322",
|
| 1752 |
+
"r10-comp-07328",
|
| 1753 |
+
"r10-comp-07340",
|
| 1754 |
+
"r10-comp-07357",
|
| 1755 |
+
"r10-comp-07361",
|
| 1756 |
+
"r10-comp-07369",
|
| 1757 |
+
"r10-comp-07381",
|
| 1758 |
+
"r10-comp-07406",
|
| 1759 |
+
"r10-comp-07419",
|
| 1760 |
+
"r10-comp-07434",
|
| 1761 |
+
"r10-comp-07437",
|
| 1762 |
+
"r10-comp-07450",
|
| 1763 |
+
"r10-comp-07465",
|
| 1764 |
+
"r10-comp-07468",
|
| 1765 |
+
"r10-comp-07483",
|
| 1766 |
+
"r10-comp-07499",
|
| 1767 |
+
"r10-comp-07513",
|
| 1768 |
+
"r10-comp-07520",
|
| 1769 |
+
"r10-comp-07532",
|
| 1770 |
+
"r10-comp-07538",
|
| 1771 |
+
"r10-comp-07543",
|
| 1772 |
+
"r10-comp-07547",
|
| 1773 |
+
"r10-comp-07554",
|
| 1774 |
+
"r10-comp-07561",
|
| 1775 |
+
"r10-comp-07564",
|
| 1776 |
+
"r10-comp-07566",
|
| 1777 |
+
"r10-comp-07569",
|
| 1778 |
+
"r10-comp-07584",
|
| 1779 |
+
"r10-comp-07587",
|
| 1780 |
+
"r10-comp-07625",
|
| 1781 |
+
"r10-comp-07642",
|
| 1782 |
+
"r10-comp-07644",
|
| 1783 |
+
"r10-comp-07645",
|
| 1784 |
+
"r10-comp-07675",
|
| 1785 |
+
"r10-comp-07766",
|
| 1786 |
+
"r10-comp-07795",
|
| 1787 |
+
"r10-comp-07796",
|
| 1788 |
+
"r10-comp-07808",
|
| 1789 |
+
"r10-comp-07810",
|
| 1790 |
+
"r10-comp-07837",
|
| 1791 |
+
"r10-comp-07847",
|
| 1792 |
+
"r10-comp-07856",
|
| 1793 |
+
"r10-comp-07859",
|
| 1794 |
+
"r10-comp-07871",
|
| 1795 |
+
"r10-comp-07877",
|
| 1796 |
+
"r10-comp-07878",
|
| 1797 |
+
"r10-comp-07913",
|
| 1798 |
+
"r10-comp-07941",
|
| 1799 |
+
"r10-comp-07960",
|
| 1800 |
+
"r10-comp-07992",
|
| 1801 |
+
"r10-comp-08013",
|
| 1802 |
+
"r10-comp-08023",
|
| 1803 |
+
"r10-comp-08027",
|
| 1804 |
+
"r10-comp-08028",
|
| 1805 |
+
"r10-comp-08031",
|
| 1806 |
+
"r10-comp-08034",
|
| 1807 |
+
"r10-comp-08037",
|
| 1808 |
+
"r10-comp-08039",
|
| 1809 |
+
"r10-comp-08040",
|
| 1810 |
+
"r10-comp-08058",
|
| 1811 |
+
"r10-comp-08065",
|
| 1812 |
+
"r10-comp-08074",
|
| 1813 |
+
"r10-comp-08075",
|
| 1814 |
+
"r10-comp-08079",
|
| 1815 |
+
"r10-comp-08084",
|
| 1816 |
+
"r10-comp-08086",
|
| 1817 |
+
"r10-comp-08093",
|
| 1818 |
+
"r10-comp-08095",
|
| 1819 |
+
"r10-comp-08103",
|
| 1820 |
+
"r10-comp-08104",
|
| 1821 |
+
"r10-comp-08120",
|
| 1822 |
+
"r10-comp-08122",
|
| 1823 |
+
"r10-comp-08138",
|
| 1824 |
+
"r10-comp-08139",
|
| 1825 |
+
"r10-comp-08145",
|
| 1826 |
+
"r10-comp-08152",
|
| 1827 |
+
"r10-comp-08154",
|
| 1828 |
+
"r10-comp-08160",
|
| 1829 |
+
"r10-comp-08162",
|
| 1830 |
+
"r10-comp-08167",
|
| 1831 |
+
"r10-comp-08175",
|
| 1832 |
+
"r10-comp-08192",
|
| 1833 |
+
"r10-comp-08203",
|
| 1834 |
+
"r10-comp-08208",
|
| 1835 |
+
"r10-comp-08214",
|
| 1836 |
+
"r10-comp-08219",
|
| 1837 |
+
"r10-comp-08222",
|
| 1838 |
+
"r10-comp-08230",
|
| 1839 |
+
"r10-comp-08234",
|
| 1840 |
+
"r10-comp-08241",
|
| 1841 |
+
"r10-comp-08255",
|
| 1842 |
+
"r10-comp-08260",
|
| 1843 |
+
"r10-comp-08273",
|
| 1844 |
+
"r10-comp-08275",
|
| 1845 |
+
"r10-comp-08276",
|
| 1846 |
+
"r10-comp-08287",
|
| 1847 |
+
"r10-comp-08289",
|
| 1848 |
+
"r10-comp-08299",
|
| 1849 |
+
"r10-comp-08317",
|
| 1850 |
+
"r10-comp-08360",
|
| 1851 |
+
"r10-comp-08391",
|
| 1852 |
+
"r10-comp-08393",
|
| 1853 |
+
"r10-comp-08394",
|
| 1854 |
+
"r10-comp-08405",
|
| 1855 |
+
"r10-comp-08406",
|
| 1856 |
+
"r10-comp-08409",
|
| 1857 |
+
"r10-comp-08420",
|
| 1858 |
+
"r10-comp-08438",
|
| 1859 |
+
"r10-comp-08440",
|
| 1860 |
+
"r10-comp-08452",
|
| 1861 |
+
"r10-comp-08453",
|
| 1862 |
+
"r10-comp-08459",
|
| 1863 |
+
"r10-comp-08465",
|
| 1864 |
+
"r10-comp-08474",
|
| 1865 |
+
"r10-comp-08505",
|
| 1866 |
+
"r10-comp-08555",
|
| 1867 |
+
"r10-comp-08560",
|
| 1868 |
+
"r10-comp-08573",
|
| 1869 |
+
"r10-comp-08575",
|
| 1870 |
+
"r10-comp-08581",
|
| 1871 |
+
"r10-comp-08587",
|
| 1872 |
+
"r10-comp-08617",
|
| 1873 |
+
"r10-comp-08621",
|
| 1874 |
+
"r10-comp-08626",
|
| 1875 |
+
"r10-comp-08631",
|
| 1876 |
+
"r10-comp-08632",
|
| 1877 |
+
"r10-comp-08639",
|
| 1878 |
+
"r10-comp-08645",
|
| 1879 |
+
"r10-comp-08649",
|
| 1880 |
+
"r10-comp-08668",
|
| 1881 |
+
"r10-comp-08682",
|
| 1882 |
+
"r10-comp-08683",
|
| 1883 |
+
"r10-comp-08685",
|
| 1884 |
+
"r10-comp-08689",
|
| 1885 |
+
"r10-comp-08710",
|
| 1886 |
+
"r10-comp-08719",
|
| 1887 |
+
"r10-comp-08720",
|
| 1888 |
+
"r10-comp-08753",
|
| 1889 |
+
"r10-comp-08754",
|
| 1890 |
+
"r10-comp-08757",
|
| 1891 |
+
"r10-comp-08765",
|
| 1892 |
+
"r10-comp-08766",
|
| 1893 |
+
"r10-comp-08807",
|
| 1894 |
+
"r10-comp-08813",
|
| 1895 |
+
"r10-comp-08819",
|
| 1896 |
+
"r10-comp-08823",
|
| 1897 |
+
"r10-comp-08838",
|
| 1898 |
+
"r10-comp-08852",
|
| 1899 |
+
"r10-comp-08857",
|
| 1900 |
+
"r10-comp-08865",
|
| 1901 |
+
"r10-comp-08884",
|
| 1902 |
+
"r10-comp-08895",
|
| 1903 |
+
"r10-comp-08899",
|
| 1904 |
+
"r10-comp-08900",
|
| 1905 |
+
"r10-comp-08915",
|
| 1906 |
+
"r10-comp-08916",
|
| 1907 |
+
"r10-comp-08920",
|
| 1908 |
+
"r10-comp-08923",
|
| 1909 |
+
"r10-comp-08955",
|
| 1910 |
+
"r10-comp-08970",
|
| 1911 |
+
"r10-comp-08988",
|
| 1912 |
+
"r10-comp-09029",
|
| 1913 |
+
"r10-comp-09067",
|
| 1914 |
+
"r10-comp-09072",
|
| 1915 |
+
"r10-comp-09084",
|
| 1916 |
+
"r10-comp-09095",
|
| 1917 |
+
"r10-comp-09113",
|
| 1918 |
+
"r10-comp-09117",
|
| 1919 |
+
"r10-comp-09130",
|
| 1920 |
+
"r10-comp-09145",
|
| 1921 |
+
"r10-comp-09170",
|
| 1922 |
+
"r10-comp-09173",
|
| 1923 |
+
"r10-comp-09182",
|
| 1924 |
+
"r10-comp-09185",
|
| 1925 |
+
"r10-comp-09186",
|
| 1926 |
+
"r10-comp-09212",
|
| 1927 |
+
"r10-comp-09215",
|
| 1928 |
+
"r10-comp-09233",
|
| 1929 |
+
"r10-comp-09239",
|
| 1930 |
+
"r10-comp-09246",
|
| 1931 |
+
"r10-comp-09252",
|
| 1932 |
+
"r10-comp-09274",
|
| 1933 |
+
"r10-comp-09310",
|
| 1934 |
+
"r10-comp-09312",
|
| 1935 |
+
"r10-comp-09314",
|
| 1936 |
+
"r10-comp-09318",
|
| 1937 |
+
"r10-comp-09327"
|
| 1938 |
+
],
|
| 1939 |
+
"n_holdout": 705,
|
| 1940 |
+
"n_train": 706,
|
| 1941 |
+
"raw_thresholds": {
|
| 1942 |
+
"audiobook": 0.399,
|
| 1943 |
+
"comic": 0.399,
|
| 1944 |
+
"course": 0.399,
|
| 1945 |
+
"ebook": 0.399,
|
| 1946 |
+
"game": 0.399,
|
| 1947 |
+
"movie": 0.306,
|
| 1948 |
+
"music": 0.399,
|
| 1949 |
+
"software": 0.399,
|
| 1950 |
+
"tv_show": 0.399,
|
| 1951 |
+
"unknown": 0.399,
|
| 1952 |
+
"xxx": 0.399
|
| 1953 |
+
},
|
| 1954 |
+
"tau_global": 0.399,
|
| 1955 |
+
"thresholds": {
|
| 1956 |
+
"audiobook": 0.399,
|
| 1957 |
+
"comic": 0.399,
|
| 1958 |
+
"course": 0.399,
|
| 1959 |
+
"ebook": 0.399,
|
| 1960 |
+
"game": 0.399,
|
| 1961 |
+
"movie": 0.3178723404255319,
|
| 1962 |
+
"music": 0.399,
|
| 1963 |
+
"software": 0.399,
|
| 1964 |
+
"tv_show": 0.399,
|
| 1965 |
+
"unknown": 0.399,
|
| 1966 |
+
"xxx": 0.399
|
| 1967 |
+
},
|
| 1968 |
+
"train_groups": [
|
| 1969 |
+
"r10-comp-00005",
|
| 1970 |
+
"r10-comp-00049",
|
| 1971 |
+
"r10-comp-00059",
|
| 1972 |
+
"r10-comp-00067",
|
| 1973 |
+
"r10-comp-00084",
|
| 1974 |
+
"r10-comp-00092",
|
| 1975 |
+
"r10-comp-00122",
|
| 1976 |
+
"r10-comp-00176",
|
| 1977 |
+
"r10-comp-00204",
|
| 1978 |
+
"r10-comp-00221",
|
| 1979 |
+
"r10-comp-00236",
|
| 1980 |
+
"r10-comp-00238",
|
| 1981 |
+
"r10-comp-00240",
|
| 1982 |
+
"r10-comp-00269",
|
| 1983 |
+
"r10-comp-00311",
|
| 1984 |
+
"r10-comp-00343",
|
| 1985 |
+
"r10-comp-00371",
|
| 1986 |
+
"r10-comp-00379",
|
| 1987 |
+
"r10-comp-00384",
|
| 1988 |
+
"r10-comp-00393",
|
| 1989 |
+
"r10-comp-00450",
|
| 1990 |
+
"r10-comp-00469",
|
| 1991 |
+
"r10-comp-00542",
|
| 1992 |
+
"r10-comp-00553",
|
| 1993 |
+
"r10-comp-00558",
|
| 1994 |
+
"r10-comp-00560",
|
| 1995 |
+
"r10-comp-00563",
|
| 1996 |
+
"r10-comp-00610",
|
| 1997 |
+
"r10-comp-00614",
|
| 1998 |
+
"r10-comp-00619",
|
| 1999 |
+
"r10-comp-00626",
|
| 2000 |
+
"r10-comp-00646",
|
| 2001 |
+
"r10-comp-00675",
|
| 2002 |
+
"r10-comp-00705",
|
| 2003 |
+
"r10-comp-00732",
|
| 2004 |
+
"r10-comp-00749",
|
| 2005 |
+
"r10-comp-00768",
|
| 2006 |
+
"r10-comp-00769",
|
| 2007 |
+
"r10-comp-00777",
|
| 2008 |
+
"r10-comp-00817",
|
| 2009 |
+
"r10-comp-00848",
|
| 2010 |
+
"r10-comp-00874",
|
| 2011 |
+
"r10-comp-00892",
|
| 2012 |
+
"r10-comp-01011",
|
| 2013 |
+
"r10-comp-01014",
|
| 2014 |
+
"r10-comp-01039",
|
| 2015 |
+
"r10-comp-01047",
|
| 2016 |
+
"r10-comp-01136",
|
| 2017 |
+
"r10-comp-01277",
|
| 2018 |
+
"r10-comp-01309",
|
| 2019 |
+
"r10-comp-01320",
|
| 2020 |
+
"r10-comp-01341",
|
| 2021 |
+
"r10-comp-01353",
|
| 2022 |
+
"r10-comp-01356",
|
| 2023 |
+
"r10-comp-01405",
|
| 2024 |
+
"r10-comp-01441",
|
| 2025 |
+
"r10-comp-01445",
|
| 2026 |
+
"r10-comp-01447",
|
| 2027 |
+
"r10-comp-01485",
|
| 2028 |
+
"r10-comp-01566",
|
| 2029 |
+
"r10-comp-01622",
|
| 2030 |
+
"r10-comp-01626",
|
| 2031 |
+
"r10-comp-01647",
|
| 2032 |
+
"r10-comp-01693",
|
| 2033 |
+
"r10-comp-01725",
|
| 2034 |
+
"r10-comp-01751",
|
| 2035 |
+
"r10-comp-01760",
|
| 2036 |
+
"r10-comp-01795",
|
| 2037 |
+
"r10-comp-01808",
|
| 2038 |
+
"r10-comp-01816",
|
| 2039 |
+
"r10-comp-01826",
|
| 2040 |
+
"r10-comp-01827",
|
| 2041 |
+
"r10-comp-01860",
|
| 2042 |
+
"r10-comp-01893",
|
| 2043 |
+
"r10-comp-01907",
|
| 2044 |
+
"r10-comp-01941",
|
| 2045 |
+
"r10-comp-01972",
|
| 2046 |
+
"r10-comp-01981",
|
| 2047 |
+
"r10-comp-01999",
|
| 2048 |
+
"r10-comp-02018",
|
| 2049 |
+
"r10-comp-02137",
|
| 2050 |
+
"r10-comp-02146",
|
| 2051 |
+
"r10-comp-02206",
|
| 2052 |
+
"r10-comp-02236",
|
| 2053 |
+
"r10-comp-02302",
|
| 2054 |
+
"r10-comp-02305",
|
| 2055 |
+
"r10-comp-02341",
|
| 2056 |
+
"r10-comp-02361",
|
| 2057 |
+
"r10-comp-02373",
|
| 2058 |
+
"r10-comp-02396",
|
| 2059 |
+
"r10-comp-02421",
|
| 2060 |
+
"r10-comp-02448",
|
| 2061 |
+
"r10-comp-02497",
|
| 2062 |
+
"r10-comp-02548",
|
| 2063 |
+
"r10-comp-02553",
|
| 2064 |
+
"r10-comp-02591",
|
| 2065 |
+
"r10-comp-02593",
|
| 2066 |
+
"r10-comp-02600",
|
| 2067 |
+
"r10-comp-02622",
|
| 2068 |
+
"r10-comp-02640",
|
| 2069 |
+
"r10-comp-02654",
|
| 2070 |
+
"r10-comp-02698",
|
| 2071 |
+
"r10-comp-02764",
|
| 2072 |
+
"r10-comp-02824",
|
| 2073 |
+
"r10-comp-02839",
|
| 2074 |
+
"r10-comp-02930",
|
| 2075 |
+
"r10-comp-02983",
|
| 2076 |
+
"r10-comp-02990",
|
| 2077 |
+
"r10-comp-03005",
|
| 2078 |
+
"r10-comp-03040",
|
| 2079 |
+
"r10-comp-03133",
|
| 2080 |
+
"r10-comp-03188",
|
| 2081 |
+
"r10-comp-03229",
|
| 2082 |
+
"r10-comp-03239",
|
| 2083 |
+
"r10-comp-03260",
|
| 2084 |
+
"r10-comp-03265",
|
| 2085 |
+
"r10-comp-03274",
|
| 2086 |
+
"r10-comp-03282",
|
| 2087 |
+
"r10-comp-03298",
|
| 2088 |
+
"r10-comp-03300",
|
| 2089 |
+
"r10-comp-03317",
|
| 2090 |
+
"r10-comp-03384",
|
| 2091 |
+
"r10-comp-03436",
|
| 2092 |
+
"r10-comp-03535",
|
| 2093 |
+
"r10-comp-03565",
|
| 2094 |
+
"r10-comp-03586",
|
| 2095 |
+
"r10-comp-03590",
|
| 2096 |
+
"r10-comp-03597",
|
| 2097 |
+
"r10-comp-03603",
|
| 2098 |
+
"r10-comp-03633",
|
| 2099 |
+
"r10-comp-03648",
|
| 2100 |
+
"r10-comp-03649",
|
| 2101 |
+
"r10-comp-03681",
|
| 2102 |
+
"r10-comp-03711",
|
| 2103 |
+
"r10-comp-03823",
|
| 2104 |
+
"r10-comp-03838",
|
| 2105 |
+
"r10-comp-03846",
|
| 2106 |
+
"r10-comp-03847",
|
| 2107 |
+
"r10-comp-03857",
|
| 2108 |
+
"r10-comp-03859",
|
| 2109 |
+
"r10-comp-03877",
|
| 2110 |
+
"r10-comp-03892",
|
| 2111 |
+
"r10-comp-03894",
|
| 2112 |
+
"r10-comp-03916",
|
| 2113 |
+
"r10-comp-03924",
|
| 2114 |
+
"r10-comp-03942",
|
| 2115 |
+
"r10-comp-03957",
|
| 2116 |
+
"r10-comp-03963",
|
| 2117 |
+
"r10-comp-04000",
|
| 2118 |
+
"r10-comp-04029",
|
| 2119 |
+
"r10-comp-04049",
|
| 2120 |
+
"r10-comp-04114",
|
| 2121 |
+
"r10-comp-04119",
|
| 2122 |
+
"r10-comp-04139",
|
| 2123 |
+
"r10-comp-04197",
|
| 2124 |
+
"r10-comp-04238",
|
| 2125 |
+
"r10-comp-04251",
|
| 2126 |
+
"r10-comp-04270",
|
| 2127 |
+
"r10-comp-04271",
|
| 2128 |
+
"r10-comp-04278",
|
| 2129 |
+
"r10-comp-04286",
|
| 2130 |
+
"r10-comp-04297",
|
| 2131 |
+
"r10-comp-04322",
|
| 2132 |
+
"r10-comp-04340",
|
| 2133 |
+
"r10-comp-04342",
|
| 2134 |
+
"r10-comp-04349",
|
| 2135 |
+
"r10-comp-04365",
|
| 2136 |
+
"r10-comp-04385",
|
| 2137 |
+
"r10-comp-04393",
|
| 2138 |
+
"r10-comp-04438",
|
| 2139 |
+
"r10-comp-04450",
|
| 2140 |
+
"r10-comp-04494",
|
| 2141 |
+
"r10-comp-04505",
|
| 2142 |
+
"r10-comp-04534",
|
| 2143 |
+
"r10-comp-04544",
|
| 2144 |
+
"r10-comp-04555",
|
| 2145 |
+
"r10-comp-04594",
|
| 2146 |
+
"r10-comp-04596",
|
| 2147 |
+
"r10-comp-04605",
|
| 2148 |
+
"r10-comp-04637",
|
| 2149 |
+
"r10-comp-04656",
|
| 2150 |
+
"r10-comp-04657",
|
| 2151 |
+
"r10-comp-04660",
|
| 2152 |
+
"r10-comp-04671",
|
| 2153 |
+
"r10-comp-04701",
|
| 2154 |
+
"r10-comp-04717",
|
| 2155 |
+
"r10-comp-04725",
|
| 2156 |
+
"r10-comp-04740",
|
| 2157 |
+
"r10-comp-04742",
|
| 2158 |
+
"r10-comp-04753",
|
| 2159 |
+
"r10-comp-04755",
|
| 2160 |
+
"r10-comp-04762",
|
| 2161 |
+
"r10-comp-04772",
|
| 2162 |
+
"r10-comp-04776",
|
| 2163 |
+
"r10-comp-04783",
|
| 2164 |
+
"r10-comp-04796",
|
| 2165 |
+
"r10-comp-04812",
|
| 2166 |
+
"r10-comp-04827",
|
| 2167 |
+
"r10-comp-04845",
|
| 2168 |
+
"r10-comp-04871",
|
| 2169 |
+
"r10-comp-04876",
|
| 2170 |
+
"r10-comp-04879",
|
| 2171 |
+
"r10-comp-04887",
|
| 2172 |
+
"r10-comp-04898",
|
| 2173 |
+
"r10-comp-04913",
|
| 2174 |
+
"r10-comp-04919",
|
| 2175 |
+
"r10-comp-04956",
|
| 2176 |
+
"r10-comp-04966",
|
| 2177 |
+
"r10-comp-04986",
|
| 2178 |
+
"r10-comp-04990",
|
| 2179 |
+
"r10-comp-05023",
|
| 2180 |
+
"r10-comp-05027",
|
| 2181 |
+
"r10-comp-05029",
|
| 2182 |
+
"r10-comp-05038",
|
| 2183 |
+
"r10-comp-05061",
|
| 2184 |
+
"r10-comp-05070",
|
| 2185 |
+
"r10-comp-05080",
|
| 2186 |
+
"r10-comp-05081",
|
| 2187 |
+
"r10-comp-05083",
|
| 2188 |
+
"r10-comp-05099",
|
| 2189 |
+
"r10-comp-05106",
|
| 2190 |
+
"r10-comp-05111",
|
| 2191 |
+
"r10-comp-05151",
|
| 2192 |
+
"r10-comp-05157",
|
| 2193 |
+
"r10-comp-05162",
|
| 2194 |
+
"r10-comp-05168",
|
| 2195 |
+
"r10-comp-05178",
|
| 2196 |
+
"r10-comp-05218",
|
| 2197 |
+
"r10-comp-05224",
|
| 2198 |
+
"r10-comp-05234",
|
| 2199 |
+
"r10-comp-05276",
|
| 2200 |
+
"r10-comp-05309",
|
| 2201 |
+
"r10-comp-05310",
|
| 2202 |
+
"r10-comp-05316",
|
| 2203 |
+
"r10-comp-05337",
|
| 2204 |
+
"r10-comp-05349",
|
| 2205 |
+
"r10-comp-05350",
|
| 2206 |
+
"r10-comp-05352",
|
| 2207 |
+
"r10-comp-05356",
|
| 2208 |
+
"r10-comp-05366",
|
| 2209 |
+
"r10-comp-05369",
|
| 2210 |
+
"r10-comp-05370",
|
| 2211 |
+
"r10-comp-05381",
|
| 2212 |
+
"r10-comp-05383",
|
| 2213 |
+
"r10-comp-05384",
|
| 2214 |
+
"r10-comp-05389",
|
| 2215 |
+
"r10-comp-05390",
|
| 2216 |
+
"r10-comp-05427",
|
| 2217 |
+
"r10-comp-05440",
|
| 2218 |
+
"r10-comp-05452",
|
| 2219 |
+
"r10-comp-05465",
|
| 2220 |
+
"r10-comp-05466",
|
| 2221 |
+
"r10-comp-05481",
|
| 2222 |
+
"r10-comp-05486",
|
| 2223 |
+
"r10-comp-05490",
|
| 2224 |
+
"r10-comp-05539",
|
| 2225 |
+
"r10-comp-05542",
|
| 2226 |
+
"r10-comp-05545",
|
| 2227 |
+
"r10-comp-05547",
|
| 2228 |
+
"r10-comp-05553",
|
| 2229 |
+
"r10-comp-05606",
|
| 2230 |
+
"r10-comp-05621",
|
| 2231 |
+
"r10-comp-05630",
|
| 2232 |
+
"r10-comp-05653",
|
| 2233 |
+
"r10-comp-05668",
|
| 2234 |
+
"r10-comp-05682",
|
| 2235 |
+
"r10-comp-05685",
|
| 2236 |
+
"r10-comp-05688",
|
| 2237 |
+
"r10-comp-05691",
|
| 2238 |
+
"r10-comp-05711",
|
| 2239 |
+
"r10-comp-05723",
|
| 2240 |
+
"r10-comp-05725",
|
| 2241 |
+
"r10-comp-05737",
|
| 2242 |
+
"r10-comp-05744",
|
| 2243 |
+
"r10-comp-05751",
|
| 2244 |
+
"r10-comp-05755",
|
| 2245 |
+
"r10-comp-05783",
|
| 2246 |
+
"r10-comp-05789",
|
| 2247 |
+
"r10-comp-05802",
|
| 2248 |
+
"r10-comp-05849",
|
| 2249 |
+
"r10-comp-05877",
|
| 2250 |
+
"r10-comp-05880",
|
| 2251 |
+
"r10-comp-05892",
|
| 2252 |
+
"r10-comp-05908",
|
| 2253 |
+
"r10-comp-05928",
|
| 2254 |
+
"r10-comp-05957",
|
| 2255 |
+
"r10-comp-05958",
|
| 2256 |
+
"r10-comp-05980",
|
| 2257 |
+
"r10-comp-06003",
|
| 2258 |
+
"r10-comp-06019",
|
| 2259 |
+
"r10-comp-06024",
|
| 2260 |
+
"r10-comp-06032",
|
| 2261 |
+
"r10-comp-06051",
|
| 2262 |
+
"r10-comp-06057",
|
| 2263 |
+
"r10-comp-06079",
|
| 2264 |
+
"r10-comp-06089",
|
| 2265 |
+
"r10-comp-06109",
|
| 2266 |
+
"r10-comp-06139",
|
| 2267 |
+
"r10-comp-06140",
|
| 2268 |
+
"r10-comp-06151",
|
| 2269 |
+
"r10-comp-06194",
|
| 2270 |
+
"r10-comp-06196",
|
| 2271 |
+
"r10-comp-06216",
|
| 2272 |
+
"r10-comp-06230",
|
| 2273 |
+
"r10-comp-06231",
|
| 2274 |
+
"r10-comp-06286",
|
| 2275 |
+
"r10-comp-06293",
|
| 2276 |
+
"r10-comp-06306",
|
| 2277 |
+
"r10-comp-06330",
|
| 2278 |
+
"r10-comp-06354",
|
| 2279 |
+
"r10-comp-06366",
|
| 2280 |
+
"r10-comp-06391",
|
| 2281 |
+
"r10-comp-06412",
|
| 2282 |
+
"r10-comp-06415",
|
| 2283 |
+
"r10-comp-06420",
|
| 2284 |
+
"r10-comp-06423",
|
| 2285 |
+
"r10-comp-06451",
|
| 2286 |
+
"r10-comp-06457",
|
| 2287 |
+
"r10-comp-06467",
|
| 2288 |
+
"r10-comp-06498",
|
| 2289 |
+
"r10-comp-06505",
|
| 2290 |
+
"r10-comp-06523",
|
| 2291 |
+
"r10-comp-06524",
|
| 2292 |
+
"r10-comp-06545",
|
| 2293 |
+
"r10-comp-06579",
|
| 2294 |
+
"r10-comp-06590",
|
| 2295 |
+
"r10-comp-06595",
|
| 2296 |
+
"r10-comp-06638",
|
| 2297 |
+
"r10-comp-06654",
|
| 2298 |
+
"r10-comp-06676",
|
| 2299 |
+
"r10-comp-06691",
|
| 2300 |
+
"r10-comp-06702",
|
| 2301 |
+
"r10-comp-06711",
|
| 2302 |
+
"r10-comp-06721",
|
| 2303 |
+
"r10-comp-06744",
|
| 2304 |
+
"r10-comp-06779",
|
| 2305 |
+
"r10-comp-06785",
|
| 2306 |
+
"r10-comp-06802",
|
| 2307 |
+
"r10-comp-06815",
|
| 2308 |
+
"r10-comp-06843",
|
| 2309 |
+
"r10-comp-06863",
|
| 2310 |
+
"r10-comp-06874",
|
| 2311 |
+
"r10-comp-06885",
|
| 2312 |
+
"r10-comp-06886",
|
| 2313 |
+
"r10-comp-06911",
|
| 2314 |
+
"r10-comp-06918",
|
| 2315 |
+
"r10-comp-06927",
|
| 2316 |
+
"r10-comp-06930",
|
| 2317 |
+
"r10-comp-06931",
|
| 2318 |
+
"r10-comp-06943",
|
| 2319 |
+
"r10-comp-06959",
|
| 2320 |
+
"r10-comp-06962",
|
| 2321 |
+
"r10-comp-06971",
|
| 2322 |
+
"r10-comp-06995",
|
| 2323 |
+
"r10-comp-07009",
|
| 2324 |
+
"r10-comp-07010",
|
| 2325 |
+
"r10-comp-07014",
|
| 2326 |
+
"r10-comp-07017",
|
| 2327 |
+
"r10-comp-07018",
|
| 2328 |
+
"r10-comp-07024",
|
| 2329 |
+
"r10-comp-07025",
|
| 2330 |
+
"r10-comp-07037",
|
| 2331 |
+
"r10-comp-07069",
|
| 2332 |
+
"r10-comp-07070",
|
| 2333 |
+
"r10-comp-07076",
|
| 2334 |
+
"r10-comp-07092",
|
| 2335 |
+
"r10-comp-07097",
|
| 2336 |
+
"r10-comp-07105",
|
| 2337 |
+
"r10-comp-07110",
|
| 2338 |
+
"r10-comp-07123",
|
| 2339 |
+
"r10-comp-07127",
|
| 2340 |
+
"r10-comp-07148",
|
| 2341 |
+
"r10-comp-07153",
|
| 2342 |
+
"r10-comp-07196",
|
| 2343 |
+
"r10-comp-07224",
|
| 2344 |
+
"r10-comp-07229",
|
| 2345 |
+
"r10-comp-07239",
|
| 2346 |
+
"r10-comp-07256",
|
| 2347 |
+
"r10-comp-07268",
|
| 2348 |
+
"r10-comp-07271",
|
| 2349 |
+
"r10-comp-07284",
|
| 2350 |
+
"r10-comp-07288",
|
| 2351 |
+
"r10-comp-07293",
|
| 2352 |
+
"r10-comp-07308",
|
| 2353 |
+
"r10-comp-07313",
|
| 2354 |
+
"r10-comp-07334",
|
| 2355 |
+
"r10-comp-07347",
|
| 2356 |
+
"r10-comp-07353",
|
| 2357 |
+
"r10-comp-07365",
|
| 2358 |
+
"r10-comp-07377",
|
| 2359 |
+
"r10-comp-07387",
|
| 2360 |
+
"r10-comp-07391",
|
| 2361 |
+
"r10-comp-07404",
|
| 2362 |
+
"r10-comp-07408",
|
| 2363 |
+
"r10-comp-07410",
|
| 2364 |
+
"r10-comp-07420",
|
| 2365 |
+
"r10-comp-07423",
|
| 2366 |
+
"r10-comp-07435",
|
| 2367 |
+
"r10-comp-07441",
|
| 2368 |
+
"r10-comp-07451",
|
| 2369 |
+
"r10-comp-07453",
|
| 2370 |
+
"r10-comp-07458",
|
| 2371 |
+
"r10-comp-07466",
|
| 2372 |
+
"r10-comp-07475",
|
| 2373 |
+
"r10-comp-07487",
|
| 2374 |
+
"r10-comp-07511",
|
| 2375 |
+
"r10-comp-07517",
|
| 2376 |
+
"r10-comp-07522",
|
| 2377 |
+
"r10-comp-07530",
|
| 2378 |
+
"r10-comp-07531",
|
| 2379 |
+
"r10-comp-07533",
|
| 2380 |
+
"r10-comp-07537",
|
| 2381 |
+
"r10-comp-07539",
|
| 2382 |
+
"r10-comp-07540",
|
| 2383 |
+
"r10-comp-07548",
|
| 2384 |
+
"r10-comp-07579",
|
| 2385 |
+
"r10-comp-07585",
|
| 2386 |
+
"r10-comp-07631",
|
| 2387 |
+
"r10-comp-07641",
|
| 2388 |
+
"r10-comp-07654",
|
| 2389 |
+
"r10-comp-07657",
|
| 2390 |
+
"r10-comp-07658",
|
| 2391 |
+
"r10-comp-07660",
|
| 2392 |
+
"r10-comp-07669",
|
| 2393 |
+
"r10-comp-07674",
|
| 2394 |
+
"r10-comp-07687",
|
| 2395 |
+
"r10-comp-07750",
|
| 2396 |
+
"r10-comp-07755",
|
| 2397 |
+
"r10-comp-07767",
|
| 2398 |
+
"r10-comp-07785",
|
| 2399 |
+
"r10-comp-07786",
|
| 2400 |
+
"r10-comp-07789",
|
| 2401 |
+
"r10-comp-07801",
|
| 2402 |
+
"r10-comp-07819",
|
| 2403 |
+
"r10-comp-07821",
|
| 2404 |
+
"r10-comp-07824",
|
| 2405 |
+
"r10-comp-07827",
|
| 2406 |
+
"r10-comp-07869",
|
| 2407 |
+
"r10-comp-07884",
|
| 2408 |
+
"r10-comp-07886",
|
| 2409 |
+
"r10-comp-07887",
|
| 2410 |
+
"r10-comp-07889",
|
| 2411 |
+
"r10-comp-07895",
|
| 2412 |
+
"r10-comp-07914",
|
| 2413 |
+
"r10-comp-07933",
|
| 2414 |
+
"r10-comp-07944",
|
| 2415 |
+
"r10-comp-07972",
|
| 2416 |
+
"r10-comp-07979",
|
| 2417 |
+
"r10-comp-08007",
|
| 2418 |
+
"r10-comp-08019",
|
| 2419 |
+
"r10-comp-08026",
|
| 2420 |
+
"r10-comp-08042",
|
| 2421 |
+
"r10-comp-08048",
|
| 2422 |
+
"r10-comp-08049",
|
| 2423 |
+
"r10-comp-08050",
|
| 2424 |
+
"r10-comp-08057",
|
| 2425 |
+
"r10-comp-08068",
|
| 2426 |
+
"r10-comp-08069",
|
| 2427 |
+
"r10-comp-08070",
|
| 2428 |
+
"r10-comp-08082",
|
| 2429 |
+
"r10-comp-08087",
|
| 2430 |
+
"r10-comp-08088",
|
| 2431 |
+
"r10-comp-08117",
|
| 2432 |
+
"r10-comp-08124",
|
| 2433 |
+
"r10-comp-08126",
|
| 2434 |
+
"r10-comp-08129",
|
| 2435 |
+
"r10-comp-08132",
|
| 2436 |
+
"r10-comp-08148",
|
| 2437 |
+
"r10-comp-08163",
|
| 2438 |
+
"r10-comp-08171",
|
| 2439 |
+
"r10-comp-08189",
|
| 2440 |
+
"r10-comp-08190",
|
| 2441 |
+
"r10-comp-08196",
|
| 2442 |
+
"r10-comp-08202",
|
| 2443 |
+
"r10-comp-08212",
|
| 2444 |
+
"r10-comp-08248",
|
| 2445 |
+
"r10-comp-08253",
|
| 2446 |
+
"r10-comp-08254",
|
| 2447 |
+
"r10-comp-08257",
|
| 2448 |
+
"r10-comp-08263",
|
| 2449 |
+
"r10-comp-08274",
|
| 2450 |
+
"r10-comp-08293",
|
| 2451 |
+
"r10-comp-08294",
|
| 2452 |
+
"r10-comp-08298",
|
| 2453 |
+
"r10-comp-08301",
|
| 2454 |
+
"r10-comp-08302",
|
| 2455 |
+
"r10-comp-08333",
|
| 2456 |
+
"r10-comp-08337",
|
| 2457 |
+
"r10-comp-08356",
|
| 2458 |
+
"r10-comp-08367",
|
| 2459 |
+
"r10-comp-08378",
|
| 2460 |
+
"r10-comp-08379",
|
| 2461 |
+
"r10-comp-08383",
|
| 2462 |
+
"r10-comp-08400",
|
| 2463 |
+
"r10-comp-08408",
|
| 2464 |
+
"r10-comp-08419",
|
| 2465 |
+
"r10-comp-08433",
|
| 2466 |
+
"r10-comp-08470",
|
| 2467 |
+
"r10-comp-08477",
|
| 2468 |
+
"r10-comp-08490",
|
| 2469 |
+
"r10-comp-08493",
|
| 2470 |
+
"r10-comp-08507",
|
| 2471 |
+
"r10-comp-08515",
|
| 2472 |
+
"r10-comp-08530",
|
| 2473 |
+
"r10-comp-08538",
|
| 2474 |
+
"r10-comp-08539",
|
| 2475 |
+
"r10-comp-08556",
|
| 2476 |
+
"r10-comp-08569",
|
| 2477 |
+
"r10-comp-08590",
|
| 2478 |
+
"r10-comp-08598",
|
| 2479 |
+
"r10-comp-08634",
|
| 2480 |
+
"r10-comp-08657",
|
| 2481 |
+
"r10-comp-08674",
|
| 2482 |
+
"r10-comp-08684",
|
| 2483 |
+
"r10-comp-08686",
|
| 2484 |
+
"r10-comp-08690",
|
| 2485 |
+
"r10-comp-08695",
|
| 2486 |
+
"r10-comp-08702",
|
| 2487 |
+
"r10-comp-08706",
|
| 2488 |
+
"r10-comp-08721",
|
| 2489 |
+
"r10-comp-08742",
|
| 2490 |
+
"r10-comp-08776",
|
| 2491 |
+
"r10-comp-08795",
|
| 2492 |
+
"r10-comp-08803",
|
| 2493 |
+
"r10-comp-08806",
|
| 2494 |
+
"r10-comp-08816",
|
| 2495 |
+
"r10-comp-08817",
|
| 2496 |
+
"r10-comp-08824",
|
| 2497 |
+
"r10-comp-08835",
|
| 2498 |
+
"r10-comp-08841",
|
| 2499 |
+
"r10-comp-08843",
|
| 2500 |
+
"r10-comp-08855",
|
| 2501 |
+
"r10-comp-08870",
|
| 2502 |
+
"r10-comp-08893",
|
| 2503 |
+
"r10-comp-08909",
|
| 2504 |
+
"r10-comp-08926",
|
| 2505 |
+
"r10-comp-08932",
|
| 2506 |
+
"r10-comp-08941",
|
| 2507 |
+
"r10-comp-08946",
|
| 2508 |
+
"r10-comp-08956",
|
| 2509 |
+
"r10-comp-08972",
|
| 2510 |
+
"r10-comp-08985",
|
| 2511 |
+
"r10-comp-09006",
|
| 2512 |
+
"r10-comp-09031",
|
| 2513 |
+
"r10-comp-09041",
|
| 2514 |
+
"r10-comp-09044",
|
| 2515 |
+
"r10-comp-09051",
|
| 2516 |
+
"r10-comp-09063",
|
| 2517 |
+
"r10-comp-09066",
|
| 2518 |
+
"r10-comp-09085",
|
| 2519 |
+
"r10-comp-09086",
|
| 2520 |
+
"r10-comp-09087",
|
| 2521 |
+
"r10-comp-09092",
|
| 2522 |
+
"r10-comp-09100",
|
| 2523 |
+
"r10-comp-09137",
|
| 2524 |
+
"r10-comp-09187",
|
| 2525 |
+
"r10-comp-09189",
|
| 2526 |
+
"r10-comp-09192",
|
| 2527 |
+
"r10-comp-09196",
|
| 2528 |
+
"r10-comp-09199",
|
| 2529 |
+
"r10-comp-09207",
|
| 2530 |
+
"r10-comp-09210",
|
| 2531 |
+
"r10-comp-09222",
|
| 2532 |
+
"r10-comp-09229",
|
| 2533 |
+
"r10-comp-09240",
|
| 2534 |
+
"r10-comp-09242",
|
| 2535 |
+
"r10-comp-09249",
|
| 2536 |
+
"r10-comp-09255",
|
| 2537 |
+
"r10-comp-09258",
|
| 2538 |
+
"r10-comp-09272",
|
| 2539 |
+
"r10-comp-09273",
|
| 2540 |
+
"r10-comp-09291",
|
| 2541 |
+
"r10-comp-09292",
|
| 2542 |
+
"r10-comp-09326",
|
| 2543 |
+
"r10-comp-09329"
|
| 2544 |
+
]
|
| 2545 |
+
}
|
| 2546 |
+
],
|
| 2547 |
+
"m": 30.0,
|
| 2548 |
+
"n_folds": 2,
|
| 2549 |
+
"n_rows": 1411,
|
| 2550 |
+
"n_scored": 1411,
|
| 2551 |
+
"oof_balanced_error": 0.041957059005251776,
|
| 2552 |
+
"raw_thresholds": {
|
| 2553 |
+
"audiobook": 0.399,
|
| 2554 |
+
"comic": 0.399,
|
| 2555 |
+
"course": 0.399,
|
| 2556 |
+
"ebook": 0.399,
|
| 2557 |
+
"game": 0.399,
|
| 2558 |
+
"movie": 0.306,
|
| 2559 |
+
"music": 0.399,
|
| 2560 |
+
"software": 0.399,
|
| 2561 |
+
"tv_show": 0.399,
|
| 2562 |
+
"unknown": 0.399,
|
| 2563 |
+
"xxx": 0.399
|
| 2564 |
+
},
|
| 2565 |
+
"schema": "armiger-r12-owner-scope-v1",
|
| 2566 |
+
"split": "val",
|
| 2567 |
+
"tau_global": 0.399,
|
| 2568 |
+
"thresholds": {
|
| 2569 |
+
"audiobook": 0.399,
|
| 2570 |
+
"comic": 0.39899999999999997,
|
| 2571 |
+
"course": 0.399,
|
| 2572 |
+
"ebook": 0.399,
|
| 2573 |
+
"game": 0.399,
|
| 2574 |
+
"movie": 0.31285503685503685,
|
| 2575 |
+
"music": 0.399,
|
| 2576 |
+
"software": 0.399,
|
| 2577 |
+
"tv_show": 0.399,
|
| 2578 |
+
"unknown": 0.399,
|
| 2579 |
+
"xxx": 0.399
|
| 2580 |
+
},
|
| 2581 |
+
"val_sha256": "36b66664e7f6ee49e1aae13fc7338294d845b2c7842e682e71a3485b050d7777"
|
| 2582 |
+
},
|
| 2583 |
+
"r13_allow_categories": [
|
| 2584 |
+
"movie",
|
| 2585 |
+
"tv_show"
|
| 2586 |
+
],
|
| 2587 |
+
"stratum_weighted_refit": {
|
| 2588 |
+
"spec_targets": {
|
| 2589 |
+
"allowed_coverage_at_least_98pct_test": false,
|
| 2590 |
+
"allowed_coverage_at_least_98pct_val": true,
|
| 2591 |
+
"course_leak_below_10pct": false
|
| 2592 |
+
},
|
| 2593 |
+
"val_audit": {
|
| 2594 |
+
"ht_weighted_global": {
|
| 2595 |
+
"allowed_coverage": 0.98,
|
| 2596 |
+
"course_leak_rate": 0.4524,
|
| 2597 |
+
"course_leaked": 19,
|
| 2598 |
+
"excluded_recall": 0.9471
|
| 2599 |
+
},
|
| 2600 |
+
"legacy_global": {
|
| 2601 |
+
"allowed_coverage": 0.9811,
|
| 2602 |
+
"course_leak_rate": 0.4524,
|
| 2603 |
+
"course_leaked": 19,
|
| 2604 |
+
"excluded_recall": 0.9471
|
| 2605 |
+
},
|
| 2606 |
+
"mondrian_by_gold_category_oracle": {
|
| 2607 |
+
"allowed_coverage": 0.9845,
|
| 2608 |
+
"course_leak_rate": 0.4524,
|
| 2609 |
+
"course_leaked": 19,
|
| 2610 |
+
"excluded_recall": 0.9471
|
| 2611 |
+
},
|
| 2612 |
+
"mondrian_by_predicted_category": {
|
| 2613 |
+
"allowed_coverage": 0.98,
|
| 2614 |
+
"course_leak_rate": 0.4524,
|
| 2615 |
+
"course_leaked": 19,
|
| 2616 |
+
"excluded_recall": 0.949
|
| 2617 |
+
},
|
| 2618 |
+
"n": 1411,
|
| 2619 |
+
"n_allowed": 901,
|
| 2620 |
+
"n_course_excluded": 42
|
| 2621 |
+
},
|
| 2622 |
+
"weights": {
|
| 2623 |
+
"cells": {
|
| 2624 |
+
"allowed_type|game": 153,
|
| 2625 |
+
"allowed_type|movie": 311,
|
| 2626 |
+
"allowed_type|software": 123,
|
| 2627 |
+
"allowed_type|tv_show": 314,
|
| 2628 |
+
"excluded|audiobook": 61,
|
| 2629 |
+
"excluded|comic": 163,
|
| 2630 |
+
"excluded|course": 42,
|
| 2631 |
+
"excluded|ebook": 60,
|
| 2632 |
+
"excluded|music": 184
|
| 2633 |
+
},
|
| 2634 |
+
"class": "owner_scope",
|
| 2635 |
+
"method": "horvitz_thompson_inverse_stratum w_i = 1/n_{c,k}",
|
| 2636 |
+
"stratum": "category"
|
| 2637 |
}
|
| 2638 |
},
|
|
|
|
| 2639 |
"temperature_audit": {
|
| 2640 |
+
"anime": {
|
| 2641 |
+
"acc_pct": 81.8035,
|
| 2642 |
+
"best_T_crossfit": 0.75,
|
| 2643 |
+
"ece5_at_T1_pct": 3.3623,
|
| 2644 |
+
"ece5_guarded_pct": 1.0578,
|
| 2645 |
+
"fold_Ts": [
|
| 2646 |
+
0.825,
|
| 2647 |
+
0.675
|
| 2648 |
+
],
|
| 2649 |
+
"n": 621,
|
| 2650 |
+
"shipped_T_guarded": 0.75,
|
| 2651 |
+
"weighted_nll_at_T1": 0.345181,
|
| 2652 |
+
"weighted_nll_at_best_T": 0.335153,
|
| 2653 |
+
"weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
|
| 2654 |
+
},
|
| 2655 |
"category": {
|
| 2656 |
+
"acc_pct": 84.55,
|
| 2657 |
+
"best_T_crossfit": 1.2,
|
| 2658 |
+
"ece5_at_T1_pct": 3.2368,
|
| 2659 |
+
"ece5_guarded_pct": 6.287,
|
| 2660 |
+
"fold_Ts": [
|
| 2661 |
+
1.25,
|
| 2662 |
+
1.15
|
| 2663 |
+
],
|
| 2664 |
+
"n": 1411,
|
| 2665 |
+
"shipped_T_guarded": 1.2,
|
| 2666 |
+
"weighted_nll_at_T1": 0.597522,
|
| 2667 |
+
"weighted_nll_at_best_T": 0.58716,
|
| 2668 |
+
"weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
|
| 2669 |
+
},
|
| 2670 |
+
"english_subs": {
|
| 2671 |
+
"acc_pct": 97.1698,
|
| 2672 |
+
"best_T_crossfit": 0.55,
|
| 2673 |
+
"ece5_at_T1_pct": 4.3801,
|
| 2674 |
+
"ece5_guarded_pct": 1.8806,
|
| 2675 |
+
"fold_Ts": [
|
| 2676 |
+
0.6,
|
| 2677 |
+
0.5
|
| 2678 |
+
],
|
| 2679 |
+
"n": 212,
|
| 2680 |
+
"shipped_T_guarded": 0.55,
|
| 2681 |
+
"weighted_nll_at_T1": 0.123028,
|
| 2682 |
+
"weighted_nll_at_best_T": 0.087029,
|
| 2683 |
+
"weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
|
| 2684 |
},
|
| 2685 |
"lang": {
|
| 2686 |
+
"acc_pct": 82.7434,
|
| 2687 |
+
"best_T_crossfit": 1.5375,
|
| 2688 |
+
"ece5_at_T1_pct": 8.9621,
|
| 2689 |
+
"ece5_guarded_pct": 2.3818,
|
| 2690 |
+
"fold_Ts": [
|
| 2691 |
+
1.675,
|
| 2692 |
+
1.4
|
| 2693 |
+
],
|
| 2694 |
+
"n": 226,
|
| 2695 |
+
"shipped_T_guarded": 1.5375,
|
| 2696 |
+
"weighted_nll_at_T1": 0.358405,
|
| 2697 |
+
"weighted_nll_at_best_T": 0.324181,
|
| 2698 |
+
"weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
|
| 2699 |
},
|
| 2700 |
"owner_scope": {
|
| 2701 |
+
"acc_pct": 98.4408,
|
| 2702 |
+
"best_T_crossfit": 1.8875,
|
| 2703 |
+
"ece5_at_T1_pct": 0.9298,
|
| 2704 |
+
"ece5_guarded_pct": 2.6221,
|
| 2705 |
+
"fold_Ts": [
|
| 2706 |
+
2.05,
|
| 2707 |
+
1.725
|
| 2708 |
+
],
|
| 2709 |
+
"n": 1411,
|
| 2710 |
+
"shipped_T_guarded": 1.8875,
|
| 2711 |
+
"weighted_nll_at_T1": 0.200121,
|
| 2712 |
+
"weighted_nll_at_best_T": 0.147816,
|
| 2713 |
+
"weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
|
| 2714 |
},
|
| 2715 |
"title_language": {
|
| 2716 |
+
"acc_pct": 97.832,
|
| 2717 |
+
"best_T_crossfit": 1.0125,
|
| 2718 |
+
"ece5_at_T1_pct": 0.7029,
|
| 2719 |
+
"ece5_guarded_pct": 0.7498,
|
| 2720 |
+
"fold_Ts": [
|
| 2721 |
+
1.175,
|
| 2722 |
+
0.85
|
| 2723 |
+
],
|
| 2724 |
+
"n": 369,
|
| 2725 |
+
"shipped_T_guarded": 1.0125,
|
| 2726 |
+
"weighted_nll_at_T1": 0.070129,
|
| 2727 |
+
"weighted_nll_at_best_T": 0.070032,
|
| 2728 |
+
"weighting": "2-fold group cross-fitted horvitz_thompson stratum_weight"
|
| 2729 |
+
}
|
| 2730 |
+
},
|
| 2731 |
+
"temperature_scaling_stop_fit": {
|
| 2732 |
+
"anime": {
|
| 2733 |
+
"cal_temp_scaled": {
|
| 2734 |
+
"fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
|
| 2735 |
+
"n": 621
|
| 2736 |
+
},
|
| 2737 |
+
"temperature_T": 0.75
|
| 2738 |
},
|
| 2739 |
+
"category": {
|
| 2740 |
+
"cal_temp_scaled": {
|
| 2741 |
+
"fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
|
| 2742 |
+
"n": 1411
|
| 2743 |
+
},
|
| 2744 |
+
"temperature_T": 1.2
|
|
|
|
|
|
|
|
|
|
| 2745 |
},
|
| 2746 |
+
"english_subs": {
|
| 2747 |
+
"cal_temp_scaled": {
|
| 2748 |
+
"fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
|
| 2749 |
+
"n": 212
|
| 2750 |
+
},
|
| 2751 |
+
"temperature_T": 0.55
|
|
|
|
|
|
|
|
|
|
| 2752 |
},
|
| 2753 |
+
"lang": {
|
| 2754 |
+
"cal_temp_scaled": {
|
| 2755 |
+
"fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
|
| 2756 |
+
"n": 226
|
| 2757 |
+
},
|
| 2758 |
+
"temperature_T": 1.5375
|
| 2759 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2760 |
"owner_scope": {
|
| 2761 |
+
"cal_temp_scaled": {
|
| 2762 |
+
"fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
|
| 2763 |
+
"n": 1411
|
| 2764 |
+
},
|
| 2765 |
+
"temperature_T": 1.8875
|
| 2766 |
+
},
|
| 2767 |
+
"title_language": {
|
| 2768 |
+
"cal_temp_scaled": {
|
| 2769 |
+
"fitted_on": "snap_r13_20261024T140340Z_064c8e6ccb1f_val_2fold_ht_guarded",
|
| 2770 |
+
"n": 369
|
| 2771 |
+
},
|
| 2772 |
+
"temperature_T": 1.0125
|
| 2773 |
}
|
| 2774 |
+
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2775 |
}
|
config.json
CHANGED
|
@@ -1,28 +1,27 @@
|
|
| 1 |
{
|
| 2 |
-
"model_name": "armiger
|
| 3 |
-
"version": "3.
|
| 4 |
-
"
|
| 5 |
-
"
|
|
|
|
|
|
|
|
|
|
| 6 |
"artifacts": {
|
| 7 |
-
"
|
| 8 |
-
"
|
| 9 |
-
"
|
| 10 |
-
"sha256": "a0dab886be6252739d4c89ce01d65d8d2135204beaec6b00159d038cf420f329"
|
| 11 |
},
|
| 12 |
-
"
|
| 13 |
-
"
|
| 14 |
-
"
|
| 15 |
-
"sha256": "6459a20d7bf45d46d59efbcb50ffb8a519bacda14ae36403dcc510eceaa38f44"
|
| 16 |
},
|
| 17 |
-
"
|
| 18 |
-
"
|
| 19 |
-
"
|
| 20 |
-
"sha256": "44cdcf1fac8ebc036262b2d3150b3ba0f27002669bf81e3480b1bc65e3ef8156"
|
| 21 |
},
|
| 22 |
-
"
|
| 23 |
-
"
|
| 24 |
-
"
|
| 25 |
-
"sha256": "85effcb9e70af7bff6047d4ae45d2155444e9b691dc84431f19ad14f9daee9ea"
|
| 26 |
}
|
| 27 |
},
|
| 28 |
"heads": {
|
|
@@ -68,15 +67,6 @@
|
|
| 68 |
"title_language": [
|
| 69 |
"english_title",
|
| 70 |
"non_english_title"
|
| 71 |
-
],
|
| 72 |
-
"span_bio": [
|
| 73 |
-
"O",
|
| 74 |
-
"B-TITLE",
|
| 75 |
-
"I-TITLE",
|
| 76 |
-
"B-YEAR",
|
| 77 |
-
"I-YEAR",
|
| 78 |
-
"B-EP",
|
| 79 |
-
"I-EP"
|
| 80 |
]
|
| 81 |
},
|
| 82 |
"mondrian_thresholds": {
|
|
@@ -84,118 +74,91 @@
|
|
| 84 |
"junk": 0.101,
|
| 85 |
"owner_scope": 0.477
|
| 86 |
},
|
| 87 |
-
"
|
| 88 |
-
"
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
"cpu_int8_single_item_latency_ms": 2.512,
|
| 104 |
-
"cpu_fp32_single_item_latency_ms": 2.637,
|
| 105 |
-
"cpu_int8_batch64_items_per_sec": 396.9,
|
| 106 |
-
"cpu_fp32_batch64_items_per_sec": 376.0
|
| 107 |
-
},
|
| 108 |
-
"bundled_plugins": {
|
| 109 |
-
"armiger-swarm-survival": {
|
| 110 |
-
"standalone_repo": "NorviTech/armiger-swarm-survival",
|
| 111 |
-
"files": [
|
| 112 |
-
"swarm_adapter.pt",
|
| 113 |
-
"swarm_survival.py"
|
| 114 |
-
],
|
| 115 |
-
"sha256": {
|
| 116 |
-
"swarm_adapter.pt": "85effcb9e70af7bff6047d4ae45d2155444e9b691dc84431f19ad14f9daee9ea",
|
| 117 |
-
"swarm_survival.py": "9dfbe7c4961f5a53bb4e40fc4450ebf958398157148d9f54da100d75dd24caf0"
|
| 118 |
-
}
|
| 119 |
},
|
| 120 |
-
"
|
| 121 |
-
"
|
| 122 |
-
"
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
"
|
| 126 |
-
"swarm_priors.py": "49b2b6857802606158e4a1de8cdc335e485be83aaec5405b9f2fd103ca90b394"
|
| 127 |
-
}
|
| 128 |
},
|
| 129 |
-
"
|
| 130 |
-
"
|
| 131 |
-
"
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
"
|
| 136 |
-
"zero_cue_plugin.py": "90039f19d36f2bcd0f0103deefe77705caf46e60c9f170d23d4719bc4a2ca020",
|
| 137 |
-
"validated_zero_cue_catalog.json": "8a8b7a4ba97cb22c3aa1f75ab33762052deb09580df464b1eabfe622a2837d26"
|
| 138 |
-
}
|
| 139 |
},
|
| 140 |
-
"
|
| 141 |
-
"
|
| 142 |
-
"
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
"
|
| 146 |
-
|
| 147 |
-
|
| 148 |
},
|
| 149 |
-
"
|
| 150 |
-
"
|
| 151 |
-
"
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
],
|
| 157 |
-
"sha256": {
|
| 158 |
-
"calibration.json": "e5b58fc6a8380b982f09296822b0244b28aa5b9b38c32d690c9e25160a8860bb",
|
| 159 |
-
"calibration_and_dht.py": "b5feaf9e9c5a8304c31675e6c0d4ab14a6ec3fd40bd24bafac109c5b139cbd23",
|
| 160 |
-
"armiger_dht_rsi.py": "a765952f40c384777af82bb7a7ce4d695e7f9cf12b9487aabb08d7bc3f60c707",
|
| 161 |
-
"test_armiger_dht_rsi.py": "af765f049fe684aedd5b20cf58e8a0cb8c4cc40fd501e50a78844bcc256ad1c4"
|
| 162 |
-
}
|
| 163 |
},
|
| 164 |
-
"
|
| 165 |
-
"
|
| 166 |
-
"
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
"
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
|
| 176 |
-
|
| 177 |
},
|
| 178 |
-
"
|
| 179 |
-
"
|
| 180 |
-
"
|
| 181 |
-
"
|
| 182 |
-
"
|
| 183 |
-
|
| 184 |
-
"adult",
|
| 185 |
-
"junk",
|
| 186 |
-
"anime",
|
| 187 |
-
"english_subs",
|
| 188 |
-
"lang",
|
| 189 |
-
"owner_scope",
|
| 190 |
-
"title_language",
|
| 191 |
-
"swarm_viability",
|
| 192 |
-
"honeypot_trap",
|
| 193 |
-
"csam_safety"
|
| 194 |
-
],
|
| 195 |
-
"sha256": {
|
| 196 |
-
"armiger_ultra_pipeline.py": "5a759dec66b9e13e532240d745afbb38826af90c5a9021c0eb845e6182cd1717",
|
| 197 |
-
"test_armiger_ultra_pipeline.py": "d4aeaf2143ac192a839e5fc0e14b1b5e03c84434274526dd1f38490d4de59f47"
|
| 198 |
-
}
|
| 199 |
},
|
| 200 |
-
"parameter_count": 8050139
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"model_name": "armiger",
|
| 3 |
+
"version": "3.3.0",
|
| 4 |
+
"architecture": "armiger-v2-byte-rope-transformer-8head",
|
| 5 |
+
"checkpoint": "H350000_Curated_Wave2-seed0 (direct-side hardened junk head)",
|
| 6 |
+
"format": "onnx-int8-dynamic",
|
| 7 |
+
"size_bytes": 8283542,
|
| 8 |
+
"sha256": "0f7496a70565145d09a8b29b09de9b99171cf483e5c32ff32e89df1931231401",
|
| 9 |
"artifacts": {
|
| 10 |
+
"armiger.onnx": {
|
| 11 |
+
"size_bytes": 17599636,
|
| 12 |
+
"sha256": "c0e96936038988c7465b7fe041c4dc3c92df5a8cb4f56c2666271cd4d3e79420"
|
|
|
|
| 13 |
},
|
| 14 |
+
"calibration.json": {
|
| 15 |
+
"size_bytes": 76612,
|
| 16 |
+
"sha256": "300387779dc16292dd9e6be038ce9dd60e5bdb0ee929b383dcd4e51715f8b5f8"
|
|
|
|
| 17 |
},
|
| 18 |
+
"vocab.json": {
|
| 19 |
+
"size_bytes": 39996,
|
| 20 |
+
"sha256": "c10afea430188e0207752e93bf0b340fbb585858684037c5134a477cbc0dd7bb"
|
|
|
|
| 21 |
},
|
| 22 |
+
"armiger_inference.py": {
|
| 23 |
+
"size_bytes": 38228,
|
| 24 |
+
"sha256": "c345587c57cecc518626852d0d8fcacf5ebb1576c10c0df6608f851a9ec72c31"
|
|
|
|
| 25 |
}
|
| 26 |
},
|
| 27 |
"heads": {
|
|
|
|
| 67 |
"title_language": [
|
| 68 |
"english_title",
|
| 69 |
"non_english_title"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 70 |
]
|
| 71 |
},
|
| 72 |
"mondrian_thresholds": {
|
|
|
|
| 74 |
"junk": 0.101,
|
| 75 |
"owner_scope": 0.477
|
| 76 |
},
|
| 77 |
+
"benchmarks_unmutated_ground_truth": {
|
| 78 |
+
"oct15_confirmation_n2835": {
|
| 79 |
+
"standalone_neural_11class_acc": 95.06,
|
| 80 |
+
"fair_hybrid_with_zero_cue_plugin_11class_acc": 95.49,
|
| 81 |
+
"standalone_neural_movie_vs_tv_acc": 96.6,
|
| 82 |
+
"fair_hybrid_movie_vs_tv_acc": 97.15,
|
| 83 |
+
"standalone_neural_catalog_matching_acc": 95.85,
|
| 84 |
+
"fair_hybrid_catalog_matching_acc": 96.39,
|
| 85 |
+
"standalone_neural_audio_lang_binary_n545_acc": 96.88,
|
| 86 |
+
"with_audio_guard_binary_n545_acc": 97.8,
|
| 87 |
+
"audio_lang_3class_n772_acc": 92.75,
|
| 88 |
+
"anime_n507_acc": 99.61,
|
| 89 |
+
"adult_n2781_acc": 98.53,
|
| 90 |
+
"junk_n2170_acc": 100.0,
|
| 91 |
+
"owner_scope_n2669_acc": 98.65,
|
| 92 |
+
"title_language_n969_acc": 93.7
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 93 |
},
|
| 94 |
+
"fresh_confirmation_n1402": {
|
| 95 |
+
"standalone_neural_11class_acc": 94.44,
|
| 96 |
+
"fair_hybrid_11class_acc": 94.58,
|
| 97 |
+
"adult_n1393_acc": 98.21,
|
| 98 |
+
"junk_n1291_acc": 100.0,
|
| 99 |
+
"anime_n253_acc": 99.6
|
|
|
|
|
|
|
| 100 |
},
|
| 101 |
+
"beyond_confirmation_n1025": {
|
| 102 |
+
"standalone_neural_11class_acc": 92.88,
|
| 103 |
+
"fair_hybrid_11class_acc": 95.12,
|
| 104 |
+
"adult_n1016_acc": 98.52,
|
| 105 |
+
"junk_n977_acc": 99.9,
|
| 106 |
+
"anime_n636_acc": 98.9,
|
| 107 |
+
"title_language_n604_acc": 98.51
|
|
|
|
|
|
|
|
|
|
| 108 |
},
|
| 109 |
+
"zero_cue_2000_n2000": {
|
| 110 |
+
"standalone_neural_11class_acc": 87.35,
|
| 111 |
+
"fair_hybrid_11class_acc": 91.5,
|
| 112 |
+
"adult_acc": 99.9,
|
| 113 |
+
"junk_acc": 100.0,
|
| 114 |
+
"anime_n1998_acc": 95.8,
|
| 115 |
+
"lang_acc": 88.15,
|
| 116 |
+
"title_language_acc": 97.75
|
| 117 |
},
|
| 118 |
+
"multilang_3000_n3000": {
|
| 119 |
+
"standalone_neural_11class_acc": 97.37,
|
| 120 |
+
"fair_hybrid_11class_acc": 98.4,
|
| 121 |
+
"adult_acc": 100.0,
|
| 122 |
+
"junk_acc": 100.0,
|
| 123 |
+
"anime_n2625_acc": 96.38,
|
| 124 |
+
"lang_acc": 91.47
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 125 |
},
|
| 126 |
+
"five_split_summary_n10262": {
|
| 127 |
+
"weighted_mean_standalone_neural_acc": 93.95,
|
| 128 |
+
"weighted_mean_fair_hybrid_acc": 95.39,
|
| 129 |
+
"macro_mean_standalone_neural_acc": 93.42,
|
| 130 |
+
"macro_mean_fair_hybrid_acc": 95.0
|
| 131 |
+
},
|
| 132 |
+
"swarm_liveliness": {
|
| 133 |
+
"way4_priors_formula_auroc": 86.64,
|
| 134 |
+
"with_swarm_adapter_public_tracker_n757_auroc": 92.42,
|
| 135 |
+
"with_swarm_adapter_public_holdout_n493_auroc": 93.03,
|
| 136 |
+
"with_swarm_adapter_multi_source_corpus_n10278_auroc": 96.83
|
| 137 |
+
},
|
| 138 |
+
"overall_6dim_composite_mean": 96.5
|
| 139 |
},
|
| 140 |
+
"hardware_latency_benchmarks": {
|
| 141 |
+
"apple_m4_pro_cpu_onnx_int8_single_item_p50_ms": 2.44,
|
| 142 |
+
"apple_m4_pro_cpu_onnx_int8_batch64_per_item_ms": 3.07,
|
| 143 |
+
"apple_m4_pro_cpu_onnx_int8_batch64_throughput_items_sec": 326.0,
|
| 144 |
+
"apple_m4_pro_mps_gpu_fp32_batch64_per_item_ms": 0.29,
|
| 145 |
+
"nvidia_rtx_gpu_fp32_batch64_per_item_ms": 0.21
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 146 |
},
|
| 147 |
+
"parameter_count": 8050139,
|
| 148 |
+
"benchmarks": {
|
| 149 |
+
"heldout_10class_category_acc": 90.94,
|
| 150 |
+
"beyond_confirmation_category_acc": 90.44,
|
| 151 |
+
"catalog_matching_rate_acc": 90.34,
|
| 152 |
+
"audio_language_acc": 98.25,
|
| 153 |
+
"anime_and_dub_acc": 97.48,
|
| 154 |
+
"adult_safety_gate_acc": 100.0,
|
| 155 |
+
"junk_malware_filter_acc": 99.9,
|
| 156 |
+
"swarm_liveliness_public_auroc": 92.42,
|
| 157 |
+
"swarm_liveliness_multisource_auroc": 96.83,
|
| 158 |
+
"fake_seeder_honeypot_recall": 100.0,
|
| 159 |
+
"overall_6dim_composite_mean": 94.09,
|
| 160 |
+
"gpu_batched_latency_ms": 0.21,
|
| 161 |
+
"gpu_single_item_latency_ms": 1.12,
|
| 162 |
+
"cpu_int8_batched_latency_ms": 0.42
|
| 163 |
+
}
|
| 164 |
}
|
r13_feature_extractor.py
ADDED
|
@@ -0,0 +1,406 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""R13 S2 features and a read-only adapter for frozen R12 ONNX checkpoints.
|
| 2 |
+
|
| 3 |
+
There are two feature views. Category keeps the R12 enriched input; title
|
| 4 |
+
language sees the literal title, without file names or file statistics. S8
|
| 5 |
+
should train those views together, rather than transplanting a language head.
|
| 6 |
+
|
| 7 |
+
Only two semantic rules are admitted: identifier neutrality for the binary
|
| 8 |
+
title-language task, and a bracketed release/episode/technical-tags/video cue
|
| 9 |
+
for anime. ``audit_train_rules`` supplies their train-only precision gates.
|
| 10 |
+
Dual-audio and ASS are observable features, never standalone anime rules.
|
| 11 |
+
"""
|
| 12 |
+
from __future__ import annotations
|
| 13 |
+
|
| 14 |
+
from collections import Counter
|
| 15 |
+
from dataclasses import dataclass
|
| 16 |
+
from functools import lru_cache
|
| 17 |
+
import hashlib
|
| 18 |
+
import importlib.util
|
| 19 |
+
import math
|
| 20 |
+
from pathlib import Path
|
| 21 |
+
import re
|
| 22 |
+
from typing import Any, Mapping, Sequence
|
| 23 |
+
|
| 24 |
+
import numpy as np
|
| 25 |
+
|
| 26 |
+
PROJECT_ROOT = Path(__file__).resolve().parent
|
| 27 |
+
SIDE_DIM = 28
|
| 28 |
+
BASE_STATS_DIM = 12
|
| 29 |
+
STATS_DIM = 40
|
| 30 |
+
MIN_SUPPORT = 5
|
| 31 |
+
MIN_PRECISION = 0.90
|
| 32 |
+
|
| 33 |
+
# These are syntax families from the session brief, not title/group catalogs.
|
| 34 |
+
NUMERIC_ID_RE = re.compile(r"[0-9]+(?:-[0-9]+)?", re.ASCII)
|
| 35 |
+
PG_ID_RE = re.compile(r"pg[0-9]+(?:-[0-9]+)?", re.I | re.ASCII)
|
| 36 |
+
GUT_ID_RE = re.compile(r"[a-z0-9_.-]+gut", re.I | re.ASCII)
|
| 37 |
+
PACKAGE_ID_RE = re.compile(r"com(?:\.[a-z][a-z0-9_]*){2,}", re.I | re.ASCII)
|
| 38 |
+
FANSUB_EPISODE_RE = re.compile(
|
| 39 |
+
r"^\[(?P<group>[^\[\]\r\n]{2,80})\]\s*(?P<title>.+?)\s+-\s+"
|
| 40 |
+
r"(?P<episode>[0-9]{1,3})(?:v[0-9]+)?\s*"
|
| 41 |
+
r"(?P<tags>(?:(?:\[[^\[\]\r\n]+\]|\([^()\r\n]+\))\s*)+)"
|
| 42 |
+
r"\.(?:mkv|mp4|avi|webm)$", re.I,
|
| 43 |
+
)
|
| 44 |
+
RESOLUTION_RE = re.compile(r"\b(?:480|576|720|1080|1440|2160|4320)[pi]\b", re.I)
|
| 45 |
+
JAPANESE_RE = re.compile(r"(?<![a-z])(?:jpn|japanese)(?![a-z])", re.I)
|
| 46 |
+
ENGLISH_RE = re.compile(r"(?<![a-z])(?:eng|english)(?![a-z])", re.I)
|
| 47 |
+
DUAL_AUDIO_RE = re.compile(r"\bdual[ ._-]*audio\b", re.I)
|
| 48 |
+
VIDEO_EXTS = frozenset({"mkv", "mp4", "avi", "webm"})
|
| 49 |
+
RULE_IDENTIFIER = "identifier_title_neutrality"
|
| 50 |
+
RULE_FANSUB = "bracket_episode_resolution_video"
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
@lru_cache(maxsize=1)
|
| 54 |
+
def r12_features():
|
| 55 |
+
"""Load the named local baseline, without importing a trainer or dataset."""
|
| 56 |
+
path = PROJECT_ROOT / "_r12_features_base.py"
|
| 57 |
+
spec = importlib.util.spec_from_file_location("_s2_r12_baseline", path)
|
| 58 |
+
if spec is None or spec.loader is None:
|
| 59 |
+
raise ImportError(f"Cannot load baseline featurizer: {path}")
|
| 60 |
+
module = importlib.util.module_from_spec(spec)
|
| 61 |
+
spec.loader.exec_module(module)
|
| 62 |
+
if (module.SIDE_DIM, module.STATS_DIM) != (SIDE_DIM, STATS_DIM):
|
| 63 |
+
raise ValueError("R12 feature contract changed; requalify S2")
|
| 64 |
+
return module
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def normalize_files(files: Any) -> list[dict[str, Any]]:
|
| 68 |
+
"""Support SDK mappings, path/size pairs, and plain relative paths."""
|
| 69 |
+
result = []
|
| 70 |
+
for entry in files or []:
|
| 71 |
+
if isinstance(entry, Mapping):
|
| 72 |
+
path = entry.get("path") or entry.get("p") or entry.get("name") or ""
|
| 73 |
+
size = entry.get("size") or entry.get("s") or 0
|
| 74 |
+
elif isinstance(entry, (tuple, list)):
|
| 75 |
+
path = entry[0] if entry else ""
|
| 76 |
+
size = entry[1] if len(entry) > 1 else 0
|
| 77 |
+
else:
|
| 78 |
+
path, size = entry, 0
|
| 79 |
+
try:
|
| 80 |
+
size = max(0, int(size))
|
| 81 |
+
except (TypeError, ValueError, OverflowError):
|
| 82 |
+
size = 0
|
| 83 |
+
path = str(path or "").replace("\\", "/")
|
| 84 |
+
if path and "/.____padding_file/" not in f"/{path}":
|
| 85 |
+
result.append({"path": path, "size": size})
|
| 86 |
+
return result
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def _safe_nonneg_int(val: Any) -> int:
|
| 90 |
+
try:
|
| 91 |
+
return max(0, int(val or 0))
|
| 92 |
+
except (TypeError, ValueError, OverflowError):
|
| 93 |
+
return 0
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
def inference_input(row: Mapping[str, Any]) -> dict[str, Any]:
|
| 97 |
+
"""Use R12's production firewall: name/files/size/infohash, no language hint.
|
| 98 |
+
|
| 99 |
+
Matches active_source/armiger_bench.py::sanitize_inference_input for snapshot
|
| 100 |
+
mappings, while accepting SDK path/size aliases for clean external inputs.
|
| 101 |
+
"""
|
| 102 |
+
source = row.get("inference_input", row)
|
| 103 |
+
if not isinstance(source, Mapping):
|
| 104 |
+
raise TypeError("inference_input must be a mapping")
|
| 105 |
+
return {"name": str(source.get("name") or source.get("title") or ""),
|
| 106 |
+
"files": normalize_files(source.get("files")),
|
| 107 |
+
"size": _safe_nonneg_int(source.get("size")),
|
| 108 |
+
"infohash": str(source.get("infohash") or "").lower()}
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def _utf8_prefix(text: str, max_bytes: int) -> str:
|
| 112 |
+
if max_bytes < 1:
|
| 113 |
+
raise ValueError("max_bytes must be positive")
|
| 114 |
+
return str(text).encode("utf-8", "replace")[:max_bytes].decode("utf-8", "ignore")
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
def build_title_language_text(name: str, files: Any = None, max_bytes: int = 256) -> str:
|
| 118 |
+
"""Title-only view. Files deliberately have no effect on this representation."""
|
| 119 |
+
return _utf8_prefix(name, max_bytes)
|
| 120 |
+
|
| 121 |
+
|
| 122 |
+
def build_enriched_name_text(
|
| 123 |
+
name: str, files: Any, max_bytes: int = 256, *, head: str = "category",
|
| 124 |
+
) -> str:
|
| 125 |
+
"""Keep R12 enrichment for category; expose a separate title-language view."""
|
| 126 |
+
if head == "title_language":
|
| 127 |
+
return build_title_language_text(name, max_bytes=max_bytes)
|
| 128 |
+
text = r12_features().build_enriched_name_text(name, normalize_files(files), max_bytes=max_bytes)
|
| 129 |
+
return _utf8_prefix(text, max_bytes)
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def _id_stem(name: str) -> str:
|
| 133 |
+
stem = name.strip()
|
| 134 |
+
# Exact opaque file identifiers are also supported, but not arbitrary paths.
|
| 135 |
+
if stem.lower().endswith((".txt", ".zip")):
|
| 136 |
+
stem = stem.rsplit(".", 1)[0]
|
| 137 |
+
return stem
|
| 138 |
+
|
| 139 |
+
|
| 140 |
+
def identifier_family(name: str, files: Any = None) -> str | None:
|
| 141 |
+
"""Return a nonlinguistic-ID family, never infer a work or spoken language."""
|
| 142 |
+
stem = _id_stem(name)
|
| 143 |
+
for family, pattern in (("numeric", NUMERIC_ID_RE), ("pg_numeric", PG_ID_RE),
|
| 144 |
+
("gut_stem", GUT_ID_RE), ("ascii_package", PACKAGE_ID_RE)):
|
| 145 |
+
if pattern.fullmatch(stem):
|
| 146 |
+
return family
|
| 147 |
+
# TXT+ZIP alone is not enough: require an opaque Gutenberg-shaped stem
|
| 148 |
+
# and exactly matching relative path, rather than unrelated archive files.
|
| 149 |
+
by_ext: dict[str, set[str]] = {"txt": set(), "zip": set()}
|
| 150 |
+
for item in normalize_files(files):
|
| 151 |
+
path = item["path"].lower()
|
| 152 |
+
base, dot, ext = path.rpartition(".")
|
| 153 |
+
if dot and ext in by_ext:
|
| 154 |
+
leaf = base.rsplit("/", 1)[-1]
|
| 155 |
+
if NUMERIC_ID_RE.fullmatch(leaf) or PG_ID_RE.fullmatch(leaf):
|
| 156 |
+
by_ext[ext].add(base)
|
| 157 |
+
# A readable supplied title must retain its own language, even with book files.
|
| 158 |
+
opaque_name = not re.search(r"[a-z]{3,}", name, re.I | re.ASCII)
|
| 159 |
+
if opaque_name and by_ext["txt"] & by_ext["zip"]:
|
| 160 |
+
return "gutenberg_txt_zip"
|
| 161 |
+
return None
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
@dataclass(frozen=True)
|
| 165 |
+
class StructuralCues:
|
| 166 |
+
fansub_episode_keys: frozenset[int]
|
| 167 |
+
fansub_video_count: int
|
| 168 |
+
has_mkv: bool
|
| 169 |
+
has_ass_subtitles: bool
|
| 170 |
+
explicit_jpn_eng_dual_audio: bool
|
| 171 |
+
|
| 172 |
+
@property
|
| 173 |
+
def fansub_episode(self) -> bool:
|
| 174 |
+
return bool(self.fansub_episode_keys)
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
def extract_structural_cues(name: str, files: Any = None) -> StructuralCues:
|
| 178 |
+
"""Parse arbitrary group tags; there are no release-group or work literals."""
|
| 179 |
+
entries = normalize_files(files)
|
| 180 |
+
texts = [name, *(f["path"].rsplit("/", 1)[-1] for f in entries)]
|
| 181 |
+
episodes: set[int] = set()
|
| 182 |
+
matched: set[str] = set()
|
| 183 |
+
for text in texts:
|
| 184 |
+
match = FANSUB_EPISODE_RE.fullmatch(text)
|
| 185 |
+
if match and RESOLUTION_RE.search(match["tags"]):
|
| 186 |
+
episodes.add(int(match["episode"]))
|
| 187 |
+
matched.add(text)
|
| 188 |
+
exts = {t.rsplit(".", 1)[-1].lower() for t in texts if "." in t}
|
| 189 |
+
# These fields describe metadata only. They do not assert anime/sub language.
|
| 190 |
+
audio_text = " ".join(texts)
|
| 191 |
+
dual = bool(DUAL_AUDIO_RE.search(audio_text) and JAPANESE_RE.search(audio_text)
|
| 192 |
+
and ENGLISH_RE.search(audio_text) and "mkv" in exts)
|
| 193 |
+
return StructuralCues(frozenset(episodes), len(matched), "mkv" in exts, "ass" in exts, dual)
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
def compute_derivative_group_features(files: Any, raw_name: str = "") -> tuple[float, ...]:
|
| 197 |
+
"""R12 U/P/share semantics plus explicit bracketed episode/version parsing.
|
| 198 |
+
|
| 199 |
+
Episode numbers in SxxExx and fansub paths refer to the same episodes: use
|
| 200 |
+
the larger count rather than summing two parsers' possibly duplicate keys.
|
| 201 |
+
S8 receives raw counts as well as the bounded side-vector representation.
|
| 202 |
+
"""
|
| 203 |
+
normalized = normalize_files(files)
|
| 204 |
+
if not normalized and raw_name.rsplit(".", 1)[-1].lower() in VIDEO_EXTS:
|
| 205 |
+
normalized = [{"path": raw_name, "size": 0}]
|
| 206 |
+
# The derivative view drops resolution/version tags only after a full
|
| 207 |
+
# structural match. Keep parent+title+episode, so separate episodes/series
|
| 208 |
+
# cannot collapse merely because their codec, group, or episode number agrees.
|
| 209 |
+
derivative_view = []
|
| 210 |
+
for entry in normalized:
|
| 211 |
+
parent, sep, leaf = entry["path"].rpartition("/")
|
| 212 |
+
match = FANSUB_EPISODE_RE.fullmatch(leaf)
|
| 213 |
+
if match and RESOLUTION_RE.search(match["tags"]):
|
| 214 |
+
ext = leaf.rsplit(".", 1)[-1]
|
| 215 |
+
leaf = f"{match['title']}.E{int(match['episode']):03d}.{ext}"
|
| 216 |
+
derivative_view.append({"path": parent + sep + leaf, "size": entry["size"]})
|
| 217 |
+
else:
|
| 218 |
+
derivative_view.append(entry)
|
| 219 |
+
u, e, p, share = r12_features().compute_derivative_group_features(derivative_view, raw_name)
|
| 220 |
+
fan_count = len(extract_structural_cues(raw_name, normalized).fansub_episode_keys)
|
| 221 |
+
e = max(e, float(fan_count))
|
| 222 |
+
return u, e, (0.0 if e >= 2 else p), share
|
| 223 |
+
|
| 224 |
+
|
| 225 |
+
def encode_torrent_features(row: Mapping[str, Any], max_name_len: int = 256,
|
| 226 |
+
max_files: int = 64) -> dict[str, Any]:
|
| 227 |
+
"""S8 training/inference encoder. R12 checkpoints use the adapter below.
|
| 228 |
+
|
| 229 |
+
New keys: title_name_bytes, title_name_text, title_identifier_family,
|
| 230 |
+
structural_cues. Existing stats remain 40 wide. side[13]/side[15] gain
|
| 231 |
+
subtitle presence and a validated generic fansub cue. side[24:28] preserve
|
| 232 |
+
their U/E/P/share meaning; no new checkpoint width is silently introduced.
|
| 233 |
+
"""
|
| 234 |
+
clean = inference_input(row)
|
| 235 |
+
name = str(clean.get("name") or clean.get("title") or "")
|
| 236 |
+
encoded = r12_features().encode_torrent_features(clean, max_name_len, max_files)
|
| 237 |
+
cues = extract_structural_cues(name, clean["files"])
|
| 238 |
+
deriv = compute_derivative_group_features(clean["files"], name)
|
| 239 |
+
stats = list(encoded["stats"])
|
| 240 |
+
if cues.fansub_episode:
|
| 241 |
+
stats[BASE_STATS_DIM + 15] = 1.0
|
| 242 |
+
if cues.has_ass_subtitles:
|
| 243 |
+
stats[BASE_STATS_DIM + 13] = 1.0
|
| 244 |
+
u, e, p, share = deriv
|
| 245 |
+
stats[BASE_STATS_DIM + 24:] = [min(1.0, math.log1p(u) / math.log1p(64)),
|
| 246 |
+
min(1.0, math.log1p(e) / math.log1p(64)), p, share]
|
| 247 |
+
title = build_title_language_text(name, max_bytes=max_name_len)
|
| 248 |
+
encoded.update(stats=stats, deriv_features=deriv, structural_cues=cues,
|
| 249 |
+
title_name_text=title, title_name_bytes=list(title.encode("utf-8")) or [32],
|
| 250 |
+
title_identifier_family=identifier_family(name, clean["files"]))
|
| 251 |
+
return encoded
|
| 252 |
+
|
| 253 |
+
|
| 254 |
+
def build_title_language_batch_inputs(rows: Sequence[Mapping[str, Any]],
|
| 255 |
+
stats_dim: int = STATS_DIM) -> dict[str, np.ndarray]:
|
| 256 |
+
"""A fully separate name view: no file bytes, sizes, metadata, or language hint.
|
| 257 |
+
|
| 258 |
+
Directly replacing bytes avoids R12's file fallback for dotted names. All
|
| 259 |
+
non-title tensors have the same constant value for every title.
|
| 260 |
+
"""
|
| 261 |
+
names = [str((r := inference_input(row)).get("name") or r.get("title") or "") for row in rows]
|
| 262 |
+
data = [list(build_title_language_text(n).encode("utf-8")) or [32] for n in names]
|
| 263 |
+
batch = len(data)
|
| 264 |
+
if batch == 0:
|
| 265 |
+
raise ValueError("rows must not be empty")
|
| 266 |
+
name_bytes = np.zeros((batch, max(map(len, data))), dtype=np.int64)
|
| 267 |
+
for i, values in enumerate(data):
|
| 268 |
+
name_bytes[i, :len(values)] = values
|
| 269 |
+
return {"name_bytes": name_bytes, "name_len": np.asarray(list(map(len, data)), dtype=np.int64),
|
| 270 |
+
"ext": np.zeros((batch, 1), dtype=np.int32),
|
| 271 |
+
"size": np.zeros((batch, 1), dtype=np.int64),
|
| 272 |
+
"depth": np.zeros((batch, 1), dtype=np.int64),
|
| 273 |
+
"words": np.zeros((batch, 1, 4), dtype=np.int32),
|
| 274 |
+
"stats": np.zeros((batch, stats_dim), dtype=np.float32)}
|
| 275 |
+
|
| 276 |
+
|
| 277 |
+
def _label(row: Mapping[str, Any], head: str) -> str | None:
|
| 278 |
+
target = (row.get("targets") or {}).get(head) or {}
|
| 279 |
+
return target.get("value") if target.get("known_mask") else None
|
| 280 |
+
|
| 281 |
+
|
| 282 |
+
def audit_train_rules(rows: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
|
| 283 |
+
"""Freeze support using train labels only; unknown masks never count as gold.
|
| 284 |
+
|
| 285 |
+
The identifier rule is one pre-specified union from the user's brief.
|
| 286 |
+
Subgroup counts are retained so absent supervision is explicit.
|
| 287 |
+
"""
|
| 288 |
+
counts = {RULE_IDENTIFIER: Counter(), RULE_FANSUB: Counter()}
|
| 289 |
+
subgroups: dict[str, Counter] = {}
|
| 290 |
+
for row in rows:
|
| 291 |
+
if row.get("split") != "train":
|
| 292 |
+
raise ValueError("Rule admission accepts train-marked rows only")
|
| 293 |
+
clean = inference_input(row)
|
| 294 |
+
name = str(clean.get("name") or clean.get("title") or "")
|
| 295 |
+
family = identifier_family(name, clean["files"])
|
| 296 |
+
if family:
|
| 297 |
+
value = _label(row, "title_language") or "unknown"
|
| 298 |
+
counts[RULE_IDENTIFIER][value] += 1
|
| 299 |
+
subgroups.setdefault(family, Counter())[value] += 1
|
| 300 |
+
if extract_structural_cues(name, clean["files"]).fansub_episode:
|
| 301 |
+
counts[RULE_FANSUB][_label(row, "anime") or "unknown"] += 1
|
| 302 |
+
report: dict[str, Any] = {}
|
| 303 |
+
for rule, positive in ((RULE_IDENTIFIER, "english_title"), (RULE_FANSUB, "anime")):
|
| 304 |
+
c = counts[rule]
|
| 305 |
+
support = sum(v for k, v in c.items() if k != "unknown")
|
| 306 |
+
precision = c[positive] / support if support else 0.0
|
| 307 |
+
report[rule] = {"counts": dict(c), "known_support": support, "precision": precision,
|
| 308 |
+
"admitted": support >= MIN_SUPPORT and precision >= MIN_PRECISION}
|
| 309 |
+
report["identifier_subgroups"] = {k: dict(v) for k, v in subgroups.items()}
|
| 310 |
+
report["dual_audio_ass_standalone_rule_admitted"] = False
|
| 311 |
+
return report
|
| 312 |
+
|
| 313 |
+
|
| 314 |
+
def apply_head_guards(row: Mapping[str, Any], predictions: Mapping[str, str],
|
| 315 |
+
train_audit: Mapping[str, Any], *,
|
| 316 |
+
enable_identifier_guard: bool = False) -> tuple[dict[str, str], dict[str, str]]:
|
| 317 |
+
"""Copy predictions, modify only title_language/anime, and expose provenance.
|
| 318 |
+
|
| 319 |
+
english_title is the existing binary task's compatibility value for an
|
| 320 |
+
identifier without a readable title. It is not an assertion about book text,
|
| 321 |
+
spoken language, subtitle language, country, or the title behind an ID.
|
| 322 |
+
"""
|
| 323 |
+
clean = inference_input(row)
|
| 324 |
+
name = str(clean.get("name") or clean.get("title") or "")
|
| 325 |
+
result, reasons = dict(predictions), {}
|
| 326 |
+
family = identifier_family(name, clean["files"])
|
| 327 |
+
if (enable_identifier_guard and family
|
| 328 |
+
and train_audit.get(RULE_IDENTIFIER, {}).get("admitted")):
|
| 329 |
+
result["title_language"] = "english_title"
|
| 330 |
+
reasons["title_language"] = f"{RULE_IDENTIFIER}:{family}"
|
| 331 |
+
if (extract_structural_cues(name, clean["files"]).fansub_episode
|
| 332 |
+
and train_audit.get(RULE_FANSUB, {}).get("admitted")):
|
| 333 |
+
result["anime"] = "anime"
|
| 334 |
+
reasons["anime"] = RULE_FANSUB
|
| 335 |
+
return result, reasons
|
| 336 |
+
|
| 337 |
+
|
| 338 |
+
class R13FrozenR12Adapter:
|
| 339 |
+
"""Head-local compatibility probe, never writes/changes model weights.
|
| 340 |
+
|
| 341 |
+
Category and all unrelated heads come from the original R12 feature feed.
|
| 342 |
+
Title-only inference is used for opaque IDs or names R12 enriched; ordinary
|
| 343 |
+
titles retain the frozen model's trained feature distribution. S8 can use
|
| 344 |
+
title_name_bytes for every row during native joint training.
|
| 345 |
+
|
| 346 |
+
Blanket identifier neutrality is an opt-in compatibility candidate: it
|
| 347 |
+
passes the pooled train gate but regresses known validation titles. Always
|
| 348 |
+
expose its ablation separately; the safe default uses title-only inference.
|
| 349 |
+
"""
|
| 350 |
+
def __init__(self, model_path: str | Path, train_audit: Mapping[str, Any], threads: int = 2,
|
| 351 |
+
*, enable_identifier_guard: bool = False):
|
| 352 |
+
import onnxruntime as ort
|
| 353 |
+
options = ort.SessionOptions()
|
| 354 |
+
options.intra_op_num_threads = threads
|
| 355 |
+
options.inter_op_num_threads = 1
|
| 356 |
+
self.session = ort.InferenceSession(str(model_path), sess_options=options,
|
| 357 |
+
providers=["CPUExecutionProvider"])
|
| 358 |
+
self.stats_dim = next(i.shape[1] for i in self.session.get_inputs() if i.name == "stats")
|
| 359 |
+
self.train_audit = train_audit
|
| 360 |
+
self.enable_identifier_guard = enable_identifier_guard
|
| 361 |
+
|
| 362 |
+
def predict_batch(self, rows: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]:
|
| 363 |
+
if not rows:
|
| 364 |
+
return []
|
| 365 |
+
baseline = r12_features()
|
| 366 |
+
clean = [inference_input(r) for r in rows]
|
| 367 |
+
head_names = [o.name for o in self.session.get_outputs()
|
| 368 |
+
if o.name in baseline.NEURAL_HEAD_CLASSES and len(o.shape) == 2]
|
| 369 |
+
feed = baseline.build_onnx_batch_inputs(clean, stats_dim=self.stats_dim)
|
| 370 |
+
logits = dict(zip(head_names, self.session.run(head_names, feed)))
|
| 371 |
+
routed = []
|
| 372 |
+
for i, row in enumerate(clean):
|
| 373 |
+
name = str(row.get("name") or row.get("title") or "")
|
| 374 |
+
if (identifier_family(name, row["files"])
|
| 375 |
+
or baseline.build_enriched_name_text(name, row["files"]) != name):
|
| 376 |
+
routed.append(i)
|
| 377 |
+
title_logits = None
|
| 378 |
+
if routed:
|
| 379 |
+
title_feed = build_title_language_batch_inputs([clean[i] for i in routed], self.stats_dim)
|
| 380 |
+
title_logits = self.session.run(["title_language"], title_feed)[0]
|
| 381 |
+
route_index = {index: j for j, index in enumerate(routed)}
|
| 382 |
+
result = []
|
| 383 |
+
for i, row in enumerate(clean):
|
| 384 |
+
raw = {h: baseline.NEURAL_HEAD_CLASSES[h][int(np.argmax(values[i]))]
|
| 385 |
+
for h, values in logits.items()}
|
| 386 |
+
view = dict(raw)
|
| 387 |
+
if i in route_index:
|
| 388 |
+
view["title_language"] = baseline.NEURAL_HEAD_CLASSES["title_language"][
|
| 389 |
+
int(np.argmax(title_logits[route_index[i]]))]
|
| 390 |
+
guarded, reasons = apply_head_guards(row, view, self.train_audit,
|
| 391 |
+
enable_identifier_guard=self.enable_identifier_guard)
|
| 392 |
+
candidate, candidate_reasons = apply_head_guards(row, view, self.train_audit,
|
| 393 |
+
enable_identifier_guard=True)
|
| 394 |
+
result.append({"baseline": raw, "title_only": view, "predictions": guarded,
|
| 395 |
+
"guard_reasons": reasons, "title_view_routed": i in route_index,
|
| 396 |
+
"identifier_policy_candidate": candidate,
|
| 397 |
+
"identifier_policy_guard_reasons": candidate_reasons})
|
| 398 |
+
return result
|
| 399 |
+
|
| 400 |
+
|
| 401 |
+
def sha256_file(path: str | Path) -> str:
|
| 402 |
+
digest = hashlib.sha256()
|
| 403 |
+
with Path(path).open("rb") as stream:
|
| 404 |
+
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
| 405 |
+
digest.update(chunk)
|
| 406 |
+
return digest.hexdigest()
|
r13_features.py
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Explicit R13 tensor ABI: 40 S2 features plus raw title byte length.
|
| 2 |
+
|
| 3 |
+
The last scalar is a mask boundary, never a learned feature. Both views share
|
| 4 |
+
the transformer; the title head cannot attend to appended file names.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import numpy as np
|
| 10 |
+
|
| 11 |
+
try:
|
| 12 |
+
from . import r13_feature_extractor as s2
|
| 13 |
+
except ImportError:
|
| 14 |
+
try:
|
| 15 |
+
import r13_feature_extractor as s2
|
| 16 |
+
except ImportError:
|
| 17 |
+
from r13_master_program.s2_neural_features import r13_feature_extractor as s2
|
| 18 |
+
|
| 19 |
+
BASE_STATS_DIM, SIDE_DIM, STATS_DIM = 12, 28, 41
|
| 20 |
+
INPUTS = ("name_bytes", "name_len", "ext", "size", "depth", "words", "stats")
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def encode_torrent_features(row, max_name_len=256, max_files=64):
|
| 24 |
+
encoded = s2.encode_torrent_features(row, max_name_len, max_files)
|
| 25 |
+
title = encoded["title_name_bytes"]
|
| 26 |
+
# S2's enriched representation starts with the supplied title. Empty titles
|
| 27 |
+
# have a stable space token rather than consuming any file-derived byte.
|
| 28 |
+
if not str(
|
| 29 |
+
s2.inference_input(row).get("name")
|
| 30 |
+
or s2.inference_input(row).get("title")
|
| 31 |
+
or ""
|
| 32 |
+
):
|
| 33 |
+
encoded["name_bytes"] = [32]
|
| 34 |
+
if encoded["name_bytes"][: len(title)] != title:
|
| 35 |
+
raise ValueError("Enriched input no longer preserves the raw title prefix")
|
| 36 |
+
encoded["stats"] = list(encoded["stats"]) + [float(len(title))]
|
| 37 |
+
return encoded
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def build_onnx_batch_inputs(rows, max_name_len=256, max_files=64, stats_dim=STATS_DIM):
|
| 41 |
+
if isinstance(stats_dim, bool) or not isinstance(stats_dim, (int, np.integer)) or stats_dim != STATS_DIM:
|
| 42 |
+
raise ValueError("R13 requires exactly 41 stats components; padding or truncation changes its feature ABI")
|
| 43 |
+
if not rows:
|
| 44 |
+
raise ValueError("Cannot collate an empty batch")
|
| 45 |
+
enc = [encode_torrent_features(r, max_name_len, max_files) for r in rows]
|
| 46 |
+
b, length, files = (
|
| 47 |
+
len(enc),
|
| 48 |
+
max(len(e["name_bytes"]) for e in enc),
|
| 49 |
+
max(1, max(len(e["ext"]) for e in enc)),
|
| 50 |
+
)
|
| 51 |
+
out = {
|
| 52 |
+
"name_bytes": np.zeros((b, length), np.int64),
|
| 53 |
+
"name_len": np.zeros(b, np.int64),
|
| 54 |
+
"ext": np.zeros((b, files), np.int32),
|
| 55 |
+
"size": np.zeros((b, files), np.int64),
|
| 56 |
+
"depth": np.zeros((b, files), np.int64),
|
| 57 |
+
"words": np.zeros((b, files, 4), np.int32),
|
| 58 |
+
"stats": np.zeros((b, stats_dim), np.float32),
|
| 59 |
+
}
|
| 60 |
+
for i, e in enumerate(enc):
|
| 61 |
+
n, f = len(e["name_bytes"]), len(e["ext"])
|
| 62 |
+
out["name_bytes"][i, :n], out["name_len"][i] = e["name_bytes"], n
|
| 63 |
+
for k in ("ext", "size", "depth", "words"):
|
| 64 |
+
out[k][i, :f] = e[k]
|
| 65 |
+
out["stats"][i] = e["stats"]
|
| 66 |
+
return out
|