Release v0.2.0
Browse files- README.md +28 -19
- category_classifier.onnx +2 -2
- labels.json +1 -1
- manifest.json +19 -15
- metrics.json +229 -61
README.md
CHANGED
|
@@ -15,25 +15,31 @@ datasets:
|
|
| 15 |
- Recommenders/MIND
|
| 16 |
---
|
| 17 |
|
| 18 |
-
# FirstRead models v0.
|
| 19 |
|
| 20 |
Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-time news recommender that ranks brand-new articles before they have any clicks.
|
| 21 |
|
| 22 |
-
- **Hub:** [Unmesh30/firstread-models](https://huggingface.co/Unmesh30/firstread-models). This release is the git tag [`v0.1.0`](https://huggingface.co/Unmesh30/firstread-models/tree/v0.1.0).
|
| 23 |
- **Pin a tag, not `main`.** Every file's SHA256 is listed in `manifest.json`.
|
| 24 |
|
| 25 |
> **License: these weights are MIND-derived and non-commercial.** They were trained on the Microsoft News Dataset (MIND) and are released under its [Microsoft Research License Terms](https://github.com/msnews/MIND/blob/master/MSR%20License_Data.pdf), **not** under the MIT license of the FirstRead code. Use them for non-commercial research and demonstration only.
|
| 26 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
## What's in this release
|
| 28 |
|
| 29 |
| File | What it is |
|
| 30 |
|---|---|
|
| 31 |
-
| `user_encoder.onnx` | NRMS-lite user encoder. Turns a reader's last 50 clicked-article embeddings into a user vector. |
|
| 32 |
-
| `category_classifier.onnx` | Logistic regression that maps an article embedding to one of
|
| 33 |
-
| `labels.json` |
|
| 34 |
| `population.npy` | The 384-d user vector for a reader with no history: the normalized mean embedding of all training clicks. It is also baked into `user_encoder.onnx`. |
|
| 35 |
| `config.yaml` | The training config of the shipped model. |
|
| 36 |
-
| `metrics.json` | Dev results for all three seeds, bootstrap CIs, classifier metrics, parity, the real-data eval gate and latency. |
|
| 37 |
| `manifest.json` | SHA256 and size of every file, plus the git commit, embedding-model pins, feature constants and library versions. |
|
| 38 |
|
| 39 |
**What is not in this release:** MIND rows, article text, impression logs, user ids or embedding caches. Only learned weights and aggregate numbers are shipped. A build fails if its folder holds any file outside the list above.
|
|
@@ -58,10 +64,12 @@ Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-t
|
|
| 58 |
- `BAAI/bge-small-en-v1.5`, the int8 ONNX export `Qdrant/bge-small-en-v1.5-onnx-Q` at a pinned revision, run through fastembed 0.9.0.
|
| 59 |
- Text is `title + ". " + abstract`, cleaned (text-prep version 1). Vectors are L2-normalized and 384-d.
|
| 60 |
- **Embed with exactly these pins** (see `manifest.json` → `embedding`). Vectors from any other embedding model are meaningless to this encoder.
|
| 61 |
-
- **
|
| 62 |
- Input `embedding`: float32 `[batch, 384]`.
|
| 63 |
-
- Outputs: `label` (int64 index into `labels.json`) and `probabilities` (float32 `[batch,
|
| 64 |
-
- Multinomial logistic regression, C =
|
|
|
|
|
|
|
| 65 |
- **Export:** PyTorch 2.14 dynamo exporter, opset 18; skl2onnx with zipmap off. Only the batch dimension is dynamic.
|
| 66 |
- **Parity (ONNX Runtime vs PyTorch/sklearn):**
|
| 67 |
- Max abs difference ≤ 1e-4, checked on empty, partial and full histories, on real MIND histories and on mixed batches.
|
|
@@ -73,7 +81,7 @@ Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-t
|
|
| 73 |
## Intended use
|
| 74 |
|
| 75 |
- **Use it for:** ranking English news articles for a reader, given that reader's recent clicks, in the non-commercial FirstRead demo. In particular, scoring **brand-new articles with no clicks**, from their text alone.
|
| 76 |
-
- **Use the classifier for:** a
|
| 77 |
- **Out of scope:**
|
| 78 |
- commercial use (the license forbids it);
|
| 79 |
- non-English text;
|
|
@@ -120,7 +128,7 @@ Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-t
|
|
| 120 |
**Real-data release gate:**
|
| 121 |
- Before release, the exported ONNX file is scored on a fixed 5,000-impression MIND-small dev subset.
|
| 122 |
- It must beat mean_history and must not regress more than 1% against the committed reference.
|
| 123 |
-
- v0.1.0: nDCG@10 0.4217 vs 0.3946 overall, and 0.4600 vs 0.4329 on cold-start candidates.
|
| 124 |
|
| 125 |
### Where the model fails
|
| 126 |
|
|
@@ -149,15 +157,16 @@ Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-t
|
|
| 149 |
- **Popularity and exposure bias.** Clicks are only observed on what MSN chose to show.
|
| 150 |
- **Text-only.** The model knows an article only by its headline and summary, through a frozen general-purpose embedding. It can't see images, the publisher's reputation or the full text.
|
| 151 |
- **English only.**
|
| 152 |
-
- **The classifier
|
| 153 |
-
- On dev-only articles: accuracy 0.
|
| 154 |
-
- The weakest
|
| 155 |
-
-
|
| 156 |
-
- Report: [`ml/reports/classifier-mind-large.md`](https://github.com/Unmesh30/firstread/blob/main/ml/reports/classifier-mind-large.md).
|
|
|
|
| 157 |
|
| 158 |
## Latency (ONNX Runtime, CPU)
|
| 159 |
|
| 160 |
-
User encoder, full 50-click histories, after 50 warm-up runs, 1,000 timed runs per row. **Measured on the owner's Mac (Apple Silicon), not on Cloud Run
|
| 161 |
|
| 162 |
Apple M4 (10 cores), macOS 26.3, ONNX Runtime 1.31.0:
|
| 163 |
|
|
@@ -182,7 +191,7 @@ These weights may not be used commercially. Before any commercial launch, FirstR
|
|
| 182 |
## Verifying a download
|
| 183 |
|
| 184 |
```bash
|
| 185 |
-
hf download Unmesh30/firstread-models --revision v0.
|
| 186 |
cd firstread-models && python -c "
|
| 187 |
import hashlib, json; m = json.load(open('manifest.json'))
|
| 188 |
for n, f in m['files'].items():
|
|
@@ -190,7 +199,7 @@ for n, f in m['files'].items():
|
|
| 190 |
print('all', len(m['files']), 'files match')"
|
| 191 |
```
|
| 192 |
|
| 193 |
-
Or, from the FirstRead repo: `make release-verify VERSION=0.
|
| 194 |
|
| 195 |
## Citation
|
| 196 |
|
|
|
|
| 15 |
- Recommenders/MIND
|
| 16 |
---
|
| 17 |
|
| 18 |
+
# FirstRead models v0.2.0
|
| 19 |
|
| 20 |
Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-time news recommender that ranks brand-new articles before they have any clicks.
|
| 21 |
|
| 22 |
+
- **Hub:** [Unmesh30/firstread-models](https://huggingface.co/Unmesh30/firstread-models). This release is the git tag [`v0.2.0`](https://huggingface.co/Unmesh30/firstread-models/tree/v0.2.0). The previous release is [`v0.1.0`](https://huggingface.co/Unmesh30/firstread-models/tree/v0.1.0).
|
| 23 |
- **Pin a tag, not `main`.** Every file's SHA256 is listed in `manifest.json`.
|
| 24 |
|
| 25 |
> **License: these weights are MIND-derived and non-commercial.** They were trained on the Microsoft News Dataset (MIND) and are released under its [Microsoft Research License Terms](https://github.com/msnews/MIND/blob/master/MSR%20License_Data.pdf), **not** under the MIT license of the FirstRead code. Use them for non-commercial research and demonstration only.
|
| 26 |
|
| 27 |
+
## What changed in v0.2.0
|
| 28 |
+
|
| 29 |
+
- **The classifier now predicts 13 app topics** (us, world, politics, business, tech_science, health, sports, entertainment, lifestyle, food_drink, travel, autos, weather) instead of MIND's 15 categories. Labels come from MIND (category, subcategory) pairs through one reviewed table, [`ml/configs/topic_map.yaml`](https://github.com/Unmesh30/firstread/blob/main/ml/configs/topic_map.yaml) ([ADR 0006](https://github.com/Unmesh30/firstread/blob/main/docs/adr/0006-topic-taxonomy-and-classifier.md)). MIND's categories have no tech, politics or world label, and "news" covered about a third of articles.
|
| 30 |
+
- **It is trained for headlines without summaries.** Each training article contributes its title+abstract embedding and its title-only embedding, because live GDELT articles have no summary.
|
| 31 |
+
- **The user encoder is unchanged.** `user_encoder.onnx`, `population.npy` and `config.yaml` are byte-for-byte copies of v0.1.0 (same SHA256), verified against the v0.1.0 manifest at build time. `manifest.json` → `user_encoder.inherited_from` says so. All user-encoder numbers below are from v0.1.0 and still apply.
|
| 32 |
+
|
| 33 |
## What's in this release
|
| 34 |
|
| 35 |
| File | What it is |
|
| 36 |
|---|---|
|
| 37 |
+
| `user_encoder.onnx` | NRMS-lite user encoder. Turns a reader's last 50 clicked-article embeddings into a user vector. Same file as v0.1.0. |
|
| 38 |
+
| `category_classifier.onnx` | Logistic regression that maps an article embedding to one of 13 app topics. |
|
| 39 |
+
| `labels.json` | Topic ids, in the classifier's output order. |
|
| 40 |
| `population.npy` | The 384-d user vector for a reader with no history: the normalized mean embedding of all training clicks. It is also baked into `user_encoder.onnx`. |
|
| 41 |
| `config.yaml` | The training config of the shipped model. |
|
| 42 |
+
| `metrics.json` | Dev results for all three seeds, bootstrap CIs, topic-classifier metrics (both text modes), parity, the real-data eval gate and latency. |
|
| 43 |
| `manifest.json` | SHA256 and size of every file, plus the git commit, embedding-model pins, feature constants and library versions. |
|
| 44 |
|
| 45 |
**What is not in this release:** MIND rows, article text, impression logs, user ids or embedding caches. Only learned weights and aggregate numbers are shipped. A build fails if its folder holds any file outside the list above.
|
|
|
|
| 64 |
- `BAAI/bge-small-en-v1.5`, the int8 ONNX export `Qdrant/bge-small-en-v1.5-onnx-Q` at a pinned revision, run through fastembed 0.9.0.
|
| 65 |
- Text is `title + ". " + abstract`, cleaned (text-prep version 1). Vectors are L2-normalized and 384-d.
|
| 66 |
- **Embed with exactly these pins** (see `manifest.json` → `embedding`). Vectors from any other embedding model are meaningless to this encoder.
|
| 67 |
+
- **Topic classifier:**
|
| 68 |
- Input `embedding`: float32 `[batch, 384]`.
|
| 69 |
+
- Outputs: `label` (int64 index into `labels.json`) and `probabilities` (float32 `[batch, 13]`).
|
| 70 |
+
- Multinomial logistic regression, C = 1.0, trained on 97,651 MIND-large train articles × 2 text modes. C and class weighting were chosen by mean validation macro-F1 over both modes.
|
| 71 |
+
- `manifest.json` → `category_classifier.low_confidence_threshold` = 0.50: below it, FirstRead lets a feed's topic hint break the tie if the hint is in the model's top 3. It is the lowest validation confidence from which every 0.05-wide bin is at least 50% accurate.
|
| 72 |
+
- **Topic meaning:** `us` and `politics` are US domestic news (MIND is US news), `world` is news about places outside the US, the other ten are global subjects. FirstRead applies a geo rule on top: an article from a non-US source labelled us or politics becomes world.
|
| 73 |
- **Export:** PyTorch 2.14 dynamo exporter, opset 18; skl2onnx with zipmap off. Only the batch dimension is dynamic.
|
| 74 |
- **Parity (ONNX Runtime vs PyTorch/sklearn):**
|
| 75 |
- Max abs difference ≤ 1e-4, checked on empty, partial and full histories, on real MIND histories and on mixed batches.
|
|
|
|
| 81 |
## Intended use
|
| 82 |
|
| 83 |
- **Use it for:** ranking English news articles for a reader, given that reader's recent clicks, in the non-commercial FirstRead demo. In particular, scoring **brand-new articles with no clicks**, from their text alone.
|
| 84 |
+
- **Use the classifier for:** a topic label for live English headlines during ingestion, with or without a summary.
|
| 85 |
- **Out of scope:**
|
| 86 |
- commercial use (the license forbids it);
|
| 87 |
- non-English text;
|
|
|
|
| 128 |
**Real-data release gate:**
|
| 129 |
- Before release, the exported ONNX file is scored on a fixed 5,000-impression MIND-small dev subset.
|
| 130 |
- It must beat mean_history and must not regress more than 1% against the committed reference.
|
| 131 |
+
- v0.1.0 and v0.2.0 (same encoder): nDCG@10 0.4217 vs 0.3946 overall, and 0.4600 vs 0.4329 on cold-start candidates.
|
| 132 |
|
| 133 |
### Where the model fails
|
| 134 |
|
|
|
|
| 157 |
- **Popularity and exposure bias.** Clicks are only observed on what MSN chose to show.
|
| 158 |
- **Text-only.** The model knows an article only by its headline and summary, through a frozen general-purpose embedding. It can't see images, the publisher's reputation or the full text.
|
| 159 |
- **English only.**
|
| 160 |
+
- **The topic classifier learned US news from 2019.**
|
| 161 |
+
- On 2,457 dev-only articles: accuracy 0.7346, macro-F1 0.6371 with title + abstract; accuracy 0.7184, macro-F1 0.6244 from titles only. Majority-class accuracy is 0.3268.
|
| 162 |
+
- The weakest topics from titles only are world (F1 0.44; recall 0.35), travel (0.45) and autos (0.53). us/politics/world boundaries in MIND are noisy.
|
| 163 |
+
- World has only 2,531 training articles. Live news from outside the US looks different from MIND's, so FirstRead audits the classifier on live articles separately (learn chapter 02).
|
| 164 |
+
- Report: [`ml/reports/topic-classifier-mind-large.md`](https://github.com/Unmesh30/firstread/blob/main/ml/reports/topic-classifier-mind-large.md).
|
| 165 |
+
- **Headlines without summaries cost little.** On the real-gate subset, scoring with titles only (history and candidates) lowers the user encoder's nDCG@10 by 2.1% relative (0.4217 → 0.4130); it still beats mean_history (0.3748). Cold users lose more. Report: [`ml/reports/skew-title-only-mind-small.md`](https://github.com/Unmesh30/firstread/blob/main/ml/reports/skew-title-only-mind-small.md).
|
| 166 |
|
| 167 |
## Latency (ONNX Runtime, CPU)
|
| 168 |
|
| 169 |
+
User encoder, full 50-click histories, after 50 warm-up runs, 1,000 timed runs per row. **Measured on the owner's Mac (Apple Silicon) for v0.1.0, not on Cloud Run;** v0.2.0 ships the same file, and each build's own run is in `metrics.json`. P3 re-measures inside the API container.
|
| 170 |
|
| 171 |
Apple M4 (10 cores), macOS 26.3, ONNX Runtime 1.31.0:
|
| 172 |
|
|
|
|
| 191 |
## Verifying a download
|
| 192 |
|
| 193 |
```bash
|
| 194 |
+
hf download Unmesh30/firstread-models --revision v0.2.0 --local-dir firstread-models
|
| 195 |
cd firstread-models && python -c "
|
| 196 |
import hashlib, json; m = json.load(open('manifest.json'))
|
| 197 |
for n, f in m['files'].items():
|
|
|
|
| 199 |
print('all', len(m['files']), 'files match')"
|
| 200 |
```
|
| 201 |
|
| 202 |
+
Or, from the FirstRead repo: `make release-verify VERSION=0.2.0`.
|
| 203 |
|
| 204 |
## Citation
|
| 205 |
|
category_classifier.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6436f63f2995191bb39bb441419578bf8ab533ad5f92175579aa7f4842299153
|
| 3 |
+
size 25598
|
labels.json
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
["
|
|
|
|
| 1 |
+
["us", "world", "politics", "business", "tech_science", "health", "sports", "entertainment", "lifestyle", "food_drink", "travel", "autos", "weather"]
|
manifest.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
{
|
| 2 |
-
"built_at": "2026-10-
|
| 3 |
"category_classifier": {
|
| 4 |
"file": "category_classifier.onnx",
|
| 5 |
"input": {
|
|
@@ -8,17 +8,20 @@
|
|
| 8 |
384
|
| 9 |
]
|
| 10 |
},
|
|
|
|
| 11 |
"labels": "labels.json",
|
| 12 |
-
"
|
|
|
|
| 13 |
"outputs": {
|
| 14 |
"label": [
|
| 15 |
"batch"
|
| 16 |
],
|
| 17 |
"probabilities": [
|
| 18 |
"batch",
|
| 19 |
-
|
| 20 |
]
|
| 21 |
-
}
|
|
|
|
| 22 |
},
|
| 23 |
"dataset": {
|
| 24 |
"license": "MSR License Terms",
|
|
@@ -42,24 +45,24 @@
|
|
| 42 |
},
|
| 43 |
"files": {
|
| 44 |
"README.md": {
|
| 45 |
-
"bytes":
|
| 46 |
-
"sha256": "
|
| 47 |
},
|
| 48 |
"category_classifier.onnx": {
|
| 49 |
-
"bytes":
|
| 50 |
-
"sha256": "
|
| 51 |
},
|
| 52 |
"config.yaml": {
|
| 53 |
"bytes": 236,
|
| 54 |
"sha256": "bfa55a860da3fa69035d626355f9f195b7737aeac1832044cc1c585673e4e7ad"
|
| 55 |
},
|
| 56 |
"labels.json": {
|
| 57 |
-
"bytes":
|
| 58 |
-
"sha256": "
|
| 59 |
},
|
| 60 |
"metrics.json": {
|
| 61 |
-
"bytes":
|
| 62 |
-
"sha256": "
|
| 63 |
},
|
| 64 |
"population.npy": {
|
| 65 |
"bytes": 1664,
|
|
@@ -70,15 +73,16 @@
|
|
| 70 |
"sha256": "adbfb203897441188d81d72c8cde71b664d921e52f17ce482c60c2dd807a30bf"
|
| 71 |
}
|
| 72 |
},
|
| 73 |
-
"git_commit": "
|
| 74 |
"hf_repo": "Unmesh30/firstread-models",
|
| 75 |
"license": "other (MIND-derived weights, non-commercial; see README.md)",
|
| 76 |
"name": "firstread-models",
|
| 77 |
-
"tag": "v0.
|
| 78 |
"user_encoder": {
|
| 79 |
"alpha": 0.020342299714684486,
|
| 80 |
"checkpoint_id": "49bf8339c2250819",
|
| 81 |
"file": "user_encoder.onnx",
|
|
|
|
| 82 |
"input_dtypes": {
|
| 83 |
"hist": "float32",
|
| 84 |
"mask": "bool"
|
|
@@ -105,7 +109,7 @@
|
|
| 105 |
"seed": 0,
|
| 106 |
"trained_on": "MIND-large train (all days)"
|
| 107 |
},
|
| 108 |
-
"version": "0.
|
| 109 |
"versions": {
|
| 110 |
"firstread_core": "0.1.0",
|
| 111 |
"onnx": "1.23.2",
|
|
|
|
| 1 |
{
|
| 2 |
+
"built_at": "2026-10-14T18:38:24Z",
|
| 3 |
"category_classifier": {
|
| 4 |
"file": "category_classifier.onnx",
|
| 5 |
"input": {
|
|
|
|
| 8 |
384
|
| 9 |
]
|
| 10 |
},
|
| 11 |
+
"kind": "topics",
|
| 12 |
"labels": "labels.json",
|
| 13 |
+
"low_confidence_threshold": 0.5,
|
| 14 |
+
"n_labels": 13,
|
| 15 |
"outputs": {
|
| 16 |
"label": [
|
| 17 |
"batch"
|
| 18 |
],
|
| 19 |
"probabilities": [
|
| 20 |
"batch",
|
| 21 |
+
13
|
| 22 |
]
|
| 23 |
+
},
|
| 24 |
+
"topic_map_sha256": "ed1b13889e9ed2f6964b98af94fa4b5c3a32cb224cbbadaa4c7be387e2f73220"
|
| 25 |
},
|
| 26 |
"dataset": {
|
| 27 |
"license": "MSR License Terms",
|
|
|
|
| 45 |
},
|
| 46 |
"files": {
|
| 47 |
"README.md": {
|
| 48 |
+
"bytes": 14583,
|
| 49 |
+
"sha256": "3b313b0e5f34d316e92c2b9f828560399a351806ca825a4fa446d2baf04e3807"
|
| 50 |
},
|
| 51 |
"category_classifier.onnx": {
|
| 52 |
+
"bytes": 25598,
|
| 53 |
+
"sha256": "6436f63f2995191bb39bb441419578bf8ab533ad5f92175579aa7f4842299153"
|
| 54 |
},
|
| 55 |
"config.yaml": {
|
| 56 |
"bytes": 236,
|
| 57 |
"sha256": "bfa55a860da3fa69035d626355f9f195b7737aeac1832044cc1c585673e4e7ad"
|
| 58 |
},
|
| 59 |
"labels.json": {
|
| 60 |
+
"bytes": 150,
|
| 61 |
+
"sha256": "227df36a45cd9f8176f82879a7b705dbfb866bb31851d59d75850e351c014928"
|
| 62 |
},
|
| 63 |
"metrics.json": {
|
| 64 |
+
"bytes": 13307,
|
| 65 |
+
"sha256": "c22e0426d6cf2bca37b252e1adaab394aa656e9e018a593aff08ba36d6a9f43a"
|
| 66 |
},
|
| 67 |
"population.npy": {
|
| 68 |
"bytes": 1664,
|
|
|
|
| 73 |
"sha256": "adbfb203897441188d81d72c8cde71b664d921e52f17ce482c60c2dd807a30bf"
|
| 74 |
}
|
| 75 |
},
|
| 76 |
+
"git_commit": "d2a294b1c53278ffcfd58dfd1b61f9568a60cfaf",
|
| 77 |
"hf_repo": "Unmesh30/firstread-models",
|
| 78 |
"license": "other (MIND-derived weights, non-commercial; see README.md)",
|
| 79 |
"name": "firstread-models",
|
| 80 |
+
"tag": "v0.2.0",
|
| 81 |
"user_encoder": {
|
| 82 |
"alpha": 0.020342299714684486,
|
| 83 |
"checkpoint_id": "49bf8339c2250819",
|
| 84 |
"file": "user_encoder.onnx",
|
| 85 |
+
"inherited_from": "v0.1.0",
|
| 86 |
"input_dtypes": {
|
| 87 |
"hist": "float32",
|
| 88 |
"mask": "bool"
|
|
|
|
| 109 |
"seed": 0,
|
| 110 |
"trained_on": "MIND-large train (all days)"
|
| 111 |
},
|
| 112 |
+
"version": "0.2.0",
|
| 113 |
"versions": {
|
| 114 |
"firstread_core": "0.1.0",
|
| 115 |
"onnx": "1.23.2",
|
metrics.json
CHANGED
|
@@ -34,33 +34,228 @@
|
|
| 34 |
}
|
| 35 |
},
|
| 36 |
"classifier": {
|
| 37 |
-
"accuracy": 0.7214176829268293,
|
| 38 |
"dataset": "MIND-large, articles only in dev news.tsv",
|
| 39 |
-
"
|
| 40 |
-
"
|
| 41 |
-
"
|
| 42 |
-
"
|
| 43 |
-
|
|
|
|
|
|
|
| 44 |
"labels": [
|
| 45 |
-
"
|
| 46 |
-
"
|
| 47 |
-
"
|
| 48 |
-
"
|
|
|
|
| 49 |
"health",
|
| 50 |
-
"kids",
|
| 51 |
-
"lifestyle",
|
| 52 |
-
"movies",
|
| 53 |
-
"music",
|
| 54 |
-
"news",
|
| 55 |
"sports",
|
|
|
|
|
|
|
|
|
|
| 56 |
"travel",
|
| 57 |
-
"
|
| 58 |
-
"video",
|
| 59 |
"weather"
|
| 60 |
],
|
| 61 |
-
"
|
| 62 |
-
"
|
| 63 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
},
|
| 65 |
"dataset": "MIND-large dev",
|
| 66 |
"eval_gate_real": {
|
|
@@ -119,32 +314,32 @@
|
|
| 119 |
{
|
| 120 |
"batch": 1,
|
| 121 |
"mean_ms": 0.178,
|
| 122 |
-
"p50_ms": 0.
|
| 123 |
-
"p95_ms": 0.
|
| 124 |
"runs": 1000,
|
| 125 |
"threads": 1
|
| 126 |
},
|
| 127 |
{
|
| 128 |
"batch": 32,
|
| 129 |
-
"mean_ms": 4.
|
| 130 |
-
"p50_ms": 4.
|
| 131 |
-
"p95_ms":
|
| 132 |
"runs": 1000,
|
| 133 |
"threads": 1
|
| 134 |
},
|
| 135 |
{
|
| 136 |
"batch": 1,
|
| 137 |
-
"mean_ms": 0.
|
| 138 |
"p50_ms": 0.181,
|
| 139 |
-
"p95_ms": 0.
|
| 140 |
"runs": 1000,
|
| 141 |
"threads": 4
|
| 142 |
},
|
| 143 |
{
|
| 144 |
"batch": 32,
|
| 145 |
-
"mean_ms": 3.
|
| 146 |
-
"p50_ms": 3.
|
| 147 |
-
"p95_ms": 4.
|
| 148 |
"runs": 1000,
|
| 149 |
"threads": 4
|
| 150 |
}
|
|
@@ -153,41 +348,14 @@
|
|
| 153 |
"parity": {
|
| 154 |
"classifier": {
|
| 155 |
"label_mismatches": 0,
|
| 156 |
-
"max_abs_diff":
|
| 157 |
"n": 2048,
|
| 158 |
"ok": true,
|
| 159 |
"tolerance": 0.0001
|
| 160 |
},
|
| 161 |
-
"
|
| 162 |
-
"
|
| 163 |
-
"
|
| 164 |
-
"batch32": 1.7881393432617188e-07,
|
| 165 |
-
"len0": 0.0,
|
| 166 |
-
"len1": 3.2782554626464844e-07,
|
| 167 |
-
"len49": 1.7881393432617188e-07,
|
| 168 |
-
"len50": 1.2665987014770508e-07,
|
| 169 |
-
"len7": 1.4901161193847656e-07,
|
| 170 |
-
"mixed": 2.384185791015625e-07,
|
| 171 |
-
"real_histories": 6.258487701416016e-07
|
| 172 |
-
},
|
| 173 |
-
"ok": true,
|
| 174 |
-
"tolerance": 0.0001,
|
| 175 |
-
"worst": 6.258487701416016e-07
|
| 176 |
-
},
|
| 177 |
-
"user_encoder_synthetic": {
|
| 178 |
-
"batch_invariance": 0.0,
|
| 179 |
-
"max_abs_diff": {
|
| 180 |
-
"batch32": 1.7881393432617188e-07,
|
| 181 |
-
"len0": 0.0,
|
| 182 |
-
"len1": 3.2782554626464844e-07,
|
| 183 |
-
"len49": 1.7881393432617188e-07,
|
| 184 |
-
"len50": 1.2665987014770508e-07,
|
| 185 |
-
"len7": 1.4901161193847656e-07,
|
| 186 |
-
"mixed": 2.384185791015625e-07
|
| 187 |
-
},
|
| 188 |
-
"ok": true,
|
| 189 |
-
"tolerance": 0.0001,
|
| 190 |
-
"worst": 3.2782554626464844e-07
|
| 191 |
}
|
| 192 |
},
|
| 193 |
"results": {
|
|
|
|
| 34 |
}
|
| 35 |
},
|
| 36 |
"classifier": {
|
|
|
|
| 37 |
"dataset": "MIND-large, articles only in dev news.tsv",
|
| 38 |
+
"gate": {
|
| 39 |
+
"failures": [],
|
| 40 |
+
"max_macro_f1_gap": 0.1,
|
| 41 |
+
"min_topic_f1_title_only": 0.3,
|
| 42 |
+
"passed": true
|
| 43 |
+
},
|
| 44 |
+
"kind": "topics",
|
| 45 |
"labels": [
|
| 46 |
+
"us",
|
| 47 |
+
"world",
|
| 48 |
+
"politics",
|
| 49 |
+
"business",
|
| 50 |
+
"tech_science",
|
| 51 |
"health",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
"sports",
|
| 53 |
+
"entertainment",
|
| 54 |
+
"lifestyle",
|
| 55 |
+
"food_drink",
|
| 56 |
"travel",
|
| 57 |
+
"autos",
|
|
|
|
| 58 |
"weather"
|
| 59 |
],
|
| 60 |
+
"n_test": 2457,
|
| 61 |
+
"threshold": 0.5,
|
| 62 |
+
"title_abstract": {
|
| 63 |
+
"accuracy": 0.7346357346357346,
|
| 64 |
+
"macro_f1": 0.6371489968601759,
|
| 65 |
+
"majority_class_accuracy": 0.3268213268213268,
|
| 66 |
+
"per_class": [
|
| 67 |
+
{
|
| 68 |
+
"f1": 0.6606170598911071,
|
| 69 |
+
"label": "us",
|
| 70 |
+
"precision": 0.6201022146507666,
|
| 71 |
+
"recall": 0.7067961165048544,
|
| 72 |
+
"support": 515
|
| 73 |
+
},
|
| 74 |
+
{
|
| 75 |
+
"f1": 0.45977011494252873,
|
| 76 |
+
"label": "world",
|
| 77 |
+
"precision": 0.6666666666666666,
|
| 78 |
+
"recall": 0.3508771929824561,
|
| 79 |
+
"support": 57
|
| 80 |
+
},
|
| 81 |
+
{
|
| 82 |
+
"f1": 0.815,
|
| 83 |
+
"label": "politics",
|
| 84 |
+
"precision": 0.7688679245283019,
|
| 85 |
+
"recall": 0.8670212765957447,
|
| 86 |
+
"support": 188
|
| 87 |
+
},
|
| 88 |
+
{
|
| 89 |
+
"f1": 0.5623003194888179,
|
| 90 |
+
"label": "business",
|
| 91 |
+
"precision": 0.5398773006134969,
|
| 92 |
+
"recall": 0.5866666666666667,
|
| 93 |
+
"support": 150
|
| 94 |
+
},
|
| 95 |
+
{
|
| 96 |
+
"f1": 0.6511627906976745,
|
| 97 |
+
"label": "tech_science",
|
| 98 |
+
"precision": 0.6829268292682927,
|
| 99 |
+
"recall": 0.6222222222222222,
|
| 100 |
+
"support": 90
|
| 101 |
+
},
|
| 102 |
+
{
|
| 103 |
+
"f1": 0.628099173553719,
|
| 104 |
+
"label": "health",
|
| 105 |
+
"precision": 0.6031746031746031,
|
| 106 |
+
"recall": 0.6551724137931034,
|
| 107 |
+
"support": 58
|
| 108 |
+
},
|
| 109 |
+
{
|
| 110 |
+
"f1": 0.9357682619647355,
|
| 111 |
+
"label": "sports",
|
| 112 |
+
"precision": 0.9464968152866242,
|
| 113 |
+
"recall": 0.925280199252802,
|
| 114 |
+
"support": 803
|
| 115 |
+
},
|
| 116 |
+
{
|
| 117 |
+
"f1": 0.6628571428571428,
|
| 118 |
+
"label": "entertainment",
|
| 119 |
+
"precision": 0.7160493827160493,
|
| 120 |
+
"recall": 0.6170212765957447,
|
| 121 |
+
"support": 94
|
| 122 |
+
},
|
| 123 |
+
{
|
| 124 |
+
"f1": 0.5610859728506787,
|
| 125 |
+
"label": "lifestyle",
|
| 126 |
+
"precision": 0.6262626262626263,
|
| 127 |
+
"recall": 0.5081967213114754,
|
| 128 |
+
"support": 122
|
| 129 |
+
},
|
| 130 |
+
{
|
| 131 |
+
"f1": 0.6605504587155964,
|
| 132 |
+
"label": "food_drink",
|
| 133 |
+
"precision": 0.6260869565217392,
|
| 134 |
+
"recall": 0.6990291262135923,
|
| 135 |
+
"support": 103
|
| 136 |
+
},
|
| 137 |
+
{
|
| 138 |
+
"f1": 0.4691358024691358,
|
| 139 |
+
"label": "travel",
|
| 140 |
+
"precision": 0.5229357798165137,
|
| 141 |
+
"recall": 0.4253731343283582,
|
| 142 |
+
"support": 134
|
| 143 |
+
},
|
| 144 |
+
{
|
| 145 |
+
"f1": 0.5714285714285714,
|
| 146 |
+
"label": "autos",
|
| 147 |
+
"precision": 0.5862068965517241,
|
| 148 |
+
"recall": 0.5573770491803278,
|
| 149 |
+
"support": 61
|
| 150 |
+
},
|
| 151 |
+
{
|
| 152 |
+
"f1": 0.6451612903225806,
|
| 153 |
+
"label": "weather",
|
| 154 |
+
"precision": 0.684931506849315,
|
| 155 |
+
"recall": 0.6097560975609756,
|
| 156 |
+
"support": 82
|
| 157 |
+
}
|
| 158 |
+
]
|
| 159 |
+
},
|
| 160 |
+
"title_only": {
|
| 161 |
+
"accuracy": 0.7183557183557183,
|
| 162 |
+
"macro_f1": 0.624372144861339,
|
| 163 |
+
"majority_class_accuracy": 0.3268213268213268,
|
| 164 |
+
"per_class": [
|
| 165 |
+
{
|
| 166 |
+
"f1": 0.6268939393939394,
|
| 167 |
+
"label": "us",
|
| 168 |
+
"precision": 0.6118299445471349,
|
| 169 |
+
"recall": 0.6427184466019418,
|
| 170 |
+
"support": 515
|
| 171 |
+
},
|
| 172 |
+
{
|
| 173 |
+
"f1": 0.43956043956043955,
|
| 174 |
+
"label": "world",
|
| 175 |
+
"precision": 0.5882352941176471,
|
| 176 |
+
"recall": 0.3508771929824561,
|
| 177 |
+
"support": 57
|
| 178 |
+
},
|
| 179 |
+
{
|
| 180 |
+
"f1": 0.8136482939632546,
|
| 181 |
+
"label": "politics",
|
| 182 |
+
"precision": 0.8031088082901554,
|
| 183 |
+
"recall": 0.824468085106383,
|
| 184 |
+
"support": 188
|
| 185 |
+
},
|
| 186 |
+
{
|
| 187 |
+
"f1": 0.5625,
|
| 188 |
+
"label": "business",
|
| 189 |
+
"precision": 0.5294117647058824,
|
| 190 |
+
"recall": 0.6,
|
| 191 |
+
"support": 150
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"f1": 0.6545454545454545,
|
| 195 |
+
"label": "tech_science",
|
| 196 |
+
"precision": 0.72,
|
| 197 |
+
"recall": 0.6,
|
| 198 |
+
"support": 90
|
| 199 |
+
},
|
| 200 |
+
{
|
| 201 |
+
"f1": 0.5833333333333334,
|
| 202 |
+
"label": "health",
|
| 203 |
+
"precision": 0.5645161290322581,
|
| 204 |
+
"recall": 0.603448275862069,
|
| 205 |
+
"support": 58
|
| 206 |
+
},
|
| 207 |
+
{
|
| 208 |
+
"f1": 0.9229805886036319,
|
| 209 |
+
"label": "sports",
|
| 210 |
+
"precision": 0.9282115869017632,
|
| 211 |
+
"recall": 0.9178082191780822,
|
| 212 |
+
"support": 803
|
| 213 |
+
},
|
| 214 |
+
{
|
| 215 |
+
"f1": 0.6413043478260869,
|
| 216 |
+
"label": "entertainment",
|
| 217 |
+
"precision": 0.6555555555555556,
|
| 218 |
+
"recall": 0.6276595744680851,
|
| 219 |
+
"support": 94
|
| 220 |
+
},
|
| 221 |
+
{
|
| 222 |
+
"f1": 0.575,
|
| 223 |
+
"label": "lifestyle",
|
| 224 |
+
"precision": 0.5847457627118644,
|
| 225 |
+
"recall": 0.5655737704918032,
|
| 226 |
+
"support": 122
|
| 227 |
+
},
|
| 228 |
+
{
|
| 229 |
+
"f1": 0.6278026905829597,
|
| 230 |
+
"label": "food_drink",
|
| 231 |
+
"precision": 0.5833333333333334,
|
| 232 |
+
"recall": 0.6796116504854369,
|
| 233 |
+
"support": 103
|
| 234 |
+
},
|
| 235 |
+
{
|
| 236 |
+
"f1": 0.44881889763779526,
|
| 237 |
+
"label": "travel",
|
| 238 |
+
"precision": 0.475,
|
| 239 |
+
"recall": 0.4253731343283582,
|
| 240 |
+
"support": 134
|
| 241 |
+
},
|
| 242 |
+
{
|
| 243 |
+
"f1": 0.5333333333333333,
|
| 244 |
+
"label": "autos",
|
| 245 |
+
"precision": 0.5423728813559322,
|
| 246 |
+
"recall": 0.5245901639344263,
|
| 247 |
+
"support": 61
|
| 248 |
+
},
|
| 249 |
+
{
|
| 250 |
+
"f1": 0.6871165644171779,
|
| 251 |
+
"label": "weather",
|
| 252 |
+
"precision": 0.691358024691358,
|
| 253 |
+
"recall": 0.6829268292682927,
|
| 254 |
+
"support": 82
|
| 255 |
+
}
|
| 256 |
+
]
|
| 257 |
+
},
|
| 258 |
+
"topic_map_sha256": "ed1b13889e9ed2f6964b98af94fa4b5c3a32cb224cbbadaa4c7be387e2f73220"
|
| 259 |
},
|
| 260 |
"dataset": "MIND-large dev",
|
| 261 |
"eval_gate_real": {
|
|
|
|
| 314 |
{
|
| 315 |
"batch": 1,
|
| 316 |
"mean_ms": 0.178,
|
| 317 |
+
"p50_ms": 0.176,
|
| 318 |
+
"p95_ms": 0.186,
|
| 319 |
"runs": 1000,
|
| 320 |
"threads": 1
|
| 321 |
},
|
| 322 |
{
|
| 323 |
"batch": 32,
|
| 324 |
+
"mean_ms": 4.715,
|
| 325 |
+
"p50_ms": 4.407,
|
| 326 |
+
"p95_ms": 5.708,
|
| 327 |
"runs": 1000,
|
| 328 |
"threads": 1
|
| 329 |
},
|
| 330 |
{
|
| 331 |
"batch": 1,
|
| 332 |
+
"mean_ms": 0.211,
|
| 333 |
"p50_ms": 0.181,
|
| 334 |
+
"p95_ms": 0.38,
|
| 335 |
"runs": 1000,
|
| 336 |
"threads": 4
|
| 337 |
},
|
| 338 |
{
|
| 339 |
"batch": 32,
|
| 340 |
+
"mean_ms": 3.671,
|
| 341 |
+
"p50_ms": 3.484,
|
| 342 |
+
"p95_ms": 4.568,
|
| 343 |
"runs": 1000,
|
| 344 |
"threads": 4
|
| 345 |
}
|
|
|
|
| 348 |
"parity": {
|
| 349 |
"classifier": {
|
| 350 |
"label_mismatches": 0,
|
| 351 |
+
"max_abs_diff": 3.5762786865234375e-07,
|
| 352 |
"n": 2048,
|
| 353 |
"ok": true,
|
| 354 |
"tolerance": 0.0001
|
| 355 |
},
|
| 356 |
+
"user_encoder": {
|
| 357 |
+
"inherited_from": "v0.1.0",
|
| 358 |
+
"note": "byte-identical copy; parity was checked when that release was built"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 359 |
}
|
| 360 |
},
|
| 361 |
"results": {
|