Unmesh30 commited on
Commit
099201f
·
verified ·
1 Parent(s): f29ec56

Release v0.2.0

Browse files
Files changed (5) hide show
  1. README.md +28 -19
  2. category_classifier.onnx +2 -2
  3. labels.json +1 -1
  4. manifest.json +19 -15
  5. metrics.json +229 -61
README.md CHANGED
@@ -15,25 +15,31 @@ datasets:
15
  - Recommenders/MIND
16
  ---
17
 
18
- # FirstRead models v0.1.0
19
 
20
  Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-time news recommender that ranks brand-new articles before they have any clicks.
21
 
22
- - **Hub:** [Unmesh30/firstread-models](https://huggingface.co/Unmesh30/firstread-models). This release is the git tag [`v0.1.0`](https://huggingface.co/Unmesh30/firstread-models/tree/v0.1.0).
23
  - **Pin a tag, not `main`.** Every file's SHA256 is listed in `manifest.json`.
24
 
25
  > **License: these weights are MIND-derived and non-commercial.** They were trained on the Microsoft News Dataset (MIND) and are released under its [Microsoft Research License Terms](https://github.com/msnews/MIND/blob/master/MSR%20License_Data.pdf), **not** under the MIT license of the FirstRead code. Use them for non-commercial research and demonstration only.
26
 
 
 
 
 
 
 
27
  ## What's in this release
28
 
29
  | File | What it is |
30
  |---|---|
31
- | `user_encoder.onnx` | NRMS-lite user encoder. Turns a reader's last 50 clicked-article embeddings into a user vector. |
32
- | `category_classifier.onnx` | Logistic regression that maps an article embedding to one of 15 MIND categories. |
33
- | `labels.json` | Category names, in the classifier's output order. |
34
  | `population.npy` | The 384-d user vector for a reader with no history: the normalized mean embedding of all training clicks. It is also baked into `user_encoder.onnx`. |
35
  | `config.yaml` | The training config of the shipped model. |
36
- | `metrics.json` | Dev results for all three seeds, bootstrap CIs, classifier metrics, parity, the real-data eval gate and latency. |
37
  | `manifest.json` | SHA256 and size of every file, plus the git commit, embedding-model pins, feature constants and library versions. |
38
 
39
  **What is not in this release:** MIND rows, article text, impression logs, user ids or embedding caches. Only learned weights and aggregate numbers are shipped. A build fails if its folder holds any file outside the list above.
@@ -58,10 +64,12 @@ Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-t
58
  - `BAAI/bge-small-en-v1.5`, the int8 ONNX export `Qdrant/bge-small-en-v1.5-onnx-Q` at a pinned revision, run through fastembed 0.9.0.
59
  - Text is `title + ". " + abstract`, cleaned (text-prep version 1). Vectors are L2-normalized and 384-d.
60
  - **Embed with exactly these pins** (see `manifest.json` → `embedding`). Vectors from any other embedding model are meaningless to this encoder.
61
- - **Category classifier:**
62
  - Input `embedding`: float32 `[batch, 384]`.
63
- - Outputs: `label` (int64 index into `labels.json`) and `probabilities` (float32 `[batch, 15]`).
64
- - Multinomial logistic regression, C = 30.
 
 
65
  - **Export:** PyTorch 2.14 dynamo exporter, opset 18; skl2onnx with zipmap off. Only the batch dimension is dynamic.
66
  - **Parity (ONNX Runtime vs PyTorch/sklearn):**
67
  - Max abs difference ≤ 1e-4, checked on empty, partial and full histories, on real MIND histories and on mixed batches.
@@ -73,7 +81,7 @@ Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-t
73
  ## Intended use
74
 
75
  - **Use it for:** ranking English news articles for a reader, given that reader's recent clicks, in the non-commercial FirstRead demo. In particular, scoring **brand-new articles with no clicks**, from their text alone.
76
- - **Use the classifier for:** a coarse topic label for live articles during ingestion.
77
  - **Out of scope:**
78
  - commercial use (the license forbids it);
79
  - non-English text;
@@ -120,7 +128,7 @@ Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-t
120
  **Real-data release gate:**
121
  - Before release, the exported ONNX file is scored on a fixed 5,000-impression MIND-small dev subset.
122
  - It must beat mean_history and must not regress more than 1% against the committed reference.
123
- - v0.1.0: nDCG@10 0.4217 vs 0.3946 overall, and 0.4600 vs 0.4329 on cold-start candidates.
124
 
125
  ### Where the model fails
126
 
@@ -149,15 +157,16 @@ Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-t
149
  - **Popularity and exposure bias.** Clicks are only observed on what MSN chose to show.
150
  - **Text-only.** The model knows an article only by its headline and summary, through a frozen general-purpose embedding. It can't see images, the publisher's reputation or the full text.
151
  - **English only.**
152
- - **The classifier's labels are MIND's categories.**
153
- - On dev-only articles: accuracy 0.7214, macro-F1 0.5984; majority-class accuracy is 0.3255.
154
- - The weakest classes are video (F1 0.34), tv (0.40) and travel (0.44); kids has only 3 test articles.
155
- - Three rare MIND categories (games, middleeast, northamerica) are not predicted at all.
156
- - Report: [`ml/reports/classifier-mind-large.md`](https://github.com/Unmesh30/firstread/blob/main/ml/reports/classifier-mind-large.md).
 
157
 
158
  ## Latency (ONNX Runtime, CPU)
159
 
160
- User encoder, full 50-click histories, after 50 warm-up runs, 1,000 timed runs per row. **Measured on the owner's Mac (Apple Silicon), not on Cloud Run.** P3 re-measures inside the API container.
161
 
162
  Apple M4 (10 cores), macOS 26.3, ONNX Runtime 1.31.0:
163
 
@@ -182,7 +191,7 @@ These weights may not be used commercially. Before any commercial launch, FirstR
182
  ## Verifying a download
183
 
184
  ```bash
185
- hf download Unmesh30/firstread-models --revision v0.1.0 --local-dir firstread-models
186
  cd firstread-models && python -c "
187
  import hashlib, json; m = json.load(open('manifest.json'))
188
  for n, f in m['files'].items():
@@ -190,7 +199,7 @@ for n, f in m['files'].items():
190
  print('all', len(m['files']), 'files match')"
191
  ```
192
 
193
- Or, from the FirstRead repo: `make release-verify VERSION=0.1.0`.
194
 
195
  ## Citation
196
 
 
15
  - Recommenders/MIND
16
  ---
17
 
18
+ # FirstRead models v0.2.0
19
 
20
  Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-time news recommender that ranks brand-new articles before they have any clicks.
21
 
22
+ - **Hub:** [Unmesh30/firstread-models](https://huggingface.co/Unmesh30/firstread-models). This release is the git tag [`v0.2.0`](https://huggingface.co/Unmesh30/firstread-models/tree/v0.2.0). The previous release is [`v0.1.0`](https://huggingface.co/Unmesh30/firstread-models/tree/v0.1.0).
23
  - **Pin a tag, not `main`.** Every file's SHA256 is listed in `manifest.json`.
24
 
25
  > **License: these weights are MIND-derived and non-commercial.** They were trained on the Microsoft News Dataset (MIND) and are released under its [Microsoft Research License Terms](https://github.com/msnews/MIND/blob/master/MSR%20License_Data.pdf), **not** under the MIT license of the FirstRead code. Use them for non-commercial research and demonstration only.
26
 
27
+ ## What changed in v0.2.0
28
+
29
+ - **The classifier now predicts 13 app topics** (us, world, politics, business, tech_science, health, sports, entertainment, lifestyle, food_drink, travel, autos, weather) instead of MIND's 15 categories. Labels come from MIND (category, subcategory) pairs through one reviewed table, [`ml/configs/topic_map.yaml`](https://github.com/Unmesh30/firstread/blob/main/ml/configs/topic_map.yaml) ([ADR 0006](https://github.com/Unmesh30/firstread/blob/main/docs/adr/0006-topic-taxonomy-and-classifier.md)). MIND's categories have no tech, politics or world label, and "news" covered about a third of articles.
30
+ - **It is trained for headlines without summaries.** Each training article contributes its title+abstract embedding and its title-only embedding, because live GDELT articles have no summary.
31
+ - **The user encoder is unchanged.** `user_encoder.onnx`, `population.npy` and `config.yaml` are byte-for-byte copies of v0.1.0 (same SHA256), verified against the v0.1.0 manifest at build time. `manifest.json` → `user_encoder.inherited_from` says so. All user-encoder numbers below are from v0.1.0 and still apply.
32
+
33
  ## What's in this release
34
 
35
  | File | What it is |
36
  |---|---|
37
+ | `user_encoder.onnx` | NRMS-lite user encoder. Turns a reader's last 50 clicked-article embeddings into a user vector. Same file as v0.1.0. |
38
+ | `category_classifier.onnx` | Logistic regression that maps an article embedding to one of 13 app topics. |
39
+ | `labels.json` | Topic ids, in the classifier's output order. |
40
  | `population.npy` | The 384-d user vector for a reader with no history: the normalized mean embedding of all training clicks. It is also baked into `user_encoder.onnx`. |
41
  | `config.yaml` | The training config of the shipped model. |
42
+ | `metrics.json` | Dev results for all three seeds, bootstrap CIs, topic-classifier metrics (both text modes), parity, the real-data eval gate and latency. |
43
  | `manifest.json` | SHA256 and size of every file, plus the git commit, embedding-model pins, feature constants and library versions. |
44
 
45
  **What is not in this release:** MIND rows, article text, impression logs, user ids or embedding caches. Only learned weights and aggregate numbers are shipped. A build fails if its folder holds any file outside the list above.
 
64
  - `BAAI/bge-small-en-v1.5`, the int8 ONNX export `Qdrant/bge-small-en-v1.5-onnx-Q` at a pinned revision, run through fastembed 0.9.0.
65
  - Text is `title + ". " + abstract`, cleaned (text-prep version 1). Vectors are L2-normalized and 384-d.
66
  - **Embed with exactly these pins** (see `manifest.json` → `embedding`). Vectors from any other embedding model are meaningless to this encoder.
67
+ - **Topic classifier:**
68
  - Input `embedding`: float32 `[batch, 384]`.
69
+ - Outputs: `label` (int64 index into `labels.json`) and `probabilities` (float32 `[batch, 13]`).
70
+ - Multinomial logistic regression, C = 1.0, trained on 97,651 MIND-large train articles × 2 text modes. C and class weighting were chosen by mean validation macro-F1 over both modes.
71
+ - `manifest.json` → `category_classifier.low_confidence_threshold` = 0.50: below it, FirstRead lets a feed's topic hint break the tie if the hint is in the model's top 3. It is the lowest validation confidence from which every 0.05-wide bin is at least 50% accurate.
72
+ - **Topic meaning:** `us` and `politics` are US domestic news (MIND is US news), `world` is news about places outside the US, the other ten are global subjects. FirstRead applies a geo rule on top: an article from a non-US source labelled us or politics becomes world.
73
  - **Export:** PyTorch 2.14 dynamo exporter, opset 18; skl2onnx with zipmap off. Only the batch dimension is dynamic.
74
  - **Parity (ONNX Runtime vs PyTorch/sklearn):**
75
  - Max abs difference ≤ 1e-4, checked on empty, partial and full histories, on real MIND histories and on mixed batches.
 
81
  ## Intended use
82
 
83
  - **Use it for:** ranking English news articles for a reader, given that reader's recent clicks, in the non-commercial FirstRead demo. In particular, scoring **brand-new articles with no clicks**, from their text alone.
84
+ - **Use the classifier for:** a topic label for live English headlines during ingestion, with or without a summary.
85
  - **Out of scope:**
86
  - commercial use (the license forbids it);
87
  - non-English text;
 
128
  **Real-data release gate:**
129
  - Before release, the exported ONNX file is scored on a fixed 5,000-impression MIND-small dev subset.
130
  - It must beat mean_history and must not regress more than 1% against the committed reference.
131
+ - v0.1.0 and v0.2.0 (same encoder): nDCG@10 0.4217 vs 0.3946 overall, and 0.4600 vs 0.4329 on cold-start candidates.
132
 
133
  ### Where the model fails
134
 
 
157
  - **Popularity and exposure bias.** Clicks are only observed on what MSN chose to show.
158
  - **Text-only.** The model knows an article only by its headline and summary, through a frozen general-purpose embedding. It can't see images, the publisher's reputation or the full text.
159
  - **English only.**
160
+ - **The topic classifier learned US news from 2019.**
161
+ - On 2,457 dev-only articles: accuracy 0.7346, macro-F1 0.6371 with title + abstract; accuracy 0.7184, macro-F1 0.6244 from titles only. Majority-class accuracy is 0.3268.
162
+ - The weakest topics from titles only are world (F1 0.44; recall 0.35), travel (0.45) and autos (0.53). us/politics/world boundaries in MIND are noisy.
163
+ - World has only 2,531 training articles. Live news from outside the US looks different from MIND's, so FirstRead audits the classifier on live articles separately (learn chapter 02).
164
+ - Report: [`ml/reports/topic-classifier-mind-large.md`](https://github.com/Unmesh30/firstread/blob/main/ml/reports/topic-classifier-mind-large.md).
165
+ - **Headlines without summaries cost little.** On the real-gate subset, scoring with titles only (history and candidates) lowers the user encoder's nDCG@10 by 2.1% relative (0.4217 → 0.4130); it still beats mean_history (0.3748). Cold users lose more. Report: [`ml/reports/skew-title-only-mind-small.md`](https://github.com/Unmesh30/firstread/blob/main/ml/reports/skew-title-only-mind-small.md).
166
 
167
  ## Latency (ONNX Runtime, CPU)
168
 
169
+ User encoder, full 50-click histories, after 50 warm-up runs, 1,000 timed runs per row. **Measured on the owner's Mac (Apple Silicon) for v0.1.0, not on Cloud Run;** v0.2.0 ships the same file, and each build's own run is in `metrics.json`. P3 re-measures inside the API container.
170
 
171
  Apple M4 (10 cores), macOS 26.3, ONNX Runtime 1.31.0:
172
 
 
191
  ## Verifying a download
192
 
193
  ```bash
194
+ hf download Unmesh30/firstread-models --revision v0.2.0 --local-dir firstread-models
195
  cd firstread-models && python -c "
196
  import hashlib, json; m = json.load(open('manifest.json'))
197
  for n, f in m['files'].items():
 
199
  print('all', len(m['files']), 'files match')"
200
  ```
201
 
202
+ Or, from the FirstRead repo: `make release-verify VERSION=0.2.0`.
203
 
204
  ## Citation
205
 
category_classifier.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:669cafa314762ad1721e5a7780ac580a6d0ce8e59a04e570b71622d0cb060b87
3
- size 29452
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6436f63f2995191bb39bb441419578bf8ab533ad5f92175579aa7f4842299153
3
+ size 25598
labels.json CHANGED
@@ -1 +1 @@
1
- ["autos", "entertainment", "finance", "foodanddrink", "health", "kids", "lifestyle", "movies", "music", "news", "sports", "travel", "tv", "video", "weather"]
 
1
+ ["us", "world", "politics", "business", "tech_science", "health", "sports", "entertainment", "lifestyle", "food_drink", "travel", "autos", "weather"]
manifest.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
- "built_at": "2026-10-14T16:35:04Z",
3
  "category_classifier": {
4
  "file": "category_classifier.onnx",
5
  "input": {
@@ -8,17 +8,20 @@
8
  384
9
  ]
10
  },
 
11
  "labels": "labels.json",
12
- "n_labels": 15,
 
13
  "outputs": {
14
  "label": [
15
  "batch"
16
  ],
17
  "probabilities": [
18
  "batch",
19
- 15
20
  ]
21
- }
 
22
  },
23
  "dataset": {
24
  "license": "MSR License Terms",
@@ -42,24 +45,24 @@
42
  },
43
  "files": {
44
  "README.md": {
45
- "bytes": 11855,
46
- "sha256": "178ce9daf603491a609b29992baf03c0381be04524b9992f2db02e85a9cecc9b"
47
  },
48
  "category_classifier.onnx": {
49
- "bytes": 29452,
50
- "sha256": "669cafa314762ad1721e5a7780ac580a6d0ce8e59a04e570b71622d0cb060b87"
51
  },
52
  "config.yaml": {
53
  "bytes": 236,
54
  "sha256": "bfa55a860da3fa69035d626355f9f195b7737aeac1832044cc1c585673e4e7ad"
55
  },
56
  "labels.json": {
57
- "bytes": 158,
58
- "sha256": "c6647a46aceafa50ceef453794bc944de7ef94b1eb55b4dbf7efa7d5e1ff43ff"
59
  },
60
  "metrics.json": {
61
- "bytes": 9520,
62
- "sha256": "cf1cfe4e8713a9f873dbf60ae1a34237b72f83a0c01589821b79d720f1058950"
63
  },
64
  "population.npy": {
65
  "bytes": 1664,
@@ -70,15 +73,16 @@
70
  "sha256": "adbfb203897441188d81d72c8cde71b664d921e52f17ce482c60c2dd807a30bf"
71
  }
72
  },
73
- "git_commit": "5db524e0eaf58e48a5d1ef7ed4374acaf2bdc494",
74
  "hf_repo": "Unmesh30/firstread-models",
75
  "license": "other (MIND-derived weights, non-commercial; see README.md)",
76
  "name": "firstread-models",
77
- "tag": "v0.1.0",
78
  "user_encoder": {
79
  "alpha": 0.020342299714684486,
80
  "checkpoint_id": "49bf8339c2250819",
81
  "file": "user_encoder.onnx",
 
82
  "input_dtypes": {
83
  "hist": "float32",
84
  "mask": "bool"
@@ -105,7 +109,7 @@
105
  "seed": 0,
106
  "trained_on": "MIND-large train (all days)"
107
  },
108
- "version": "0.1.0",
109
  "versions": {
110
  "firstread_core": "0.1.0",
111
  "onnx": "1.23.2",
 
1
  {
2
+ "built_at": "2026-10-14T18:38:24Z",
3
  "category_classifier": {
4
  "file": "category_classifier.onnx",
5
  "input": {
 
8
  384
9
  ]
10
  },
11
+ "kind": "topics",
12
  "labels": "labels.json",
13
+ "low_confidence_threshold": 0.5,
14
+ "n_labels": 13,
15
  "outputs": {
16
  "label": [
17
  "batch"
18
  ],
19
  "probabilities": [
20
  "batch",
21
+ 13
22
  ]
23
+ },
24
+ "topic_map_sha256": "ed1b13889e9ed2f6964b98af94fa4b5c3a32cb224cbbadaa4c7be387e2f73220"
25
  },
26
  "dataset": {
27
  "license": "MSR License Terms",
 
45
  },
46
  "files": {
47
  "README.md": {
48
+ "bytes": 14583,
49
+ "sha256": "3b313b0e5f34d316e92c2b9f828560399a351806ca825a4fa446d2baf04e3807"
50
  },
51
  "category_classifier.onnx": {
52
+ "bytes": 25598,
53
+ "sha256": "6436f63f2995191bb39bb441419578bf8ab533ad5f92175579aa7f4842299153"
54
  },
55
  "config.yaml": {
56
  "bytes": 236,
57
  "sha256": "bfa55a860da3fa69035d626355f9f195b7737aeac1832044cc1c585673e4e7ad"
58
  },
59
  "labels.json": {
60
+ "bytes": 150,
61
+ "sha256": "227df36a45cd9f8176f82879a7b705dbfb866bb31851d59d75850e351c014928"
62
  },
63
  "metrics.json": {
64
+ "bytes": 13307,
65
+ "sha256": "c22e0426d6cf2bca37b252e1adaab394aa656e9e018a593aff08ba36d6a9f43a"
66
  },
67
  "population.npy": {
68
  "bytes": 1664,
 
73
  "sha256": "adbfb203897441188d81d72c8cde71b664d921e52f17ce482c60c2dd807a30bf"
74
  }
75
  },
76
+ "git_commit": "d2a294b1c53278ffcfd58dfd1b61f9568a60cfaf",
77
  "hf_repo": "Unmesh30/firstread-models",
78
  "license": "other (MIND-derived weights, non-commercial; see README.md)",
79
  "name": "firstread-models",
80
+ "tag": "v0.2.0",
81
  "user_encoder": {
82
  "alpha": 0.020342299714684486,
83
  "checkpoint_id": "49bf8339c2250819",
84
  "file": "user_encoder.onnx",
85
+ "inherited_from": "v0.1.0",
86
  "input_dtypes": {
87
  "hist": "float32",
88
  "mask": "bool"
 
109
  "seed": 0,
110
  "trained_on": "MIND-large train (all days)"
111
  },
112
+ "version": "0.2.0",
113
  "versions": {
114
  "firstread_core": "0.1.0",
115
  "onnx": "1.23.2",
metrics.json CHANGED
@@ -34,33 +34,228 @@
34
  }
35
  },
36
  "classifier": {
37
- "accuracy": 0.7214176829268293,
38
  "dataset": "MIND-large, articles only in dev news.tsv",
39
- "dropped_labels": [
40
- "games",
41
- "middleeast",
42
- "northamerica"
43
- ],
 
 
44
  "labels": [
45
- "autos",
46
- "entertainment",
47
- "finance",
48
- "foodanddrink",
 
49
  "health",
50
- "kids",
51
- "lifestyle",
52
- "movies",
53
- "music",
54
- "news",
55
  "sports",
 
 
 
56
  "travel",
57
- "tv",
58
- "video",
59
  "weather"
60
  ],
61
- "macro_f1": 0.5984492354428844,
62
- "majority_class_accuracy": 0.3254573170731707,
63
- "n_test": 2624
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
64
  },
65
  "dataset": "MIND-large dev",
66
  "eval_gate_real": {
@@ -119,32 +314,32 @@
119
  {
120
  "batch": 1,
121
  "mean_ms": 0.178,
122
- "p50_ms": 0.177,
123
- "p95_ms": 0.183,
124
  "runs": 1000,
125
  "threads": 1
126
  },
127
  {
128
  "batch": 32,
129
- "mean_ms": 4.476,
130
- "p50_ms": 4.399,
131
- "p95_ms": 4.94,
132
  "runs": 1000,
133
  "threads": 1
134
  },
135
  {
136
  "batch": 1,
137
- "mean_ms": 0.22,
138
  "p50_ms": 0.181,
139
- "p95_ms": 0.457,
140
  "runs": 1000,
141
  "threads": 4
142
  },
143
  {
144
  "batch": 32,
145
- "mean_ms": 3.727,
146
- "p50_ms": 3.417,
147
- "p95_ms": 4.417,
148
  "runs": 1000,
149
  "threads": 4
150
  }
@@ -153,41 +348,14 @@
153
  "parity": {
154
  "classifier": {
155
  "label_mismatches": 0,
156
- "max_abs_diff": 2.384185791015625e-07,
157
  "n": 2048,
158
  "ok": true,
159
  "tolerance": 0.0001
160
  },
161
- "user_encoder_real": {
162
- "batch_invariance": 0.0,
163
- "max_abs_diff": {
164
- "batch32": 1.7881393432617188e-07,
165
- "len0": 0.0,
166
- "len1": 3.2782554626464844e-07,
167
- "len49": 1.7881393432617188e-07,
168
- "len50": 1.2665987014770508e-07,
169
- "len7": 1.4901161193847656e-07,
170
- "mixed": 2.384185791015625e-07,
171
- "real_histories": 6.258487701416016e-07
172
- },
173
- "ok": true,
174
- "tolerance": 0.0001,
175
- "worst": 6.258487701416016e-07
176
- },
177
- "user_encoder_synthetic": {
178
- "batch_invariance": 0.0,
179
- "max_abs_diff": {
180
- "batch32": 1.7881393432617188e-07,
181
- "len0": 0.0,
182
- "len1": 3.2782554626464844e-07,
183
- "len49": 1.7881393432617188e-07,
184
- "len50": 1.2665987014770508e-07,
185
- "len7": 1.4901161193847656e-07,
186
- "mixed": 2.384185791015625e-07
187
- },
188
- "ok": true,
189
- "tolerance": 0.0001,
190
- "worst": 3.2782554626464844e-07
191
  }
192
  },
193
  "results": {
 
34
  }
35
  },
36
  "classifier": {
 
37
  "dataset": "MIND-large, articles only in dev news.tsv",
38
+ "gate": {
39
+ "failures": [],
40
+ "max_macro_f1_gap": 0.1,
41
+ "min_topic_f1_title_only": 0.3,
42
+ "passed": true
43
+ },
44
+ "kind": "topics",
45
  "labels": [
46
+ "us",
47
+ "world",
48
+ "politics",
49
+ "business",
50
+ "tech_science",
51
  "health",
 
 
 
 
 
52
  "sports",
53
+ "entertainment",
54
+ "lifestyle",
55
+ "food_drink",
56
  "travel",
57
+ "autos",
 
58
  "weather"
59
  ],
60
+ "n_test": 2457,
61
+ "threshold": 0.5,
62
+ "title_abstract": {
63
+ "accuracy": 0.7346357346357346,
64
+ "macro_f1": 0.6371489968601759,
65
+ "majority_class_accuracy": 0.3268213268213268,
66
+ "per_class": [
67
+ {
68
+ "f1": 0.6606170598911071,
69
+ "label": "us",
70
+ "precision": 0.6201022146507666,
71
+ "recall": 0.7067961165048544,
72
+ "support": 515
73
+ },
74
+ {
75
+ "f1": 0.45977011494252873,
76
+ "label": "world",
77
+ "precision": 0.6666666666666666,
78
+ "recall": 0.3508771929824561,
79
+ "support": 57
80
+ },
81
+ {
82
+ "f1": 0.815,
83
+ "label": "politics",
84
+ "precision": 0.7688679245283019,
85
+ "recall": 0.8670212765957447,
86
+ "support": 188
87
+ },
88
+ {
89
+ "f1": 0.5623003194888179,
90
+ "label": "business",
91
+ "precision": 0.5398773006134969,
92
+ "recall": 0.5866666666666667,
93
+ "support": 150
94
+ },
95
+ {
96
+ "f1": 0.6511627906976745,
97
+ "label": "tech_science",
98
+ "precision": 0.6829268292682927,
99
+ "recall": 0.6222222222222222,
100
+ "support": 90
101
+ },
102
+ {
103
+ "f1": 0.628099173553719,
104
+ "label": "health",
105
+ "precision": 0.6031746031746031,
106
+ "recall": 0.6551724137931034,
107
+ "support": 58
108
+ },
109
+ {
110
+ "f1": 0.9357682619647355,
111
+ "label": "sports",
112
+ "precision": 0.9464968152866242,
113
+ "recall": 0.925280199252802,
114
+ "support": 803
115
+ },
116
+ {
117
+ "f1": 0.6628571428571428,
118
+ "label": "entertainment",
119
+ "precision": 0.7160493827160493,
120
+ "recall": 0.6170212765957447,
121
+ "support": 94
122
+ },
123
+ {
124
+ "f1": 0.5610859728506787,
125
+ "label": "lifestyle",
126
+ "precision": 0.6262626262626263,
127
+ "recall": 0.5081967213114754,
128
+ "support": 122
129
+ },
130
+ {
131
+ "f1": 0.6605504587155964,
132
+ "label": "food_drink",
133
+ "precision": 0.6260869565217392,
134
+ "recall": 0.6990291262135923,
135
+ "support": 103
136
+ },
137
+ {
138
+ "f1": 0.4691358024691358,
139
+ "label": "travel",
140
+ "precision": 0.5229357798165137,
141
+ "recall": 0.4253731343283582,
142
+ "support": 134
143
+ },
144
+ {
145
+ "f1": 0.5714285714285714,
146
+ "label": "autos",
147
+ "precision": 0.5862068965517241,
148
+ "recall": 0.5573770491803278,
149
+ "support": 61
150
+ },
151
+ {
152
+ "f1": 0.6451612903225806,
153
+ "label": "weather",
154
+ "precision": 0.684931506849315,
155
+ "recall": 0.6097560975609756,
156
+ "support": 82
157
+ }
158
+ ]
159
+ },
160
+ "title_only": {
161
+ "accuracy": 0.7183557183557183,
162
+ "macro_f1": 0.624372144861339,
163
+ "majority_class_accuracy": 0.3268213268213268,
164
+ "per_class": [
165
+ {
166
+ "f1": 0.6268939393939394,
167
+ "label": "us",
168
+ "precision": 0.6118299445471349,
169
+ "recall": 0.6427184466019418,
170
+ "support": 515
171
+ },
172
+ {
173
+ "f1": 0.43956043956043955,
174
+ "label": "world",
175
+ "precision": 0.5882352941176471,
176
+ "recall": 0.3508771929824561,
177
+ "support": 57
178
+ },
179
+ {
180
+ "f1": 0.8136482939632546,
181
+ "label": "politics",
182
+ "precision": 0.8031088082901554,
183
+ "recall": 0.824468085106383,
184
+ "support": 188
185
+ },
186
+ {
187
+ "f1": 0.5625,
188
+ "label": "business",
189
+ "precision": 0.5294117647058824,
190
+ "recall": 0.6,
191
+ "support": 150
192
+ },
193
+ {
194
+ "f1": 0.6545454545454545,
195
+ "label": "tech_science",
196
+ "precision": 0.72,
197
+ "recall": 0.6,
198
+ "support": 90
199
+ },
200
+ {
201
+ "f1": 0.5833333333333334,
202
+ "label": "health",
203
+ "precision": 0.5645161290322581,
204
+ "recall": 0.603448275862069,
205
+ "support": 58
206
+ },
207
+ {
208
+ "f1": 0.9229805886036319,
209
+ "label": "sports",
210
+ "precision": 0.9282115869017632,
211
+ "recall": 0.9178082191780822,
212
+ "support": 803
213
+ },
214
+ {
215
+ "f1": 0.6413043478260869,
216
+ "label": "entertainment",
217
+ "precision": 0.6555555555555556,
218
+ "recall": 0.6276595744680851,
219
+ "support": 94
220
+ },
221
+ {
222
+ "f1": 0.575,
223
+ "label": "lifestyle",
224
+ "precision": 0.5847457627118644,
225
+ "recall": 0.5655737704918032,
226
+ "support": 122
227
+ },
228
+ {
229
+ "f1": 0.6278026905829597,
230
+ "label": "food_drink",
231
+ "precision": 0.5833333333333334,
232
+ "recall": 0.6796116504854369,
233
+ "support": 103
234
+ },
235
+ {
236
+ "f1": 0.44881889763779526,
237
+ "label": "travel",
238
+ "precision": 0.475,
239
+ "recall": 0.4253731343283582,
240
+ "support": 134
241
+ },
242
+ {
243
+ "f1": 0.5333333333333333,
244
+ "label": "autos",
245
+ "precision": 0.5423728813559322,
246
+ "recall": 0.5245901639344263,
247
+ "support": 61
248
+ },
249
+ {
250
+ "f1": 0.6871165644171779,
251
+ "label": "weather",
252
+ "precision": 0.691358024691358,
253
+ "recall": 0.6829268292682927,
254
+ "support": 82
255
+ }
256
+ ]
257
+ },
258
+ "topic_map_sha256": "ed1b13889e9ed2f6964b98af94fa4b5c3a32cb224cbbadaa4c7be387e2f73220"
259
  },
260
  "dataset": "MIND-large dev",
261
  "eval_gate_real": {
 
314
  {
315
  "batch": 1,
316
  "mean_ms": 0.178,
317
+ "p50_ms": 0.176,
318
+ "p95_ms": 0.186,
319
  "runs": 1000,
320
  "threads": 1
321
  },
322
  {
323
  "batch": 32,
324
+ "mean_ms": 4.715,
325
+ "p50_ms": 4.407,
326
+ "p95_ms": 5.708,
327
  "runs": 1000,
328
  "threads": 1
329
  },
330
  {
331
  "batch": 1,
332
+ "mean_ms": 0.211,
333
  "p50_ms": 0.181,
334
+ "p95_ms": 0.38,
335
  "runs": 1000,
336
  "threads": 4
337
  },
338
  {
339
  "batch": 32,
340
+ "mean_ms": 3.671,
341
+ "p50_ms": 3.484,
342
+ "p95_ms": 4.568,
343
  "runs": 1000,
344
  "threads": 4
345
  }
 
348
  "parity": {
349
  "classifier": {
350
  "label_mismatches": 0,
351
+ "max_abs_diff": 3.5762786865234375e-07,
352
  "n": 2048,
353
  "ok": true,
354
  "tolerance": 0.0001
355
  },
356
+ "user_encoder": {
357
+ "inherited_from": "v0.1.0",
358
+ "note": "byte-identical copy; parity was checked when that release was built"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
359
  }
360
  },
361
  "results": {