Unmesh30 commited on
Commit
f29ec56
·
verified ·
1 Parent(s): 5ab5426

Release v0.1.0

Browse files
Files changed (8) hide show
  1. README.md +197 -0
  2. category_classifier.onnx +3 -0
  3. config.yaml +16 -0
  4. labels.json +1 -0
  5. manifest.json +118 -0
  6. metrics.json +422 -0
  7. population.npy +3 -0
  8. user_encoder.onnx +3 -0
README.md ADDED
@@ -0,0 +1,197 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: microsoft-research-license-mind
4
+ license_link: https://github.com/msnews/MIND/blob/master/MSR%20License_Data.pdf
5
+ library_name: onnx
6
+ language:
7
+ - en
8
+ tags:
9
+ - recommender-systems
10
+ - news-recommendation
11
+ - cold-start
12
+ - onnx
13
+ - mind
14
+ datasets:
15
+ - Recommenders/MIND
16
+ ---
17
+
18
+ # FirstRead models v0.1.0
19
+
20
+ Model artifacts for [FirstRead](https://github.com/Unmesh30/firstread), a real-time news recommender that ranks brand-new articles before they have any clicks.
21
+
22
+ - **Hub:** [Unmesh30/firstread-models](https://huggingface.co/Unmesh30/firstread-models). This release is the git tag [`v0.1.0`](https://huggingface.co/Unmesh30/firstread-models/tree/v0.1.0).
23
+ - **Pin a tag, not `main`.** Every file's SHA256 is listed in `manifest.json`.
24
+
25
+ > **License: these weights are MIND-derived and non-commercial.** They were trained on the Microsoft News Dataset (MIND) and are released under its [Microsoft Research License Terms](https://github.com/msnews/MIND/blob/master/MSR%20License_Data.pdf), **not** under the MIT license of the FirstRead code. Use them for non-commercial research and demonstration only.
26
+
27
+ ## What's in this release
28
+
29
+ | File | What it is |
30
+ |---|---|
31
+ | `user_encoder.onnx` | NRMS-lite user encoder. Turns a reader's last 50 clicked-article embeddings into a user vector. |
32
+ | `category_classifier.onnx` | Logistic regression that maps an article embedding to one of 15 MIND categories. |
33
+ | `labels.json` | Category names, in the classifier's output order. |
34
+ | `population.npy` | The 384-d user vector for a reader with no history: the normalized mean embedding of all training clicks. It is also baked into `user_encoder.onnx`. |
35
+ | `config.yaml` | The training config of the shipped model. |
36
+ | `metrics.json` | Dev results for all three seeds, bootstrap CIs, classifier metrics, parity, the real-data eval gate and latency. |
37
+ | `manifest.json` | SHA256 and size of every file, plus the git commit, embedding-model pins, feature constants and library versions. |
38
+
39
+ **What is not in this release:** MIND rows, article text, impression logs, user ids or embedding caches. Only learned weights and aggregate numbers are shipped. A build fails if its folder holds any file outside the list above.
40
+
41
+ ## Model details
42
+
43
+ **User encoder (NRMS-lite).**
44
+ - **Input:**
45
+ - `hist`: float32 `[batch, 50, 384]`. The reader's most recent clicked-article embeddings, oldest first and **right-aligned** (padding on the left, zeros).
46
+ - `mask`: bool `[batch, 50]`. True where a click exists.
47
+ - **Output:** `user_vec`, float32 `[batch, 384]`.
48
+ - **Architecture:** multi-head self-attention (8 heads) over the history, then additive attention pooling, then a linear projection. The result is added **residually** to the normalized mean of the history:
49
+
50
+ `u = normalize(mean(hist)) + α · proj(pool(MHSA(hist)))`
51
+
52
+ α starts at 0, so the untrained model is the mean-of-history baseline. The trained α is 0.0203.
53
+ - **Empty histories** return `population.npy`.
54
+ - **Scoring:** `score = user_vec · article_embedding`.
55
+ - The item tower is the identity on frozen article embeddings ([ADR 0004](https://github.com/Unmesh30/firstread/blob/main/docs/adr/0004-identity-item-tower.md)). Stored article vectors stay valid across model releases, and retrieval can run ANN directly on them.
56
+ - The training temperature is `score_scale` = 7.26, recorded in `manifest.json` and the ONNX metadata. It doesn't change rankings.
57
+ - **Article embeddings:**
58
+ - `BAAI/bge-small-en-v1.5`, the int8 ONNX export `Qdrant/bge-small-en-v1.5-onnx-Q` at a pinned revision, run through fastembed 0.9.0.
59
+ - Text is `title + ". " + abstract`, cleaned (text-prep version 1). Vectors are L2-normalized and 384-d.
60
+ - **Embed with exactly these pins** (see `manifest.json` → `embedding`). Vectors from any other embedding model are meaningless to this encoder.
61
+ - **Category classifier:**
62
+ - Input `embedding`: float32 `[batch, 384]`.
63
+ - Outputs: `label` (int64 index into `labels.json`) and `probabilities` (float32 `[batch, 15]`).
64
+ - Multinomial logistic regression, C = 30.
65
+ - **Export:** PyTorch 2.14 dynamo exporter, opset 18; skl2onnx with zipmap off. Only the batch dimension is dynamic.
66
+ - **Parity (ONNX Runtime vs PyTorch/sklearn):**
67
+ - Max abs difference ≤ 1e-4, checked on empty, partial and full histories, on real MIND histories and on mixed batches.
68
+ - A row scores identically alone and inside a batch.
69
+ - The measured diffs are in `metrics.json` → `parity` (all below 1e-6).
70
+
71
+ **Which weights ship:** seed 0, trained on all of MIND-large train (2019-11-09 to 2019-11-14). Seed 0 is the config's pre-declared seed, not the best of three. All three seeds are reported below.
72
+
73
+ ## Intended use
74
+
75
+ - **Use it for:** ranking English news articles for a reader, given that reader's recent clicks, in the non-commercial FirstRead demo. In particular, scoring **brand-new articles with no clicks**, from their text alone.
76
+ - **Use the classifier for:** a coarse topic label for live articles during ingestion.
77
+ - **Out of scope:**
78
+ - commercial use (the license forbids it);
79
+ - non-English text;
80
+ - decisions about people;
81
+ - any domain other than news;
82
+ - use with a different embedding model.
83
+
84
+ ## Training data
85
+
86
+ **MIND-large** (Wu et al., ACL 2020): impression logs from 1M Microsoft News (MSN) users over six weeks of October–November 2019. Training uses the train split's impressions (Nov 9–14, 2.23M impressions) with K=4 negatives drawn from the same impression and a softmax cross-entropy loss.
87
+
88
+ - Hyperparameters were chosen on a **time-based holdout** (the last day of MIND-small train), never on dev.
89
+ - Data handling and the eval protocol are described in [ADR 0003](https://github.com/Unmesh30/firstread/blob/main/docs/adr/0003-mind-data-and-eval-protocol.md) and the [data card](https://github.com/Unmesh30/firstread/blob/main/docs/data-card.md).
90
+
91
+ ## Evaluation
92
+
93
+ **Setup:**
94
+ - MIND-large **dev**: 376,471 impressions, evaluated once.
95
+ - Per-impression AUC, MRR, nDCG@5 and nDCG@10, averaged over impressions. Ties are broken by a seeded random key.
96
+ - Full report: [`ml/reports/results-mind-large-dev.md`](https://github.com/Unmesh30/firstread/blob/main/ml/reports/results-mind-large-dev.md).
97
+
98
+ | Model (nDCG@10) | Overall | Cold-start candidates | Cold users (≤ 2 clicks) |
99
+ |---|---:|---:|---:|
100
+ | random | 0.2864 | 0.4302 | 0.3119 |
101
+ | popularity | 0.3079 | 0.4303 | 0.3348 |
102
+ | recency | 0.2946 | 0.4830 | 0.3075 |
103
+ | mean_history (best baseline) | 0.3906 | 0.5332 | 0.3552 |
104
+ | **NRMS-lite seed 0 (shipped)** | **0.4149** | **0.5515** | **0.3625** |
105
+ | NRMS-lite, mean ± std over 3 seeds | 0.4145 ± 0.0004 | 0.5506 ± 0.0008 | 0.3623 ± 0.0001 |
106
+
107
+ - Shipped model, overall: AUC 0.6624, MRR 0.3190, nDCG@5 0.3536.
108
+ - **Cold-start candidates** are articles never clicked in training: 125,472 impressions. These are the articles FirstRead exists for.
109
+ - **Cold users:** 25,688 impressions, 11,270 of them with an empty history.
110
+
111
+ **Gain over mean_history (paired bootstrap):**
112
+ - Mean of the 3 seeds, 1,000 resamples of impressions.
113
+ - nDCG@10 +0.0239 overall (95% CI [+0.0232, +0.0248]) and +0.0174 on cold-start candidates (95% CI [+0.0159, +0.0188]).
114
+ - No resample showed a gain ≤ 0.
115
+
116
+ **Reference, not directly comparable:** NRMS in the MIND paper reports nDCG@10 0.4163 on the MIND-large **test** set (Wu et al., 2020, Table 3).
117
+ - It uses a different split (test, not our dev).
118
+ - It learns its own news encoder, while ours is frozen bge-small.
119
+
120
+ **Real-data release gate:**
121
+ - Before release, the exported ONNX file is scored on a fixed 5,000-impression MIND-small dev subset.
122
+ - It must beat mean_history and must not regress more than 1% against the committed reference.
123
+ - v0.1.0: nDCG@10 0.4217 vs 0.3946 overall, and 0.4600 vs 0.4329 on cold-start candidates.
124
+
125
+ ### Where the model fails
126
+
127
+ **Per category,** NRMS-lite loses to the simple mean_history baseline on four categories (nDCG@10, seed 0 vs mean_history):
128
+
129
+ | Category | NRMS-lite | mean_history |
130
+ |---|---:|---:|
131
+ | tv | 0.7193 | 0.7481 |
132
+ | weather | 0.7970 | 0.8059 |
133
+ | video | 0.6953 | 0.7033 |
134
+ | lifestyle | 0.6754 | 0.6786 |
135
+
136
+ - It wins on the other ten categories with enough data, by the largest margins on news (+0.050) and travel (+0.054).
137
+ - A plausible reading, not verified: in these categories a reader's interest is narrow and stable, and the plain average of their history already captures it.
138
+ - **Cold users gain least:** +0.007 nDCG@10. With 0–2 clicks there is little history for attention to work with, and empty histories fall back to one population vector for everyone.
139
+
140
+ ### Caveats on the numbers
141
+
142
+ - **The bootstrap resamples impressions, not users.** One user contributes many correlated impressions, so the CIs above somewhat **understate** the uncertainty. A user-level (cluster) bootstrap would be wider.
143
+ - **Seeds:** the std over 3 seeds measures training noise only, not data noise.
144
+ - **Dev is not test.** MIND test labels aren't public. Our dev numbers are reported once, after tuning on a separate holdout.
145
+
146
+ ## Bias and limitations
147
+
148
+ - **2019, US, one platform.** MIND reflects what MSN users in the US clicked in late 2019, under MSN's own ranking and layout (position bias, editorial choices). The topics, publishers and events of that period don't match today's news. The model has never seen FirstRead's live sources (GDELT, RSS), whose distribution differs.
149
+ - **Popularity and exposure bias.** Clicks are only observed on what MSN chose to show.
150
+ - **Text-only.** The model knows an article only by its headline and summary, through a frozen general-purpose embedding. It can't see images, the publisher's reputation or the full text.
151
+ - **English only.**
152
+ - **The classifier's labels are MIND's categories.**
153
+ - On dev-only articles: accuracy 0.7214, macro-F1 0.5984; majority-class accuracy is 0.3255.
154
+ - The weakest classes are video (F1 0.34), tv (0.40) and travel (0.44); kids has only 3 test articles.
155
+ - Three rare MIND categories (games, middleeast, northamerica) are not predicted at all.
156
+ - Report: [`ml/reports/classifier-mind-large.md`](https://github.com/Unmesh30/firstread/blob/main/ml/reports/classifier-mind-large.md).
157
+
158
+ ## Latency (ONNX Runtime, CPU)
159
+
160
+ User encoder, full 50-click histories, after 50 warm-up runs, 1,000 timed runs per row. **Measured on the owner's Mac (Apple Silicon), not on Cloud Run.** P3 re-measures inside the API container.
161
+
162
+ Apple M4 (10 cores), macOS 26.3, ONNX Runtime 1.31.0:
163
+
164
+ | Batch | intra-op threads | p50 | p95 |
165
+ |---:|---:|---:|---:|
166
+ | 1 | 1 | 0.174 ms | 0.179 ms |
167
+ | 32 | 1 | 4.273 ms | 4.744 ms |
168
+ | 1 | 4 | 0.181 ms | 0.183 ms |
169
+ | 32 | 4 | 3.363 ms | 3.885 ms |
170
+
171
+ - Batching 32 readers costs about 0.13 ms per reader, against 0.17 ms one at a time.
172
+ - Extra threads help only at batch 32.
173
+ - Even if Cloud Run's vCPU is several times slower than an M4 core, the encoder should be a small part of the 20 ms scoring budget (SPEC §8). P3 verifies this.
174
+
175
+ ## Commercial launch
176
+
177
+ These weights may not be used commercially. Before any commercial launch, FirstRead must:
178
+ - retrain on its own consented interaction data;
179
+ - switch to a licensed news source;
180
+ - publish the result as a new release under a new license.
181
+
182
+ ## Verifying a download
183
+
184
+ ```bash
185
+ hf download Unmesh30/firstread-models --revision v0.1.0 --local-dir firstread-models
186
+ cd firstread-models && python -c "
187
+ import hashlib, json; m = json.load(open('manifest.json'))
188
+ for n, f in m['files'].items():
189
+ assert hashlib.sha256(open(n, 'rb').read()).hexdigest() == f['sha256'], n
190
+ print('all', len(m['files']), 'files match')"
191
+ ```
192
+
193
+ Or, from the FirstRead repo: `make release-verify VERSION=0.1.0`.
194
+
195
+ ## Citation
196
+
197
+ Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu and Ming Zhou. 2020. *MIND: A Large-scale Dataset for News Recommendation.* In Proceedings of ACL 2020, pages 3597–3606.
category_classifier.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:669cafa314762ad1721e5a7780ac580a6d0ce8e59a04e570b71622d0cb060b87
3
+ size 29452
config.yaml ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ seed: 0
2
+ model:
3
+ heads: 8
4
+ query_dim: 200
5
+ dropout: 0.2
6
+ attn_dropout: 0.1
7
+ init_scale: 10.0
8
+ train:
9
+ epochs: 5
10
+ batch_size: 256
11
+ lr: 0.001
12
+ weight_decay: 0.01
13
+ negatives: 4
14
+ grad_clip: 1.0
15
+ lr_schedule: cosine
16
+ device: auto
labels.json ADDED
@@ -0,0 +1 @@
 
 
1
+ ["autos", "entertainment", "finance", "foodanddrink", "health", "kids", "lifestyle", "movies", "music", "news", "sports", "travel", "tv", "video", "weather"]
manifest.json ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "built_at": "2026-10-14T16:35:04Z",
3
+ "category_classifier": {
4
+ "file": "category_classifier.onnx",
5
+ "input": {
6
+ "embedding": [
7
+ "batch",
8
+ 384
9
+ ]
10
+ },
11
+ "labels": "labels.json",
12
+ "n_labels": 15,
13
+ "outputs": {
14
+ "label": [
15
+ "batch"
16
+ ],
17
+ "probabilities": [
18
+ "batch",
19
+ 15
20
+ ]
21
+ }
22
+ },
23
+ "dataset": {
24
+ "license": "MSR License Terms",
25
+ "name": "MIND",
26
+ "revision": "2115068f457b577f93a268fffe22373e45727d89"
27
+ },
28
+ "dry_run": false,
29
+ "embedding": {
30
+ "fastembed": "0.9.0",
31
+ "file": "model_optimized.onnx",
32
+ "l2_normalized": true,
33
+ "model": "BAAI/bge-small-en-v1.5",
34
+ "revision": "aa8f8b060edb00e03bfdd08813a2949946c8ba55",
35
+ "source": "Qdrant/bge-small-en-v1.5-onnx-Q",
36
+ "text_prep_version": 1
37
+ },
38
+ "features": {
39
+ "embed_dim": 384,
40
+ "history_len": 50,
41
+ "right_aligned": true
42
+ },
43
+ "files": {
44
+ "README.md": {
45
+ "bytes": 11855,
46
+ "sha256": "178ce9daf603491a609b29992baf03c0381be04524b9992f2db02e85a9cecc9b"
47
+ },
48
+ "category_classifier.onnx": {
49
+ "bytes": 29452,
50
+ "sha256": "669cafa314762ad1721e5a7780ac580a6d0ce8e59a04e570b71622d0cb060b87"
51
+ },
52
+ "config.yaml": {
53
+ "bytes": 236,
54
+ "sha256": "bfa55a860da3fa69035d626355f9f195b7737aeac1832044cc1c585673e4e7ad"
55
+ },
56
+ "labels.json": {
57
+ "bytes": 158,
58
+ "sha256": "c6647a46aceafa50ceef453794bc944de7ef94b1eb55b4dbf7efa7d5e1ff43ff"
59
+ },
60
+ "metrics.json": {
61
+ "bytes": 9520,
62
+ "sha256": "cf1cfe4e8713a9f873dbf60ae1a34237b72f83a0c01589821b79d720f1058950"
63
+ },
64
+ "population.npy": {
65
+ "bytes": 1664,
66
+ "sha256": "2b7bfc89d363b759e4d245c7f3d63b63dd9d1b2880d7d1355fabafe421599594"
67
+ },
68
+ "user_encoder.onnx": {
69
+ "bytes": 3352804,
70
+ "sha256": "adbfb203897441188d81d72c8cde71b664d921e52f17ce482c60c2dd807a30bf"
71
+ }
72
+ },
73
+ "git_commit": "5db524e0eaf58e48a5d1ef7ed4374acaf2bdc494",
74
+ "hf_repo": "Unmesh30/firstread-models",
75
+ "license": "other (MIND-derived weights, non-commercial; see README.md)",
76
+ "name": "firstread-models",
77
+ "tag": "v0.1.0",
78
+ "user_encoder": {
79
+ "alpha": 0.020342299714684486,
80
+ "checkpoint_id": "49bf8339c2250819",
81
+ "file": "user_encoder.onnx",
82
+ "input_dtypes": {
83
+ "hist": "float32",
84
+ "mask": "bool"
85
+ },
86
+ "inputs": {
87
+ "hist": [
88
+ "batch",
89
+ 50,
90
+ 384
91
+ ],
92
+ "mask": [
93
+ "batch",
94
+ 50
95
+ ]
96
+ },
97
+ "opset": 18,
98
+ "output": {
99
+ "user_vec": [
100
+ "batch",
101
+ 384
102
+ ]
103
+ },
104
+ "score_scale": 7.262218475341797,
105
+ "seed": 0,
106
+ "trained_on": "MIND-large train (all days)"
107
+ },
108
+ "version": "0.1.0",
109
+ "versions": {
110
+ "firstread_core": "0.1.0",
111
+ "onnx": "1.23.2",
112
+ "onnxruntime": "1.31.0",
113
+ "python": "3.12.13",
114
+ "scikit-learn": "1.9.1",
115
+ "skl2onnx": "1.20.0",
116
+ "torch": "2.14.1"
117
+ }
118
+ }
metrics.json ADDED
@@ -0,0 +1,422 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bootstrap": {
3
+ "baseline": "mean_history",
4
+ "metric": "ndcg@10",
5
+ "slices": {
6
+ "cold_candidates": {
7
+ "bootstrap_seed": 12345,
8
+ "ci95_high": 0.01878891816194426,
9
+ "ci95_low": 0.015859448836582184,
10
+ "gain_mean": 0.01736395319661053,
11
+ "n_impressions": 125472,
12
+ "per_seed_gain": [
13
+ 0.018245716935635255,
14
+ 0.016926750348847792,
15
+ 0.016919392305348543
16
+ ],
17
+ "resamples": 1000,
18
+ "share_resamples_gain_le_0": 0.0
19
+ },
20
+ "overall": {
21
+ "bootstrap_seed": 12345,
22
+ "ci95_high": 0.024816258579701853,
23
+ "ci95_low": 0.023167105190676558,
24
+ "gain_mean": 0.023932217960012436,
25
+ "n_impressions": 376471,
26
+ "per_seed_gain": [
27
+ 0.024344917862921273,
28
+ 0.023524082491960308,
29
+ 0.02392765352515573
30
+ ],
31
+ "resamples": 1000,
32
+ "share_resamples_gain_le_0": 0.0
33
+ }
34
+ }
35
+ },
36
+ "classifier": {
37
+ "accuracy": 0.7214176829268293,
38
+ "dataset": "MIND-large, articles only in dev news.tsv",
39
+ "dropped_labels": [
40
+ "games",
41
+ "middleeast",
42
+ "northamerica"
43
+ ],
44
+ "labels": [
45
+ "autos",
46
+ "entertainment",
47
+ "finance",
48
+ "foodanddrink",
49
+ "health",
50
+ "kids",
51
+ "lifestyle",
52
+ "movies",
53
+ "music",
54
+ "news",
55
+ "sports",
56
+ "travel",
57
+ "tv",
58
+ "video",
59
+ "weather"
60
+ ],
61
+ "macro_f1": 0.5984492354428844,
62
+ "majority_class_accuracy": 0.3254573170731707,
63
+ "n_test": 2624
64
+ },
65
+ "dataset": "MIND-large dev",
66
+ "eval_gate_real": {
67
+ "meta": {
68
+ "dataset_revision": "2115068f457b577f93a268fffe22373e45727d89",
69
+ "split": "dev",
70
+ "subset_seed": 0,
71
+ "subset_size": 5000,
72
+ "variant": "small"
73
+ },
74
+ "ndcg@10": {
75
+ "mean_history": {
76
+ "cold_candidates": 0.432943,
77
+ "cold_users": 0.348226,
78
+ "overall": 0.39458
79
+ },
80
+ "onnx_release": {
81
+ "cold_candidates": 0.460018,
82
+ "cold_users": 0.366311,
83
+ "overall": 0.421722
84
+ }
85
+ },
86
+ "passed": true,
87
+ "slices": {
88
+ "category:autos": 149,
89
+ "category:entertainment": 167,
90
+ "category:finance": 231,
91
+ "category:foodanddrink": 338,
92
+ "category:games": 0,
93
+ "category:health": 219,
94
+ "category:kids": 0,
95
+ "category:lifestyle": 933,
96
+ "category:middleeast": 0,
97
+ "category:movies": 143,
98
+ "category:music": 104,
99
+ "category:news": 1191,
100
+ "category:northamerica": 0,
101
+ "category:sports": 1432,
102
+ "category:travel": 106,
103
+ "category:tv": 286,
104
+ "category:video": 158,
105
+ "category:weather": 37,
106
+ "cold_candidates": 2745,
107
+ "cold_users": 340,
108
+ "overall": 5000
109
+ }
110
+ },
111
+ "latency": {
112
+ "machine": {
113
+ "chip": "Apple M4",
114
+ "cpus": "10",
115
+ "os": "macOS-26.3.1-arm64-arm-64bit"
116
+ },
117
+ "onnxruntime": "1.31.0",
118
+ "rows": [
119
+ {
120
+ "batch": 1,
121
+ "mean_ms": 0.178,
122
+ "p50_ms": 0.177,
123
+ "p95_ms": 0.183,
124
+ "runs": 1000,
125
+ "threads": 1
126
+ },
127
+ {
128
+ "batch": 32,
129
+ "mean_ms": 4.476,
130
+ "p50_ms": 4.399,
131
+ "p95_ms": 4.94,
132
+ "runs": 1000,
133
+ "threads": 1
134
+ },
135
+ {
136
+ "batch": 1,
137
+ "mean_ms": 0.22,
138
+ "p50_ms": 0.181,
139
+ "p95_ms": 0.457,
140
+ "runs": 1000,
141
+ "threads": 4
142
+ },
143
+ {
144
+ "batch": 32,
145
+ "mean_ms": 3.727,
146
+ "p50_ms": 3.417,
147
+ "p95_ms": 4.417,
148
+ "runs": 1000,
149
+ "threads": 4
150
+ }
151
+ ]
152
+ },
153
+ "parity": {
154
+ "classifier": {
155
+ "label_mismatches": 0,
156
+ "max_abs_diff": 2.384185791015625e-07,
157
+ "n": 2048,
158
+ "ok": true,
159
+ "tolerance": 0.0001
160
+ },
161
+ "user_encoder_real": {
162
+ "batch_invariance": 0.0,
163
+ "max_abs_diff": {
164
+ "batch32": 1.7881393432617188e-07,
165
+ "len0": 0.0,
166
+ "len1": 3.2782554626464844e-07,
167
+ "len49": 1.7881393432617188e-07,
168
+ "len50": 1.2665987014770508e-07,
169
+ "len7": 1.4901161193847656e-07,
170
+ "mixed": 2.384185791015625e-07,
171
+ "real_histories": 6.258487701416016e-07
172
+ },
173
+ "ok": true,
174
+ "tolerance": 0.0001,
175
+ "worst": 6.258487701416016e-07
176
+ },
177
+ "user_encoder_synthetic": {
178
+ "batch_invariance": 0.0,
179
+ "max_abs_diff": {
180
+ "batch32": 1.7881393432617188e-07,
181
+ "len0": 0.0,
182
+ "len1": 3.2782554626464844e-07,
183
+ "len49": 1.7881393432617188e-07,
184
+ "len50": 1.2665987014770508e-07,
185
+ "len7": 1.4901161193847656e-07,
186
+ "mixed": 2.384185791015625e-07
187
+ },
188
+ "ok": true,
189
+ "tolerance": 0.0001,
190
+ "worst": 3.2782554626464844e-07
191
+ }
192
+ },
193
+ "results": {
194
+ "mean_history": {
195
+ "cold_candidates": {
196
+ "auc": 0.6324316689136716,
197
+ "mrr": 0.42461835559498284,
198
+ "ndcg@10": 0.5332137084239148,
199
+ "ndcg@5": 0.4780115575532085
200
+ },
201
+ "cold_users": {
202
+ "auc": 0.5581429913735874,
203
+ "mrr": 0.2729486864021774,
204
+ "ndcg@10": 0.3551670278023427,
205
+ "ndcg@5": 0.29362105050668746
206
+ },
207
+ "overall": {
208
+ "auc": 0.6246578761530922,
209
+ "mrr": 0.2999019545432084,
210
+ "ndcg@10": 0.3906026055566603,
211
+ "ndcg@5": 0.3291824815400495
212
+ }
213
+ },
214
+ "nrms_lite_s0": {
215
+ "cold_candidates": {
216
+ "auc": 0.6591007517080358,
217
+ "mrr": 0.4406796580760931,
218
+ "ndcg@10": 0.55145942535955,
219
+ "ndcg@5": 0.498055574055922
220
+ },
221
+ "cold_users": {
222
+ "auc": 0.5716962819371971,
223
+ "mrr": 0.2790963861247484,
224
+ "ndcg@10": 0.3624854473252031,
225
+ "ndcg@5": 0.30122273529552634
226
+ },
227
+ "overall": {
228
+ "auc": 0.6623871500247188,
229
+ "mrr": 0.3189863291171016,
230
+ "ndcg@10": 0.4149475234195816,
231
+ "ndcg@5": 0.3536017233878056
232
+ }
233
+ },
234
+ "nrms_lite_s1": {
235
+ "cold_candidates": {
236
+ "auc": 0.6581422557535548,
237
+ "mrr": 0.43927381767112933,
238
+ "ndcg@10": 0.5501404587727624,
239
+ "ndcg@5": 0.4966467644259163
240
+ },
241
+ "cold_users": {
242
+ "auc": 0.5712849029040606,
243
+ "mrr": 0.2788251954731917,
244
+ "ndcg@10": 0.3622156275549781,
245
+ "ndcg@5": 0.30126478240214766
246
+ },
247
+ "overall": {
248
+ "auc": 0.6617699420349328,
249
+ "mrr": 0.3180469168821803,
250
+ "ndcg@10": 0.41412668804862063,
251
+ "ndcg@5": 0.35259653822267617
252
+ }
253
+ },
254
+ "nrms_lite_s2": {
255
+ "cold_candidates": {
256
+ "auc": 0.6576318646482496,
257
+ "mrr": 0.43916890011004145,
258
+ "ndcg@10": 0.5501331007292632,
259
+ "ndcg@5": 0.49657430842303923
260
+ },
261
+ "cold_users": {
262
+ "auc": 0.5710677317440604,
263
+ "mrr": 0.2782410363090575,
264
+ "ndcg@10": 0.36224011028274966,
265
+ "ndcg@5": 0.30046827807797755
266
+ },
267
+ "overall": {
268
+ "auc": 0.6620545936191268,
269
+ "mrr": 0.31856653726011797,
270
+ "ndcg@10": 0.41453025908181607,
271
+ "ndcg@5": 0.3530209299323185
272
+ }
273
+ },
274
+ "popularity": {
275
+ "cold_candidates": {
276
+ "auc": 0.5,
277
+ "mrr": 0.3261811083043824,
278
+ "ndcg@10": 0.4302939392596537,
279
+ "ndcg@5": 0.36316005040780136
280
+ },
281
+ "cold_users": {
282
+ "auc": 0.5439906748380747,
283
+ "mrr": 0.25652283764954836,
284
+ "ndcg@10": 0.3347620826953331,
285
+ "ndcg@5": 0.27220816832688705
286
+ },
287
+ "overall": {
288
+ "auc": 0.5385269910412536,
289
+ "mrr": 0.23371154987845752,
290
+ "ndcg@10": 0.30790251301600663,
291
+ "ndcg@5": 0.24538664136262714
292
+ }
293
+ },
294
+ "random": {
295
+ "cold_candidates": {
296
+ "auc": 0.4990773252887991,
297
+ "mrr": 0.3256328223096701,
298
+ "ndcg@10": 0.43018173738832904,
299
+ "ndcg@5": 0.36241606893915296
300
+ },
301
+ "cold_users": {
302
+ "auc": 0.501667122371972,
303
+ "mrr": 0.24186796156568705,
304
+ "ndcg@10": 0.31191727277713,
305
+ "ndcg@5": 0.2476318560164239
306
+ },
307
+ "overall": {
308
+ "auc": 0.500015764538092,
309
+ "mrr": 0.21873468315228298,
310
+ "ndcg@10": 0.2863806054251861,
311
+ "ndcg@5": 0.22332871696768097
312
+ }
313
+ },
314
+ "recency": {
315
+ "cold_candidates": {
316
+ "auc": 0.5717780181218608,
317
+ "mrr": 0.3756080602430753,
318
+ "ndcg@10": 0.483022817009282,
319
+ "ndcg@5": 0.41937461292752437
320
+ },
321
+ "cold_users": {
322
+ "auc": 0.4908663230522278,
323
+ "mrr": 0.23064831084833823,
324
+ "ndcg@10": 0.30751459622304883,
325
+ "ndcg@5": 0.2474741348776389
326
+ },
327
+ "overall": {
328
+ "auc": 0.5116666689745792,
329
+ "mrr": 0.21875781691365018,
330
+ "ndcg@10": 0.29458689008731354,
331
+ "ndcg@5": 0.23305307154273616
332
+ }
333
+ }
334
+ },
335
+ "seed_summary": {
336
+ "cold_candidates": {
337
+ "auc": {
338
+ "max": 0.6591007517080358,
339
+ "mean": 0.6582916240366133,
340
+ "min": 0.6576318646482496,
341
+ "std": 0.0007457482561761403
342
+ },
343
+ "mrr": {
344
+ "max": 0.4406796580760931,
345
+ "mean": 0.43970745861908794,
346
+ "min": 0.43916890011004145,
347
+ "std": 0.0008435821014026912
348
+ },
349
+ "ndcg@10": {
350
+ "max": 0.55145942535955,
351
+ "mean": 0.5505776616205252,
352
+ "min": 0.5501331007292632,
353
+ "std": 0.0007636386604901632
354
+ },
355
+ "ndcg@5": {
356
+ "max": 0.498055574055922,
357
+ "mean": 0.49709221563495914,
358
+ "min": 0.49657430842303923,
359
+ "std": 0.0008350790702192205
360
+ }
361
+ },
362
+ "cold_users": {
363
+ "auc": {
364
+ "max": 0.5716962819371971,
365
+ "mean": 0.5713496388617727,
366
+ "min": 0.5710677317440604,
367
+ "std": 0.00031923642412593796
368
+ },
369
+ "mrr": {
370
+ "max": 0.2790963861247484,
371
+ "mean": 0.27872087263566586,
372
+ "min": 0.2782410363090575,
373
+ "std": 0.0004371135637596898
374
+ },
375
+ "ndcg@10": {
376
+ "max": 0.3624854473252031,
377
+ "mean": 0.36231372838764364,
378
+ "min": 0.3622156275549781,
379
+ "std": 0.0001492159379128168
380
+ },
381
+ "ndcg@5": {
382
+ "max": 0.30126478240214766,
383
+ "mean": 0.3009852652585505,
384
+ "min": 0.30046827807797755,
385
+ "std": 0.0004482173562585426
386
+ }
387
+ },
388
+ "overall": {
389
+ "auc": {
390
+ "max": 0.6623871500247188,
391
+ "mean": 0.662070561892926,
392
+ "min": 0.6617699420349328,
393
+ "std": 0.00030891368533949605
394
+ },
395
+ "mrr": {
396
+ "max": 0.3189863291171016,
397
+ "mean": 0.3185332610864666,
398
+ "min": 0.3180469168821803,
399
+ "std": 0.00047058932688664876
400
+ },
401
+ "ndcg@10": {
402
+ "max": 0.4149475234195816,
403
+ "mean": 0.41453482351667276,
404
+ "min": 0.41412668804862063,
405
+ "std": 0.0004104367211938037
406
+ },
407
+ "ndcg@5": {
408
+ "max": 0.3536017233878056,
409
+ "mean": 0.3530730638476001,
410
+ "min": 0.35259653822267617,
411
+ "std": 0.0005046164512686953
412
+ }
413
+ }
414
+ },
415
+ "shipped_model": "nrms_lite_s0",
416
+ "shipped_seed": 0,
417
+ "slices": {
418
+ "cold_candidates": 125472,
419
+ "cold_users": 25688,
420
+ "overall": 376471
421
+ }
422
+ }
population.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2b7bfc89d363b759e4d245c7f3d63b63dd9d1b2880d7d1355fabafe421599594
3
+ size 1664
user_encoder.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:adbfb203897441188d81d72c8cde71b664d921e52f17ce482c60c2dd807a30bf
3
+ size 3352804