REPORT.md: two-session re-check and new-user results
Browse files
REPORT.md
CHANGED
|
@@ -396,6 +396,42 @@ On this Reddit trace the community name genuinely predicted engagement. For exam
|
|
| 396 |
- Both are well above random everywhere.
|
| 397 |
- Single feeds (one request) are noisy: a 10-position list with 10–11 engaged posts out of 89–142.
|
| 398 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 399 |
---
|
| 400 |
|
| 401 |
## 14. Backend compatibility
|
|
|
|
| 396 |
- Both are well above random everywhere.
|
| 397 |
- Single feeds (one request) are noisy: a 10-position list with 10–11 engaged posts out of 89–142.
|
| 398 |
|
| 399 |
+
**Re-check on your two later sessions** (`engine_v2/validate_two_sessions.py` → `artifacts/validation_two_sessions.json`; each engine with its own content vectors)
|
| 400 |
+
|
| 401 |
+
| | Session 1 (89 posts, 11 engaged) | | | Session 2 (142 posts, 10 engaged) | | |
|
| 402 |
+
|---|---|---|---|---|---|---|
|
| 403 |
+
| | NDCG@10 | engaged in top 10 | AUC | NDCG@10 | engaged in top 10 | AUC |
|
| 404 |
+
| Engine 2.0 | 0.255 | 3 | 0.688 | 0.110 | 1 | 0.498 |
|
| 405 |
+
| 2.0 without content vectors | 0.249 | 3 | 0.760 | 0.110 | 1 | 0.532 |
|
| 406 |
+
| Engine 1.1 | 0.413 | 5 | 0.780 | 0.110 | 1 | 0.602 |
|
| 407 |
+
| Reddit feed order | 0.641 | 6 | 0.917 | 0.312 | 3 | 0.773 |
|
| 408 |
+
| Random | 0.124 | 1.1 | 0.50 | 0.070 | 0.6 | 0.50 |
|
| 409 |
+
|
| 410 |
+
25-post pages: 2.0 scored 0.468 and 0.403 (1.1: 0.559 and 0.381). 2.0 lost two engaged posts from the session-1 top 10. Its top two posts there were finance posts promoted by the topic-first rule, because "investing-and-markets" is one of P09's inferred interests.
|
| 411 |
+
|
| 412 |
+
**New users** (`engine_v2/validate_new_users.py` → `artifacts/new_user_report.json`)
|
| 413 |
+
|
| 414 |
+
The same real feeds, ranked as if each user had just signed up: picked interests only, no history (the code checks that w = 0). NDCG@10:
|
| 415 |
+
|
| 416 |
+
| Feed | New user (interests only) | New user, no interests | Same user, full history | Random |
|
| 417 |
+
|---|---|---|---|---|
|
| 418 |
+
| Validation, whole feeds | 0.525 | 0.564 | 0.650 | 0.350 |
|
| 419 |
+
| Validation, 25-post pages | 0.525 | 0.568 | 0.559 | 0.350 |
|
| 420 |
+
| Test, whole feeds | 0.410 | 0.303 | 0.340 | 0.201 |
|
| 421 |
+
| Test, 25-post pages | 0.426 | 0.460 | 0.415 | 0.286 |
|
| 422 |
+
| Session 1, one feed | 0.221 | 0.445 | 0.255 | 0.119 |
|
| 423 |
+
| Session 1, pages | 0.341 | 0.527 | 0.468 | 0.282 |
|
| 424 |
+
| Session 2, one feed | 0.139 | 0.139 | 0.110 | 0.079 |
|
| 425 |
+
| Session 2, pages | 0.359 | 0.377 | 0.403 | 0.245 |
|
| 426 |
+
| **Mean** | **0.368** | **0.423** | **0.400** | **0.239** |
|
| 427 |
+
|
| 428 |
+
**Reading the results**
|
| 429 |
+
- A brand-new user gets a useful feed from the first request: about 54% above random.
|
| 430 |
+
- History adds on top (0.368 → 0.400).
|
| 431 |
+
- A user with no picked interests is still served, ranked on post quality and content.
|
| 432 |
+
|
| 433 |
+
These users never picked interests themselves; theirs were inferred from behaviour, and some guesses were wrong. That is why "no interests" scored highest. Real signup picks should match better. If they don't, lower `topic_first_slots` for new users (not changed in 2.0).
|
| 434 |
+
|
| 435 |
---
|
| 436 |
|
| 437 |
## 14. Backend compatibility
|