docs/phases/PHASE7-Data-Freshness-And-Capacity.md

# PHASE 7 — Data Freshness and Capacity

**Status:** Planning · **Created:** 2026-07-30
**Prerequisite for:** Phase 8 (reranker retrain), Phase 9 (exploration / CF)

Every figure below was measured against the live deployment on 2026-07-29/30,
not estimated. Where something is inferred rather than measured it says so.

---

## 1. The one-paragraph version

The corpus is a static Kaggle snapshot whose newest paper was published
**2025-05**. It is now **2026-07**. Live arXiv holds **290,672** papers in our
43 configured categories that we do not have — for cs.AI alone, arXiv has
published nearly half again as many papers in that window as our entire index
contains. Separately, **90.0%** of stored abstracts are truncated at 500
characters, and that truncated text is what the cross-encoder reranks on. No
amount of ranking work fixes either. This phase closes both.

The blocker is disk on the Qdrant free tier, and there is an unresolved
measurement discrepancy that must be settled before anything is deleted
(§4.2).

---

## 2. Measured current state

### 2.1 Corpus

| Property | Value |
|---|---|
| Papers (Turso, sidecar) | 1,597,097 |
| Qdrant points | 1,596,587 (510 fewer; 0.03%) |
| Qdrant indexed into HNSW | 1,595,264 (1,323 pending) |
| `update_date` range | 2007-05-23 → 2025-05-30 |
| Newest *publication* (from arXiv id) | 2025-05 |
| Zilliz sparse | LoadStateLoaded |

Field completeness is perfect: zero rows missing title, authors, categories or
date. 16.7% (265,928) have zero citations.

Store alignment is **not** a problem. A 400-id sample resolved 400/400 in
Qdrant. The 510-row gap needs no action.

### 2.2 Qdrant configuration

Binary quantization **is** enabled (`quantization_config.binary.always_ram =
true`). An earlier reading of `config.params.quantization_config` returned
null and was wrong — the setting lives at `config.quantization_config`.

```
vectors      float16, 1024-dim, on_disk = true
hnsw         m=32, ef_construct=128, on_disk = true
payload      on_disk = true
segments     35, max_segment_size = 100,000
```

### 2.3 Per-point cost (measured, not estimated)

From segment telemetry: segment 0 holds 1,323 points with
`vectors_size_bytes = 2,709,504` and `payloads_size_bytes = 169,344`.

| Component | Bytes/point | × 1,596,587 |
|---|---|---|
| Vector (float16 × 1024) | 2,048 | 3.27 GB |
| Payload | 128 | 204 MB |
| Binary quantization | 128 | 204 MB |
| HNSW graph (m₀=64 × u32, inferred) | ~270 | ~431 MB |
| **Total** | **~2,574** | **≈ 4.1 GB** |

Only the graph row is inferred. `vectors_size_bytes` is a *logical* figure
(exactly `count × 1024 × 2`), and segment `disk_usage_bytes` reports 0, so
Qdrant does not expose true on-disk bytes through the collection API.

---

## 3. The gaps

### 3.1 Staleness — the primary product limit

Union queries against live arXiv, papers submitted 2025-06-01 → 2026-07-30:

| Scope | Unique papers |
|---|---|
| 8 core CS/ML categories | 170,668 |
| **All 43 configured categories** | **290,672** |

Summing categories individually gives 242,427 for the 8-category set — that
counts a paper once per tag. The overlap factor is 1.42×, so always use union
counts for capacity planning.

Per-category, since 2025-06:

| Category | New on arXiv | Entire corpus |
|---|---|---|
| cs.AI | 62,438 | 128,356 |
| cs.LG | 58,747 | 219,516 |
| cs.CV | 43,269 | 156,798 |
| cs.CL | 29,295 | 85,314 |
| quant-ph | 21,026 | 162,381 |

### 3.2 Abstract truncation

`abstract_preview` is capped at 500 chars and **1,437,656 rows (90.0%)** sit at
that cap. Sampling 200 live arXiv abstracts: mean **1,244** chars, median 1,221,
min 601 — every one exceeds the cap, so roughly **60% of each abstract is
discarded**.

Two consequences:

1. The cross-encoder — the stage whose entire purpose is precision — reranks on
   roughly one or two sentences. Since `SEARCH_RERANK_TOP_N` is now 50, it does
   this for 50 candidates per query.
2. The original embeddings were built from `abstract[:1024]`, i.e. **twice** the
   text the reranker sees. The precision stage has strictly less information
   than the retrieval stage it is supposed to refine.

### 3.3 Dependency risk found while validating

`requirements.txt` allowed `FlagEmbedding>=1.2.9` with
`transformers>=4.44,<5.0`. A fresh resolve picks FlagEmbedding 1.3.x and a
recent transformers, and BGE-M3 encoding dies inside `tokenizer.pad()` with
`AttributeError: 'list' object has no attribute 'keys'`.

This is silent in production: `main.py` catches the load error, logs "BGE-M3 not
loaded", and search degrades to the arXiv keyword API with nothing surfaced to
the user. Pinned to `1.2.10` + `4.44.2` (verified loading) in commit `4b88183`.

---

## 4. Capacity

### 4.1 What the ingest costs

At ~2,574 bytes/point:

| Scope | Papers | Qdrant disk | BQ RAM |
|---|---|---|---|
| All 43 categories | 290,672 | **+0.75 GB** | +37 MB |
| 8 core CS/ML | 170,668 | +0.44 GB | +22 MB |

Other stores: Turso ~465 MB and 290k row writes (free tier is 9 GB / 25M
writes/month — comfortable). Zilliz ~400–450 MB sparse, and its serverless
free-tier cap is **not verifiable through the API** — this is the least
characterised constraint.

### 4.2 Decision gate: the disk reading must be settled first

Two irreconcilable figures:

* This document's arithmetic: **≈ 4.1 GB** used, against a 4 GB free-tier disk.
* The Qdrant Cloud console, as reported: **400–500 MB**.

400–500 MB cannot be the on-disk total. Raw float16 vectors are not compressed,
so 1,596,587 × 2,048 bytes = 3.27 GB is a floor for the vector data alone. The
reported figure does, however, match `memory_resident_bytes = 529,870,848`
(530 MB) almost exactly, which suggests the panel is reporting **RAM, not
disk** — consistent with a 1 GB RAM tier at ~53% utilisation.

**Nothing may be deleted until this is resolved.** The two branches lead to
opposite actions:

| If the true disk usage is… | Then |
|---|---|
| ~500 MB (≈3.5 GB free) | Ingest all 290,672. **Delete nothing.** |
| ~4.1 GB (≈0 GB free) | Ingest cannot proceed as-is; see §5 |

How to resolve: in the Qdrant Cloud console, find the panel labelled *disk* or
*storage* rather than memory. Failing that, upsert 5,000 test points and observe
whether reported usage moves by ~13 MB (RAM) or ~13 MB of disk — then delete
them.

### 4.3 Eviction analysis — dropping pre-2010

Requested on 2026-07-30. Measured consequences:

| | |
|---|---|
| Papers removed | **180,197** (11.3% of corpus) |
| Disk freed | **0.46 GB** |
| Ingest still needs | 0.75 GB |
| **Net after both** | **+0.28 GB — still grows** |
| Of those removed, with >0 citations | 155,312 (**86%**) |
| Of those removed, with ≥100 citations | **17,776** |

Most affected categories:

```
quant-ph            35,763      hep-th               8,477
astro-ph            30,329      math-ph              8,178
cond-mat.stat-mech  10,639      math.MP              8,178
math.CO              9,525      math.PR              8,754
```

**Recommendation: do not do this.** Three reasons:

1. **It does not solve the problem.** 0.46 GB freed against 0.75 GB needed still
   leaves +0.28 GB of growth. If disk is genuinely full, this buys a delay, not
   a fix.
2. **The cost is high and irreversible.** 17,776 removed papers have 100+
   citations. Restoring any of them means re-fetching and re-encoding on a GPU
   we do not currently have working (§6.1).
3. **It lands on the wrong fields.** The damage concentrates in quant-ph,
   astro-ph, hep-th and mathematics — disciplines whose literature ages slowly.
   A 2008 hep-th paper is often still current in a way a 2008 cs.CV paper is
   not. Our onboarding taxonomy offers all four as first-class choices, so those
   users would lose the foundations of their fields while ML users lose nothing.

Cheaper alternatives, in preference order:

| Option | Frees / saves | Cost |
|---|---|---|
| Resolve §4.2 first; likely no action needed | up to 3.5 GB of headroom | none |
| Narrow ingest to 8 core CS/ML categories | needs 0.44 GB not 0.75 GB | no physics/maths freshness |
| Drop pre-2010 papers with **zero** citations only | 0.06 GB (24,885 papers) | negligible |
| Lower HNSW `m` 32 → 16 | ~215 MB | full re-index; small recall cost |

---

## 5. Plan

### Phase 7.0 — Resolve capacity (blocking, minutes)
Settle §4.2. Everything downstream branches on it.

### Phase 7.1 — Get a working GPU (blocking for ingest)
Kaggle assigned a **P100 (sm_60)**; its own `torch 2.10.0+cu128` dropped Pascal
support, so encoding aborts. `"accelerator": "nvidiaTeslaT4"` in
`kernel-metadata.json` is silently ignored — GPU *type* is not settable via the
API. Requires switching the notebook to **GPU T4 x2** in the Kaggle UI.

Why it matters: the arXiv rate limit (~3.5s per 200 papers) floors the job at
~85 minutes. On T4 encoding hides inside that window. On CPU it is ~13.5 hours,
over Kaggle's 12-hour session cap.

### Phase 7.2 — Ingest (scope set by 7.0)
`scripts/ingest_arxiv.py`, checkpointed per batch and idempotent. Run in slices,
checking collection status between them. Validated already: arXiv fetch, parse,
pagination, `totalResults`, and the Qdrant id-collision guard (highest live id
1,596,586 — new points continue above it).

### Phase 7.3 — Abstract backfill
New rows arrive with full abstracts automatically. The existing 1,437,656
truncated rows need re-fetching: ~7,200 requests at the arXiv rate limit ≈ **7
hours**, no GPU required. Turso and the sidecar only; vectors untouched.

### Phase 7.4 — Sidecar rebuild
`scripts/build_metadata_sidecar.py`, then re-upload to
`siddhm11/researchit-metadata`. Not urgent: `local_meta` falls back to Turso on
a miss, so newly ingested papers work immediately without it.

### Phase 7.5 — Re-embed on full abstracts (optional, later)
Only worth doing once 7.3 lands. Would improve retrieval but requires
re-encoding 1.6M papers and doubles peak Qdrant disk during the swap. Defer.

---

## 6. Risks

| Risk | Mitigation |
|---|---|
| Disk exhaustion mid-ingest | Run in slices; script is checkpointed and idempotent, so a failed write loses nothing |
| Zilliz free-tier cap unknown | Watch after the first slice; it is the least characterised store |
| Segment merges spike disk transiently | `max_segment_size=100,000` → ~3 new segments; merges write before dropping, so peak exceeds steady state |
| New vectors inconsistent with old | Embedding text and `max_length` match the original ingest exactly; verified in `ingest_backends.py` |
| Point-id collision overwriting papers | Upserter binary-searches for the highest live id and continues above it |
| Rollback | New points occupy a contiguous id range above 1,596,586 and are deletable by `arxiv_id`; no existing point is modified |

---

## 7. Open questions

1. **True Qdrant disk usage** (§4.2) — blocks everything.
2. **Zilliz free-tier storage cap** — not queryable via API.
3. **Ingest cadence after backfill** — GitHub Actions (a keepalive cron already
   exists) or HF Jobs. ~500 papers/day steady state.
4. **Citation counts for new papers** — ingested as 0. Both the reranker and the
   citation boost lean on this, so new papers are systematically disadvantaged
   until a Semantic Scholar enrichment pass runs. `S2_API_KEY` already exists
   for offline scripts.