Spaces:
Running
Running
Commit History
Today
-
data: ship the sidecar with October's papers, the re-harvest and fresh citations (0d1449d) Running
siddhm11 +1: Claude Opus 5.5Keyword search and the starter feed read the sidecar baked into the image, which still held 1,853,777 papers ending 2026-09-29. The new one holds 1,972,571, exactly Turso's: the 26,918 papers the daily job stored since, and the 91,876 re-harvested for June 2025 - September 2026, with citation counts refreshed for every paper since June 2025 (373,718 found on Semantic Scholar). Checked before pinning: SQLite quick_check and the FTS5 integrity check pass, FTS rows = papers = 1,972,571, every paper has its category rows, and keyword search finds "Kimi K3" and "Looped World Models". The published file's size (3,261,734,912 bytes) and SHA-256 match the built one, and its link serves it in full. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>ff196f8 -
docs(06): trending v2 deployed; the search step-up goes straight to on
siddhm11 +1: Claude Opus 5.5A shadow replay of 99 searches (84 logged, 15 on this week's trending subjects) moved no result; four already-ranked trending papers would be tagged and two auditions stayed out below half the best score. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>82e7d11 -
fix(trending): /trending says so when topics could not be worked out
siddhm11 +1: Claude Opus 5.5With Turso unreachable the page claimed topics were "being worked out" for as long as the outage lasted. It now says they can't be worked out right now; the week's papers, which come from Hugging Face directly, still show. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>a572e3d -
feat(health): /healthz/runtime reports the trending topics job
siddhm11 +1: Claude Opus 5.5Its switch, when it last ran, how many papers and topics it used, how many rise and cool, and its last error, beside the shelf's own status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>178a203 -
fix(trending): /trending shows the week's top 9, and one rule under its heading
siddhm11 +1: Claude Opus 5.5Ten cards in the three-column grid left one alone on the last row; nine fill three rows. The first section's top rule doubled the heading's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>2c3ec3c -
fix(trending): name a one-word topic in the plural only when titles do
siddhm11 +1: Claude Opus 5.5"Robot" read oddly as a topic, but titles use it mostly as an adjective ("robot learning": 70 to 12 against "robots" on 24 Oct), so the half-as-often rule never chose the plural. A one-word name now takes the plural when titles use it a tenth as often as the commonest form: Robots and Skills, but Memory, not "Memories" (7 to 220). Longer names keep the half-as-often rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>af7627c -
docs(06): trending v2 — why a paper trends, rising topics, a few days' step-up in search
siddhm11 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>cba4898 -
feat(trending): rising topics from 26 weeks of Daily Papers, and a /trending page
siddhm11 +1: Claude Opus 5.5Once a day, trending_topics clusters the last 26 weeks of lists the daily job stores in Turso (hf_daily_papers, PR #56) the way §3.3 clusters a reader's saves: Ward on L2-normalised vectors from Qdrant, K = 24, medoids, and topic ids kept across days by stabilize_cluster_ids. A topic is rising or cooling only when the slope of its weekly share beats the same slope with every paper's week shuffled 2,000 times, Benjamini-Hochberg q < 0.05, and it is shown only under a phrase from its papers' titles, keywords and summaries that 30% of them carry. Phrases never cross a field; no model writes names (doc 06 defers LLM cluster summaries to Phase 8). On the 24 Oct data (5,176 papers): 30 s a day on a laptop, ids stable across recomputes, rising today: Skills, Robot, World Models. /trending lists the week's papers in the reader's interests, the rising and cooling topics with their weekly-share lines and top papers, and the week's most upvoted, with a footnote on the method; its cards log expo…e51c693 -
feat(search): trending papers step up in search for a few days (SEARCH_TRENDING)
siddhm11 +1: Claude Opus 5.5A paper in the top quarter of this week's Daily Papers that retrieval already found in its first 50 may take one of the last 2 places the cross-encoder scores. If it scores at least half the best, it gains up to 0.25 x the best score, in full for its first 3 days on the list and fading to nothing at 7; at most 2 papers per search. A moved result is tagged "Trending on Hugging Face" with its strongest fact; the classic card is byte-for-byte unchanged otherwise. Switch SEARCH_TRENDING: off (default), shadow (search_meta.trending_shadow records each would-be move, from and to, and nothing changes on screen) or on. No preview: search results are cached for everyone. Doc 06, 2026-10-24. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>d3ea811 -
feat(trending): each trending card says why, in counts from the lists
siddhm11 +1: Claude Opus 5.5Every shelf card now carries its place on the day it was listed ("#1 of 92 papers on 30 Sep") and, in the top quarter of a week of 20 or more, its place in the week ("top 1% of this week's 404"); relative places because the lists ran from 101 to 404 papers a week and upvotes are heavy-tailed (median 13, p99 271 over 26 weeks). Hugging Face's one-line summary is shown labelled as theirs, stars are shortened (5.7k), and comment counts are never shown: nearly every paper has one or two. The card moves to partials/trending_card.html so /trending can reuse it; its topic line and the shelf's /trending link render only once a later change attaches a topic. rank() and shelf_for() take a size, week_top() lists the week's most upvoted, and trending_now() hands search the week's papers with their freshness (full for 3 days, gone at 7). Doc 06, 2026-10-24. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>3c99209 -
fix(trending): read every page of a Daily Papers list, and what HF says about each paper
siddhm11 +1: Claude Opus 5.5The Daily Papers API serves at most 100 papers a page and stops at 50 without a limit, which cut 8 papers from 22 Oct 2026's 58. fetch_snapshot now asks for pages of 100 until a short one, and also keeps each record's keywords, one-line summary and comment count (`extra`) for the shelf's cards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>98d4c93
Yesterday
-
feat(search): a rewritten question searches keywords in the rewrite's words
siddhm11 +1: Claude Opus 5.5When the rewriter changes a query and no paper contains every word of the original (keyword search fell back to matching any word), the original's keyword list is left out of RRF. Its dense list and the exact-title boost stay. Keyword results now carry match="all"|"any" so the search can tell a question from a title. Switch SEARCH_REWRITE_KEYWORDS, off by default. Measured on the 79 distinct logged searches (top 10, blind 0/1/2 grades, rewrites pinned): 12 change, 124 -> 138 points, 6 better, 1 worse. All 9 known titles stay first. Without the guard the DPO paper fell out of the top 10, because the rewriter shortened its title. Rejected after testing, in doc 06: requiring the rarest words (LitSearch keyword top-10 33.9% -> 11.1%), phrases for hyphenated terms (neutral), and topping up short every-word lists with any-word matches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b7cdd94 -
feat(feed): trending in your interests, a shelf of Hugging Face Daily Papers
siddhm11 +1: Claude Opus 5.5Readers in the clustered tier get up to 6 papers from Hugging Face Daily Papers (last 7 days, refreshed every 3 hours) in one block after the fourth card of page 1, loaded once the page is on screen. A paper qualifies when it is as close to one of the reader's interests as the 25th percentile of their own saves (clamped 0.55-0.70; 0.60 below four measurable saves); the order is closeness + 0.07 * log10(upvotes), each interest gets max(2, ceil(6/K)) places first, and fewer than 3 qualifying papers means no shelf. The shelf holds no feed slot, so quota, MMR and the served order are untouched (doc 06, 2026-10-23). Its cards carry their own query_id, propensity 1.0 and policy trending_shelf_v1 through links, Save and the exposure log; what it shows counts as shown for the feed and the next shelf. Interest names come from the feed page it sits on. Vectors come from Qdrant through the shared cache, so nothing is embedded on the Space. Switch TRENDING_SHELF (off by default); /healthz/runtim…727c02a -
fix(search): let long questions reach the query rewriter
siddhm11 +1: Claude Opus 5.5_looks_academic skips the rewrite when a query has more than six words and two "academic" matches. The acronym branch [A-Z]{2,} was compiled with IGNORECASE, so every ordinary word of two or more letters counted, and no query over six words has been rewritten since Phase 3 (d5a6f3e). "how do I stop a RAG system from making things up" counted 9 acronyms. The acronym branch is now case-sensitive; the topic words stay case-insensitive. Measured locally before and after on the 15 real search_log queries the change affects (top 10, graded 0/1/2 blind to which version found each paper): 171 -> 197 of 300; 4 better, 2 slightly worse, 9 unchanged. The RAG question goes from 2 to 15. Seven long paper titles still find their paper at #1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>d201982 -
feat(onboarding): describe your work in a sentence and get its areas picked
siddhm11 +1: Claude Opus 5.5A newcomer often can't map their work onto arXiv's area names. Step 1 now opens with "In a sentence": the description is searched (hybrid search with no LLM rewrite, so it reaches no model, and it is not stored), and the areas its top papers sit in are picked for the reader to check, with the evidence: "Picked Robotics, Computer Vision from 10 matching papers, like 'DexVIP: ...'". Picked areas carry a "from your words" tag, suggestions only add to what is already picked, and step 2 opens on papers for the sentence instead of popular ones. Doc 06 cites language-based preferences as competitive near cold start (Sanner et al. 2023). A paper's primary category counts 1 and a cross-list 1/2: cs.LG is cross-listed on so much that, counted fully, Machine Learning would be suggested for most descriptions. Probed on 12 descriptions with word-match search only (no BGE-M3 locally), 11 got sensible areas; "I like space stuff" matched COCO-Stuff. Logged in doc 06; CURRENT-STATE and CLAUDE.md updat…4c1781d -
feat(onboarding): a drawing of each research area on its card
siddhm11 +1: Claude Opus 5.5An area's name alone often means nothing outside the field: "Machine Learning" or "Condensed Matter" tells a newcomer little about what they would be choosing. Each of the 20 cards on /onboarding and /interests now shows the field doing its typical thing: a detector's box round a cat for Computer Vision, a line fitted to points with y = wx + b for Machine Learning, a question answered step by step for AI, a Feynman diagram, a Bell-pair circuit, the chord rule on an elliptic curve. The drawings are inline SVG written by scripts/build_area_art.py; the plotted ones (the curve, the helix, flow round a cylinder, the spectrogram) are computed. They take the page's colours in both themes and turn green with a picked card, and hover redraws the one annotated stroke (off under reduced motion). They are decorative to screen readers: the name and description still carry the meaning. Not images, because Hugging Face refuses binary files in pushed history. About 10 KB gzipped for all twenty. Co-A…ec199d7 -
fix(search): the query shows in the box, not as the heading
siddhm11 +1: Claude Opus 5.5The search page printed the query as its heading, in the box and in the result count ("10 results for ..."), and the empty state quoted it again. b804177 kept the heading in step with a search from the box: that fixed a stale heading but kept the repetition. The heading is now "Find papers", naming the page like every other masthead, and the count and the empty state no longer quote the query. The tab title still names it, so tabs and the browser's history tell searches apart. With the box the only place on the page that names the query, it has to be right after Back and Forward. htmx restores those from saved HTML, which holds the value the page was served with, not what was typed: Forward to a search run from the box showed its results under the first query. app.js now sets the box from the URL when htmx restores a page; the browser test fails on that line without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>d8d4da6 -
perf(feed): fetch candidate vectors and details side by side
siddhm11 +1: Claude Opus 5.5The multi-interest build fetched its candidates' vectors (Qdrant, in Europe) and then their details (the sidecar, else Turso, in Mumbai), one after the other, though neither needs the other. They now run together and the build waits for the slower one. The feed is unchanged: a test checks the same ranked list and tags whichever fetch finishes first. Measured 2026-10-23 from a laptop in India against the live services, 12 saves in 3 interests, four fresh processes each: vectors about 0.85 s, details 0.2-0.47 s, build 1.9-2.2 s either way; side by side the vector fetch slowed under contention, so the saving there was small and noisy. The Space is the case that matters: its own probe takes 978 ms for SELECT 1 to Turso (112 ms from the laptop), so there the details are likely the long half. feed_tier1 now reports both halves (cand_vectors_ms, cand_details_ms) next to their wall time (candidate_meta_ms), so the saving can be read off production. Co-Authored-By: Claude Opus 5.5 <noreply@an…6edee5d -
perf(feed): time every stage of the multi-interest build and the served page
siddhm11 +1: Claude Opus 5.5On the Space a first feed page for a reader with saves took 6.9 s (build_ms), and the stage timings then recorded covered 3.9 s of it: ann_retrieval 2.0 s, candidate_meta 1.8 s, clustering, MMR. The rest had no key, and candidate_meta folded two network fetches (candidate vectors from Qdrant, then their details from the sidecar or Turso) into one number. feed_tier1 now also records saved_vectors_ms, cluster_search_ms and st_search_ms (the two halves of ann_retrieval), cand_vectors_ms and cand_details_ms (the two halves of candidate_meta), features_ms, rerank_ms, exploration_ms and total_ms. A served page records feed_first_page or feed_next_page: page_ms, log_ms (impressions and exposures), cards_ms and request_ms, what the reader waited for. Timing only: the feed is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>f23a740
2 days ago
-
fix(search): the query rewriter never answers the reader
siddhm11 +1: Claude Opus 5.5On 2026-10-22 "HI HOW ARE YOU >" was rewritten as "I'm doing well, thank you! How can I help you today?", which was then searched beside the query and shown as "searched as". The prompt gave no rule for text that isn't a research topic, so the chat model answered it; the old prompt reproduces the reply word for word, and turns "hey whats up" into "general conversation". The prompt now says the text is a query, never a message, and a greeting comes back as is (Qwen: all four greetings tried came back unchanged, rewrites of real queries as before). A rewrite that talks back (you/your/thanks/I'm not in the query, a ? or ! the query didn't have, or over 12 words) is discarded; of the 39 rewrites logged in production only that one is caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>768a4ff -
fix(search): a search from the box renames the heading and the tab
siddhm11 +1: Claude Opus 5.5The search box swaps in only #search-results, so the page heading and the tab title kept the query the page was opened with: on 2026-10-23 the page read "how do I stop a RAG system from making things up" above the results for "HI HOW ARE YOU >". The htmx answer now carries the heading as an out-of-band swap and a <title> that htmx applies as the swap settles. The whole page is unchanged and has one heading. A browser test runs a second search from the box and checks the heading and title, then Back and Forward. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b804177
3 days ago
-
fix(briefing): drop an over-long point instead of failing the whole reply
siddhm11 +1: Claude Opus 5.5In the first 25 papers of the b2 evaluation (2026-10-22), 24 briefings passed and one failed outright: 2609.31119's first point ran to 404 characters against the 400 limit, and the reply was thrown away with its two good points. A point or step over its limit is now dropped like one that fails the number check, and the over-long use line is recorded as dropped. Missing or non-text fields still fail the reply. The limits are unchanged: the other 66 points had a median of 220 characters and a longest of 326. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>721aff2 -
docs(06): log the briefing passage and prompt change
siddhm11 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>a7c3c90 -
fix(briefing): send every section's opening and decline only when the paper says nothing
siddhm11 +1: Claude Opus 5.5In production the model declined 5 of the 15 papers it wrote for with {"insufficient": true}, all with 16-85 paragraphs; 4 of 5 reproduced. Passages were chosen by section title, so papers that name sections by topic ("Backbone", "Iterative Attention", "Cognitive Prompting") were sent only their introduction, and 3 paragraphs per top-level section sent 2608.11967 its experimental setup but not "Main Result Analysis". Now the picking is by place in the paper: abstract, introduction and conclusion, the first paragraph of every section and subsection, captions, then later paragraphs in rounds, the appendix last; never related work, references or front matter; sent in page order. The prompt asked for "the paper's numbers" behind a reply that opened on "insufficient", which let the model decline whenever no results were sent. Prompt b2 asks for numbers when the passages state them, counts a theory or survey paper's claims as its findings, and declines only when the passages don't say what…3223eb2 -
docs: bring the current state, the data procedure and the plan to 2026-10-21
siddhm11 +1: Claude Opus 5.5- CURRENT-STATE: what the Space runs (36c792e), corpus and citation numbers after the catch-up, the daily job, which tables reset on every deploy, local text fetched again, the paper page's back link. - PHASE7 §8.1: the daily job's stages, inputs, the four catch-up runs, and the rules learned (throughput, one run at a time, schedule delay, capacity runway on cluster 2). - Redesign plan: build-order statuses for all ten PRs, a dated status note under the next steps, and traps T19-T23 for the next agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>389e4d9 -
docs(06): log the harvest, briefing-text and back-to-search decisions
siddhm11 +1: Claude Opus 5.5Three changelog entries for 2026-10-20/21: - daily papers through arXiv OAI-PMH, licences in their own Turso table, with the measured runner throughput and the 23,098-paper catch-up; - a paper's local text is fetched again, and briefings are keyed on paper version and prompt (why not the passages sent); - Back to a search: Vary on htmx URLs, a true back link, and recent results in memory, including the query_id rule for a reader's repeat and why clearing search_log.query_id was dropped. CLAUDE.md: corpus numbers (arxiv_recent 279,778; Turso 1,876,875 to 2026-10-20), the search cache rule, two integrity rules (a replicated row doesn't mean local data is here; one URL, two bodies needs Vary), the query_id note in 3.11, the test count and the dates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>4986928 -
Merge origin/main into docs/session-2026-10-21
siddhm11Brings the harvest (#24, #41, #42) and the libsql fix (#45) under the redesign stack so the session's docs can be written against both.5938f71 -
feat(search): keep recent results in memory so going back doesn't search again
siddhm11 +1: Claude Opus 5.5Going back to a search ran the whole pipeline again: a BGE-M3 encode, a Groq rewrite, Qdrant and FTS, a metadata fetch, 1-5 s on the Space, and a second search in the analytics. app/search_cache.py keeps a query's ranked results in memory for SEARCH_CACHE_TTL_S (600 s; 120 s when a retrieval arm was missing, since a later run may do better), keyed by the query with case and spacing ignored, at most 256 queries. - Back, the paper page's back link, a reload and another reader's identical search are served from memory. - A reader coming back to the same results keeps the query_id they were shown (clicks stay tied to that search, section 3.11) and is not logged to search_log or counted in reader_days again. The reader-to-query link is memory-only and dies with the entry. - Callers get copies, so one reader's saved state or query_id never reaches another's results. Empty results are not kept. SEARCH_CACHE_TTL_S=0 turns it off. - /healthz/runtime counts search_cache_hit and sear…36c792e
4 days ago
-
fix(paper): the back link returns to where the reader came from
siddhm11 +1: Claude Opus 5.5The paper page showed "← Your feed" (to /) whenever the link carried a query id, and search results carry one too, so a paper opened from a search sent the reader to the feed. The link now follows the same-origin Referer to a known page: a search keeps its query (only `q`), and the feed, saved papers, recently opened, a collection or the previous paper get their own label. Without a usable Referer the click's surface decides; a shared link still gets no link. The browser test covers both halves: search, open a result, press Back, then use the page's own link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>ba6d91e -
fix(search): Back to a search shows the whole page, not a bare fragment
siddhm11 +1: Claude Opus 5.5/search?q=... answers htmx with the results alone and a browser with the whole page, and nothing told the browser the two differ. After search -> open a paper -> Back, the browser could show the cached results fragment as the page: no header, no styles, no title (reproduced in Chromium against the sealed test world). - Every non-static response now carries Vary: HX-Request, and htmx fragments are sent with Cache-Control: no-store. - htmx restoring history without a snapshot (HX-History-Restore-Request) gets the whole search page; it swaps the answer in as the body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>eeef7b6 -
Merge pull request #45 from siddhm11/fix/ingest-libsql-url
siddhm11Unverified signaturefix(ingest): accept libsql:// Turso URLsf894733 -
fix(ingest): accept libsql:// Turso URLs
siddhm11 +1: Claude Opus 5.5Turso hands out libsql:// URLs, which urllib can't open. daily_refresh.py and refresh_citations.py already convert them to https://, but ingest_arxiv.py and ingest_backends.py did not, so the daily job worked only because the TURSO_URL secret was entered as https://. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>3e90cc0 -
fix(ask): fetch the paper's text before answering; don't cache abstract-only
siddhm11 +1: Claude Opus 5.5Ask read the same local-only text as the briefing. After a deploy it answered from the abstract alone, and a suggested question's answer was then cached for every reader. It now fetches the text first when this container lacks it, does not cache an answer written without it when the paper has paragraphs, and skips such a cached answer if one exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>ce7c67e -
fix(briefing): no "too few passages" for missing text; never downgrade
siddhm11 +1: Claude Opus 5.5After a deploy a briefing for a paper read earlier found only the abstract and stored "insufficient: too few passages" for good. Seen in production: 2508.16989 (61 stored paragraphs) and 2410.17725 (43). - The briefing fetches the paper's text again when it is missing here. If that fails, nothing is stored and a later visit retries (counted as no_text on /admin/ai). - A stored "too few passages" row is ignored when the source records enough paragraphs, so the two broken briefings repair themselves. - The cache key is paper + arXiv version + prompt. The parser version and model came out: citations link to arXiv's own anchors on a fixed version, so neither changes what a stored briefing says, and the parser bump to 2 had thrown good briefings away (2608.11967 went from ok to insufficient). Rows stored under the old key are still found. - A worse attempt never replaces a good briefing of the same version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>5771df0 -
fix(sources): fetch a saved paper again when its local text is gone
siddhm11 +1: Claude Opus 5.5paper_sources is replicated, but paper_fulltext and paper_paragraphs are local: a deploy (or prune past FULLTEXT_MAX_PAPERS) drops the text while the restored row still says "ok, current", so ensure() never fetched it again. has_text() says whether this container holds the text, and ensure(need_text=True) fetches when it doesn't. Opening a paper now asks for the text, so it is back before the briefing or Ask needs it; figure reads and the warm-up keep needing only the row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>2d52647 -
Merge pull request #44 from siddhm11/feat/paper-briefing
siddhm11Unverified signaturefeat(briefing): three things to know, how it works, where you'd use itadd0458 -
Merge pull request #43 from siddhm11/feat/paper-page-v2
siddhm11Unverified signaturefeat(paper): the redesigned paper page behind PAPER_PAGE_V25e6972b -
Merge pull request #42 from siddhm11/fix/daily-catchup-timeout
siddhm11Unverified signaturefix(daily): let catch-up runs raise the ingest limit; job ceiling 5 h304ac98 -
fix(daily): let catch-up runs raise the ingest limit; job ceiling 5 h
siddhm11 +1: Claude Opus 5.5The first catch-up run (2026-10-20) hit the 60-minute ingest limit after 2,414 papers: a GitHub runner encodes about 2,400 papers an hour, and about 15,000 were missing. A new `ingest_timeout_min` input (default 60, so scheduled runs are unchanged) is passed through env, and the job ceiling rises from 120 to 300 minutes. The ingest limit and per-call network deadlines remain the stall guards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>2007bc5 -
Merge pull request #41 from siddhm11/feat/daily-harvest-licences
siddhm11Unverified signaturefeat(daily): harvest through arXiv OAI-PMH and keep each paper's licencea04d4b8 -
Merge remote-tracking branch 'origin/main' into feat/daily-harvest-licences
siddhm11#24 is now on main; this leaves the branch one merge base with it. No file changes.e2f2ae0 -
Merge pull request #24 from siddhm11/fix/ingest-lazy-window-split
siddhm11Unverified signaturefix(ingest): lazy window splitting, network deadlines, rate-limit back-offb19350d -
feat(daily): a `today` input for catching up in short windows
siddhm11 +1: Claude Opus 5.5The newest stored paper is from 2026-09-29. A single run with a 22-day lookback would encode every missed day at once on a CPU runner and overrun the ingest's 60-minute limit. Manual runs can now set `today` to a past day (passed as --today) and step it forward. The value reaches the script through env, never pasted into the shell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>0e6b4bd -
feat(daily): harvest through OAI-PMH and report licence coverage
siddhm11 +1: Claude Opus 5.5stage_ingest passes --source oai, so the daily job uses arXiv's OAI-PMH feed instead of per-category search queries. The summary gains a licences block: new papers with a licence row, how many are open, and how many may show figures under FIGURE_LICENCES. Reported only; it never fails the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>c36035c -
feat(ingest): keep each harvested paper's licence in Turso paper_licences
siddhm11 +1: Claude Opus 5.5parse_oai_record reads <license> into licence_url and a short code (app.licences.code_from_url). New papers' licence rows go in the same Turso pipeline as their papers rows; papers already stored get theirs when OAI lists them again. Own table, never a papers column: INSERT OR REPLACE rewrites whole papers rows (trap T5). The upsert skips rows that already say the same, so daily re-harvests write nothing. Dry run over 2026-10-16: 1,833 papers kept, all with a licence (ARXIV-NONEXCLUSIVE 865, CC-BY-4.0 773, CC-BY-NC-ND 79, CC-BY-NC-SA 67, CC-BY-SA 36, CC0 13). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>da423eb -
Merge remote-tracking branch 'origin/main' into feat/daily-harvest-licences
siddhm1138e1da6 -
feat(ingest): harvest new papers through arXiv OAI-PMH
siddhm11 +1: Claude Opus 5.5arXiv's OAI-PMH interface is built for incremental harvesting: everything changed between two dates, a whole archive per request stream, about 1,300 records a page. A day across our seven archives is about 10-15 requests, where the search API needs one query per category (about 100), which is what got the daily run throttled on 2026-10-01. --source oai harvests each archive set holding our categories, keeps new- style ids from the month before --since (OAI's <created> is not the first version's date for revised papers), and reports a set complete only when its last page carries no resumption token (exit 3 otherwise). Text is NFKC- normalised and arXiv's alias categories collapsed as the search API does: 901 overlapping papers from 2026-09-29 matched rows written from the search API exactly. Requests identify themselves; a 406, like a 403, 429 or 503, is a temporary block to wait out, and other 4xx answers fail at once. Written 2026-10-01 and left uncommitted; committed unchanged. Co…3f01cfe -
docs: record reader and search analytics and how queries are kept
siddhm11 +1: Claude Opus 5.5CURRENT-STATE gains a reader and search analytics row and lists the three new replicated tables. Doc 06 logs the decision: searches logged, query text kept 90 days and never stored against a reader, the owner's own use flagged and left out, with the production numbers that made it necessary (321 of 335 saves from search, no search ever recorded; 7 of 131 readers came back). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>c89cdb2 -
feat(admin): readers, searches and AI history dashboards
siddhm11 +1: Claude Opus 5.5Three owner-only pages next to AI usage, with tabs across all four: /admin/readers readers today, this week and month, the share who came back, what they did, new and returning readers per day, and readers per month since launch with where saves came from /admin/searches searches per day with and without results, the share that led to an opened or saved paper, the most searched queries, queries that found nothing, and the latest searches with their AI rewrite and overview /admin/ai/history every AI call by paper (title, plain summary, briefing outcome, tokens) and by search (rewrite, overview) The owner's own use is left out unless 'Include my own use' is chosen. Model-written text is shown as text, never markup. Every chart now comes from one builder with one tooltip script: whole-number ticks for small counts, about seven l…e8530b8 -
feat(analytics): reports on readers, searches and AI history
siddhm11 +1: Claude Opus 5.5readers_report: readers per day, new against returning (a reader's first day is the earlier of their first action and first counted day, so readers from before counting began aren't new), readers today and over 7 and 30 days, what share searched, opened, saved and asked, and the long view from every recorded action since launch: readers per month, how many days they came back, where saves and opens came from. The owner's browsers, known from reader_days, are left out unless asked for. searches_report: searches per day, the share that found nothing, fell back to keyword search, were rewritten, or led to an opened or saved paper (through the query_id their results carry), the most searched queries, queries that found nothing, and the latest searches with their rewrite and overview. ai_history: every AI call grouped by its paper (title, calls per feature, failures, tokens, the latest plain summary and briefing outcome) or its search (rewrite and overview). Co-Authored-By: Claude Opus 5…a7c16d8