PROJECT_LOG.md

ResearchPath — Project Log

A chronological record of decisions, work sessions, issues, and outcomes for the ResearchPath portfolio project.

Schema for each entry:

## YYYY-MM-DD HH:MM — [type] Title
**Type:** setup | decision | implementation | issue | resolution | discussion
**Status:** in-progress | complete | blocked | reverted
**Duration:** approx time

### Context
### Decisions
### Actions taken
### Issues encountered
### Files touched
### Outcome
### Next

---

2026-05-06 15:00 — [discussion] Project framing & scope

Type: decision Status: complete Duration: ~30 min

Context

Chetan (~2.5 YOE Data Scientist, Kyndryl + ex-Swiss Re, targeting top product companies) needed a portfolio project. Earlier "Agentic RAG Document Search" on his resume was copied from another GitHub — needed a genuine, defensible project.

Decisions

• Rejected Underwriter Copilot (insurance) and Equity Research Analyst (SEC filings) — too generic / didn't match his preference for research-flavored work.
• Rejected generic-RAG-over-papers framing — already exists (Perplexity, Elicit, Consensus); not enough differentiation.
• Committed: ResearchPath — agentic research-onboarding companion. Killer feature: prerequisite-chain reading-path planning. Genuinely agentic (graph traversal + recursive retrieval + reasoning over user state), not RAG-with-extra-steps.
• Demo domain: Reinforcement Learning. Architecture domain-agnostic.
• Stack: Gemini 2.5 Flash Lite + Groq Llama 3.3 70B (free tiers), BAAI/bge-small-en embeddings, FAISS, LangGraph, Streamlit, RAGAS.
• Eval-first methodology: gold dataset + ablation table is the differentiator from generic portfolios.

Outcome

Project scope locked. 5-week plan in README.

Next

Day 1 setup: env, deps, API keys, smoke test.

---

2026-05-06 15:20 — [setup] Day 1: Environment bootstrap

Type: setup Status: complete Duration: ~45 min

Context

Stand up local dev env so the agent work in Week 2+ has a working LLM substrate.

Decisions

• Use uv for Python + dependency management (already installed on user's machine).
• Default Gemini model: gemini-2.5-flash-lite (chosen after gemini-2.5-flash returned transient 503s during smoke test).

Actions taken

• Scaffolded repo files: .gitignore, .env.example, README.md, pyproject.toml, researchpath/__init__.py, researchpath/llm.py, scripts/smoke_test.py.
• uv python install 3.12 → installed CPython 3.12.9.
• uv sync → resolved 38 packages including google-genai, groq, python-dotenv, rich, pytest, ruff.
• User created .env from template and pasted Gemini + Groq API keys.
• uv run python scripts/smoke_test.py → both providers returned valid completions.

Issues encountered

• Unicode arrow → broke on Windows cp1252 codepage when rich tried to write to stdout. Fix: removed the arrow from scripts/smoke_test.py. Also set PYTHONIOENCODING=utf-8 for the run.
• Gemini 2.5 Flash returned 503 UNAVAILABLE (transient overload) despite a valid key. Fix: switched default model in researchpath/llm.py from gemini-2.5-flash to gemini-2.5-flash-lite.

Files touched

• Created: .gitignore, .env.example, README.md, pyproject.toml, researchpath/__init__.py, researchpath/llm.py, scripts/smoke_test.py, .claude/settings.local.json, PROJECT_LOG.md.
• User-created (gitignored): .env.

Outcome

Both Gemini and Groq APIs verified end-to-end. .claude/settings.local.json grants Bash + PowerShell auto-approval for this project (gitignored — personal preference, not committed).

Next

Day 2: git init → arxiv corpus fetch script → download ~17 canonical RL papers → first commit.

---

2026-05-06 16:00 — [implementation] Day 2: Corpus pipeline + first commit

Type: implementation Status: complete (with one outstanding follow-up — see "Outcome") Duration: ~75 min

Context

Build the end-to-end corpus path: fetch canonical RL papers from arXiv → parse PDFs → chunk text into a single JSONL ready for embedding. Land everything as the first git commit.

Decisions

• 17-paper canon chosen across 5 eras (value-based, policy-gradient, actor-critic, model-based, RLHF). See researchpath/corpus.py.
• Streaming downloads via requests instead of arxiv library's built-in downloader — the latter was truncating large PDFs at exact 1MB / 2MB boundaries on Windows.
• PDF validation step (open with pymupdf, read page 1) gates every download; truncated files auto-deleted and retried.
• Chunking: 800-char windows, 100-char overlap, sentence-boundary aware, hyphenation-aware.
• Accept partial corpus (10/17) rather than grind on rate-limited retries — script is idempotent so missing papers can be picked up later.

Actions taken

• Added deps: arxiv, pymupdf. uv sync resolved 5 new packages.
• researchpath/corpus.py — 17-paper canon with era tag.
• scripts/fetch_corpus.py — streaming downloader with per-paper validate-and-retry, 12s inter-request spacing, 30s backoff.
• researchpath/parsing.py — PDF text extraction + sentence-boundary chunking.
• scripts/parse_corpus.py — drives parsing across all valid PDFs into data/chunks.jsonl.
• Ran fetcher: 10 papers landed cleanly. Deleted 3 truncated stragglers (1801.01290, 1911.08265, 2301.04104).
• Ran parser: 1,093 chunks, 778,995 chars across 10 papers.
• git init -b main → first commit 270993c ("Bootstrap ResearchPath: scaffold + corpus pipeline"). 13 files, 1,611 insertions.

Issues encountered

• arXiv HTTP 429 rate-limiting on PDF endpoint. Tried 3s→8s→12s inter-request delays; eventually too much accumulated state on this IP. Pivoted to "accept partial corpus, document the gap" per user's feedback_dont_grind.md.
• arxiv library truncates large PDFs at 1MB/2MB even on retry — buffer issue specific to its downloader. Switched to direct requests streaming. (Confirmed by post-hoc validation: 3 stragglers stuck at exactly 2,097,152 / 1,048,576 bytes.)
• Unicode → in console output broke on cp1252 (already known from Day 1; reused PYTHONIOENCODING=utf-8).

Files touched

• Added: researchpath/corpus.py, researchpath/parsing.py, scripts/fetch_corpus.py, scripts/parse_corpus.py.
• Modified: pyproject.toml (+arxiv, +pymupdf).
• Generated (gitignored): data/papers/*.pdf (10 files), data/chunks.jsonl.

Outcome

End-to-end corpus path works: arxiv ID → validated PDF → chunked JSONL. Repo under git, first commit landed.

Outstanding: 4 papers still missing — 1511.05952 (PER), 1707.06347 (PPO), 1802.01561 (IMPALA), 2106.01345 (Decision Transformer). PPO is the painful gap (canonical paper). Re-run uv run python scripts/fetch_corpus.py later (script is idempotent + only retries missing/invalid).

Next

Day 3: embeddings + FAISS index over data/chunks.jsonl. Use BAAI/bge-small-en-v1.5 (CPU-friendly, ~133MB, strong on academic text). Then a tiny CLI: ask question → top-k chunks → grounded Gemini answer. That's the baseline RAG to measure against.

---

2026-05-06 17:00 — [implementation] Day 3: Embeddings, FAISS index, baseline RAG

Type: implementation Status: complete Duration: ~50 min

Context

Stand up the first measurable RAG system: chunks → embeddings → vector search → grounded LLM answer with citations. This becomes the baseline that all future improvements (hybrid retrieval, reranker, agent planning) get measured against.

Decisions

• Embedding model: BAAI/bge-small-en-v1.5 (384 dim, ~133MB, CPU-fast, strong on academic text). Documents embedded raw; queries prefixed with the BGE-recommended retrieval instruction for measurably better results.
• FAISS IndexFlatIP with L2-normalized embeddings — exact cosine search, fast enough for ~1k chunks. Will swap to HNSW if corpus grows past ~50k.
• Citation format [arxiv_id, p<page>] — parseable by future eval code, human-readable in output.
• System prompt is strict: only-from-sources, no speculation. Will tighten further once we have eval numbers.

Actions taken

• Added deps: sentence-transformers, faiss-cpu, numpy. uv sync resolved 29 new packages (~330MB inc. torch CPU).
• researchpath/embeddings.py — Embedder wrapper with separate embed_documents / embed_query (BGE wants different prefixes).
• researchpath/index.py — build_index, load_index, search. Hit dataclass carries score + citation metadata.
• researchpath/rag.py — build_prompt, answer. Strict system prompt enforcing citations + grounding.
• scripts/build_index.py — drives end-to-end embed + persist.
• scripts/ask.py — CLI for question → answer with citations. --show-sources flag for debugging retrieval.
• Built index: 1,093 chunks embedded in 148s on CPU. Persisted data/index.faiss + data/index.chunks.json.
• Smoke test query: "How does Rainbow combine multiple deep Q-learning improvements like Double DQN, Dueling networks, and prioritized replay?"
  • Answer correctly cited [1710.02298, p1] for Rainbow components and [1511.06581, p4] for Dueling architecture details.
  • Listed all 6 Rainbow components accurately. No hallucination observed.
  • 1,106 input / 196 output tokens (≈ free under Gemini quota).

Issues encountered

• HF Hub symlink warning on Windows — cache-system can't use symlinks without admin/Developer Mode. Cosmetic only; cache works in degraded mode (more disk, no functional impact).
• VIRTUAL_ENV mismatch warning from uv — leftover env var from a prior shell session pointing at a different Python. Harmless; .venv is used regardless.

Files touched

• Added: researchpath/embeddings.py, researchpath/index.py, researchpath/rag.py, scripts/build_index.py, scripts/ask.py.
• Modified: pyproject.toml (+sentence-transformers, +faiss-cpu, +numpy), uv.lock.
• Generated (gitignored): data/index.faiss, data/index.chunks.json.

Outcome

Baseline RAG is end-to-end working and producing high-quality, grounded, cited answers on a real RL question. This is the system we'll measure in Day 4.

Next

Day 4: eval harness + gold dataset. Build ~30 (question, expected answer key, expected source citations) tuples sourced from OpenAI Spinning Up problem sets, Sutton & Barto exercises, and 3-5 hand-curated questions per major paper. Then implement metrics (citation faithfulness, retrieval recall@k, answer quality blind-rated 1-5). First numbers go into the README eval table.

---

2026-05-06 18:00 — [implementation] Day 4: Eval harness + gold dataset + baseline numbers

Type: implementation Status: complete Duration: ~90 min

Context

Build a rigorous eval harness before optimizing anything. Eval-first is the portfolio differentiator — ablation table beats vibes.

Decisions

• 30-question gold dataset across 10 indexed papers (3 per single-paper + 4 cross-paper comparison). Difficulty-stratified: 7 easy / 15 medium / 8 hard. Each example: {id, question, expected_arxiv_ids, expected_key_claim, difficulty}.
• Three metrics (no citation faithfulness per-claim — too expensive on free tier):
  • retrieval_recall@k: fraction of expected arxiv IDs found in top-k hits
  • citation_presence: answer cites at least one expected paper (string check)
  • answer_correctness: LLM-as-judge (Groq llama-3.3-70b), YES/NO prompt
• Judge model: Groq not Gemini — Gemini free tier has 20 req/day limit (hit during initial test run). Groq handles batch eval at no cost.
• RAG model: Groq for batch eval. Gemini stays as default for scripts/ask.py (the demo CLI). Added --groq flag to run_eval.py to toggle.
• Retry logic added to gemini_generate() — parses retry delay from 429 error body, sleeps and retries up to 5 times.

Actions taken

• data/gold_dataset.json — 30 QA triples (hand-authored from Spinning Up + paper reading).
• researchpath/eval.py — GoldExample, EvalResult, EvalSummary dataclasses + evaluate_example, summarize, results_to_json.
• scripts/run_eval.py — full pipeline: load index → batch retrieve + generate → judge → print table + save JSON.
• Updated researchpath/rag.py — added generate_fn injection param + answer_groq() convenience wrapper.
• Updated researchpath/llm.py — retry-with-backoff on Gemini 429 (parses retryDelay from error body).
• Ran full 30-question eval (--groq). Duration: ~3.5 min. Results saved to data/eval_results.json.

Issues encountered

• Gemini 20 req/day free-tier limit — hit on second run (smoke test + partial full run = 20 calls). Fix: route eval to Groq entirely.
• Retry delay detection: Gemini's 429 body contains retryDelay: '10s' in the Details array. Regex extracts it; if missing, defaults to 15s sleep.

Files touched

• Added: data/gold_dataset.json, researchpath/eval.py, scripts/run_eval.py.
• Modified: researchpath/rag.py (generate_fn injection, answer_groq), researchpath/llm.py (retry logic), README.md (eval table row 1).
• Generated (gitignored): data/eval_results.json.

Outcome

Baseline RAG numbers (Groq llama-3.3-70b, k=5, n=30):

Metric	Value
Retrieval Recall@5	90.0%
Citation Presence	86.7%
Answer Correctness	36.7%
Avg Latency	5.07s
RAG Tokens	33,498 in / 7,000 out

Gap analysis: Retrieval finds the right paper in 27/30 cases — the FAISS index is working well. But answer correctness is only 37%, revealing two failure modes:

1. Wrong-chunk problem: Right paper retrieved, but the specific mechanistic claim is in a chunk outside top-5 (e.g., Rainbow's 6 components, distributional RL definition). Fix: hybrid retrieval (BM25 + vector) to catch exact-term hits + smaller chunks.
2. Precision failure: Right chunk retrieved, but answer paraphrases around the key claim without naming the specific mechanism (e.g., "stability" instead of "identifiability" for Dueling DQN mean subtraction). Fix: reranker + more directive prompt.

Difficulty breakdown: easy=28.6%, medium=53.3%, hard=12.5% answer correctness — harder questions show compounding retrieval + precision failures.

Next

Day 5: Optimization round 1 — implement hybrid BM25+vector retrieval and measure improvement on same 30 gold examples. Target: push Answer Correctness from 37% → 55%+.

---

2026-05-06 19:30 — [implementation] Day 5: Hybrid BM25+FAISS retrieval (RRF fusion)

Type: implementation Status: complete Duration: ~90 min

Context

Day 4 baseline showed Retrieval Recall@5=90% but Answer Correctness=37%. Root-cause: right paper retrieved but wrong chunks — dense embeddings de-prioritize specific technical terms ("conjugate gradient", "Polyak averaging", "identifiability") relative to surrounding context. BM25 keyword search catches exact-term hits that dense misses.

Decisions

• BM25Okapi (rank-bm25 library) for sparse retrieval over the same 1,093 chunks.
• Reciprocal Rank Fusion (RRF, k=60) to combine BM25 and FAISS rankings without normalizing incompatible score spaces.
• Stop-word filter in tokenizer: query "What does GAE stand for?" without stop words = ['gae', 'stand', 'fundamental', 'tradeoff'] — much better signal than including "what", "does", "for", "it".
• fetch_k=20 from each retriever before fusion, then top-k=5 from combined candidate set.

Actions taken

• researchpath/retrieval.py — BM25Retriever, HybridRetriever with RRF fusion. Stop-word list added to tokenizer.
• researchpath/index.py — added global_idx field to Hit dataclass (FAISS vector position, globally unique across papers). Fixed critical bug: original code used chunk_idx (within-paper index, 0..N) as RRF dict key, causing collisions across papers with same within-paper index.
• scripts/ask.py — added --hybrid flag.
• scripts/run_eval.py — added --hybrid flag, fixed Path.relative_to() crash on relative paths.
• Ran full hybrid eval (--groq --hybrid): 29/30 questions (cross_04 errored on Groq 100K daily TPD limit).

Issues encountered

• global_idx bug: chunk_idx is per-paper, not globally unique. Two chunks from different papers with the same within-paper index collided in the RRF dict, silently corrupting results. Fixed by adding global_idx (FAISS position) to Hit and using it in HybridRetriever. Pre-fix: hybrid was WORSE than dense in all spot checks. Post-fix: BETTER or TIED.
• BM25 stop-word pollution: Without filtering, "What does GAE manage?" matched InstructGPT appendix QA sections (had literal "What is X? A: ...") scoring 19.9. After adding stop-word filter, query tokens = ['gae', 'stand', 'fundamental', 'tradeoff', 'manage'], GAE paper correctly ranked first.
• Groq 100K daily TPD: Exhausted daily token limit on the 30th question (cross_04). n=29 eval still valid; cross_04 retry deferred.

Files touched

• Added: researchpath/retrieval.py.
• Modified: researchpath/index.py (+global_idx field), scripts/ask.py (+--hybrid), scripts/run_eval.py (+--hybrid, path fix), README.md (eval table col 2), PROJECT_LOG.md.
• Generated (gitignored): results in memory only (save failed on path bug, now fixed).

Outcome

Hybrid BM25+FAISS results (Groq llama-3.3-70b, k=5, n=29):

Metric	Baseline	Hybrid	Delta
Retrieval Recall@5	90.0%	89.7%	-0.3 pp
Citation Presence	86.7%	86.2%	-0.5 pp
Answer Correctness	36.7%	72.4%	+35.7 pp
Avg Latency	5.07s	4.58s	-0.5s

By difficulty: easy 100%, medium 73.3%, hard 42.9%. Hard questions remain the weak spot.

Why hybrid helped so much: The global_idx bug fix + stop-word filter together fixed hybrid retrieval so it correctly fuses dense + sparse signals. Many questions where the right paper was retrieved but answer was wrong (dense recall=100%, correct=N in baseline) flipped to correct under hybrid because BM25 surfaced chunks with the specific technical term rather than contextual chunks about the same topic.

Next

Retry cross_04 when Groq TPD resets (saves the 30/30 result). Then: cross-encoder reranker as column 3 in the ablation table. Target: 75%+ correctness on hard questions.

---

2026-05-07 — [implementation] Day 5 cont.: Agentic planning loop (offline prerequisite-chain planner)

Type: implementation Status: complete Duration: ~60 min

Context

The killer differentiator of ResearchPath over generic RAG is the prerequisite-chain reading-path planner: given a target RL paper and what the user already knows, compute the topologically-sorted chain of papers they need to read first. This module runs fully offline — no LLM calls, no API quota burned. It's the deterministic core that an LLM annotation layer will augment later.

Decisions

• Static prerequisite graph as the seed: hand-curate ~10 edges covering canonical chains (DQN→Double DQN→Dueling DQN→Rainbow, TRPO→GAE, DQN→DDPG, DQN→A3C, InstructGPT→DPO). Enough to demo the killer feature; LLM-extracted edges added in a future iteration.
• BFS backwards from target, stopping at known_ids — avoids pulling in papers the user already understands.
• Kahn's algorithm for topological sort with (year, arxiv_id) tie-breaking — deterministic + chronologically sensible for RL literature.
• ReadingStep annotation: each step carries bridges_to (which later papers depend on it), concepts (the bridging ideas), and why (user-facing rationale) aggregated from the static edge metadata.

Actions taken

• researchpath/prerequisites.py — Prerequisite dataclass + STATIC_PREREQUISITES tuple (10 hand-curated edges) + edges_into / edges_from accessors.
• researchpath/planning.py — ReadingStep, ReadingPlan dataclasses + _bfs_backwards, _topo_sort (Kahn's), _build_step, plan_reading_path. ReadingPlan.render_text() produces human-readable ordered list.
• scripts/plan.py — CLI: --target (arxiv ID or tag, case-insensitive), --known (comma-separated), --list. Rich Panel output.
• tests/test_planning.py — 7 pytest tests: no-prereq paper (1 step), Rainbow full chain (4 steps in order), known set truncates, known target → empty plan, topo order respects all edges, unknown target raises ValueError, render_text contains target title.

Smoke tests (pre-pytest, verified manually):

• Rainbow (1710.02298, no prior knowledge): 1312.5602 → 1509.06461 → 1511.06581 → 1710.02298 ✓
• Rainbow knowing DQN: 1509.06461 → 1511.06581 → 1710.02298 (3 steps) ✓
• DPO (2305.18290) knowing InstructGPT: 1 step (DPO itself) ✓
• DDPG (1509.02971): 1312.5602 → 1509.02971 (2 steps) ✓

Issues encountered

None. BFS + Kahn's on the static graph is deterministic and small enough that correctness is straightforward to verify.

Files touched

• Added: researchpath/prerequisites.py, researchpath/planning.py, scripts/plan.py, tests/test_planning.py.

Outcome

7/7 pytest tests pass in 0.09s. Planning loop runs offline without any API calls. Demonstrates genuine agentic behavior (graph traversal, state-aware planning) that differentiates ResearchPath from simple RAG demos.

Sample CLI output (uv run python scripts/plan.py --target Rainbow):

1. [1312.5602] DQN (2013) — Playing Atari with Deep Reinforcement Learning
     bridges: -> 1509.06461, 1511.06581, 1710.02298
     concepts: Q-learning, experience replay, target network

2. [1509.06461] DoubleDQN (2015) — Deep Reinforcement Learning with Double Q-learning
     bridges: -> 1511.06581, 1710.02298
     concepts: overestimation bias, decoupled action selection/evaluation, Double Q-learning

3. [1511.06581] DuelingDQN (2015) — Dueling Network Architectures for Deep Reinforcement Learning
     bridges: -> 1710.02298
     concepts: dueling networks

4. [1710.02298] Rainbow (2017) — Rainbow: Combining Improvements in Deep Reinforcement Learning
     bridges: <- target

Next

• Retry cross_04 when Groq TPD resets → update README hybrid result from n=29 to n=30
• Run reranker full eval (column 3 in ablation table): --hybrid --rerank
• LLM annotation layer: map free-text user background → known_ids set; per-step "what to focus on" rationale
• Fetch 4 missing papers: PER (1511.05952), PPO (1707.06347), IMPALA (1802.01561), Decision Transformer (2106.01345)

---