Spaces:
Runtime error
Runtime error
File size: 4,735 Bytes
a753e74 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 | # Implementation Options Research
## Summary
This document captures the MVP stack decisions for ResearchLink AI based on an analysis of available tools, libraries, and patterns as of 2026.
---
## PDF Parsing Options
| Tool | Pros | Cons | Decision |
|------|------|------|----------|
| **PyMuPDF (fitz)** | Fast, reliable text + structure extraction, layout-aware | C extension, requires libmupdf | **Selected** |
| pypdf | Pure Python, lightweight | Poor structure extraction, no layout | Backup only |
| pdfplumber | Good table extraction | Slow on large PDFs | Future use for tables |
| Marker | ML-based, high quality | Heavy dependencies, GPU preferred | Stage 2 |
| Grobid | Best scientific structure parsing | Java service, complex setup | Future self-hosted option |
**Recommendation:** PyMuPDF for MVP. Grobid for Stage 2 if high-quality extraction is needed.
---
## Metadata Lookup Options
| Service | API | Free Tier | Notes |
|---------|-----|-----------|-------|
| **Semantic Scholar** | REST, no key required | Yes (rate limited) | **Selected** |
| OpenAlex | REST, no key required | Yes | Good coverage, fast |
| Crossref | REST, polite pool | Yes | Best for DOIs |
| arXiv API | Atom/REST | Yes | Best for arXiv papers |
| SSRN | No public API | N/A | Manual only |
| OpenReview | REST | Yes | Good for NeurIPS/ICLR |
**Recommendation:** Semantic Scholar as primary, Crossref for DOI enrichment. Both are free and well-documented.
---
## Citation Validation Options
| Approach | Quality | Complexity |
|----------|---------|------------|
| Regex + heuristics | Low-medium | Low |
| **bibtexparser + heuristics** | Medium | Low | **Selected for MVP** |
| Crossref/Semantic Scholar lookup | High | Medium |
| anystyle-parser | High (Ruby) | High |
**Recommendation:** bibtexparser for BibTeX parsing + Crossref for DOI verification in Stage 2.
---
## GitHub Repository Analysis Options
| Approach | Coverage | Auth Required |
|----------|----------|---------------|
| **GitHub REST API (unauthenticated)** | Public repos | No (rate-limited) | **Selected** |
| GitHub REST API (token) | Public + higher rate limit | Yes | Recommended for production |
| GitHub GraphQL API | Rich queries | Yes |
| GitPython (local clone) | Full access | No (local) |
**Recommendation:** GitHub REST API unauthenticated for MVP. Support `GITHUB_TOKEN` env var for higher rate limits.
---
## Multi-Agent Orchestration Options
| Framework | Description | Decision |
|-----------|-------------|----------|
| **Custom sequential pipeline** | Simple, transparent, no overhead | **Selected for MVP** |
| LangChain Agents | Many integrations, complex | Overkill for MVP |
| LlamaIndex | Good for RAG, less for pipeline | Stage 2 option |
| CrewAI | Role-based agents | Stage 2 option |
| AutoGen | Conversational multi-agent | Research tool option |
**Recommendation:** Custom sequential pipeline for MVP. Migrate to CrewAI or custom async orchestration in Stage 2.
---
## Markdown and Documentation Generation Options
| Tool | Use Case | Decision |
|------|----------|----------|
| **Jinja2** | Template rendering | **Selected** |
| **Direct LLM generation** | Content writing | **Selected** |
| MkDocs | Documentation site | Stage 2 |
| Pandoc | Format conversion | Future |
**Recommendation:** Jinja2 for structure templates. Direct LLM (Claude) calls for content generation.
---
## Recommended MVP Stack
```
Python 3.11+
typer — CLI
pydantic v2 — schemas and validation
PyYAML — YAML I/O
jinja2 — templates
httpx — HTTP client
pymupdf — PDF text extraction
anthropic SDK — LLM calls (claude-sonnet-4-6)
bibtexparser — BibTeX parsing
rich — CLI output
python-slugify — slug generation
pytest — tests
ruff — linting
```
---
## Future SaaS Stack
```
FastAPI — REST API backend
Next.js — web frontend
PostgreSQL — persistent storage
Redis — caching and task queue
Celery — background workers
MinIO / S3 — PDF and asset storage
GitHub App — OAuth + webhook integration
Docker + Kubernetes — deployment
Stripe — billing
```
---
## Risks and Tradeoffs
| Risk | Impact | Mitigation |
|------|--------|------------|
| PDF extraction quality varies | Medium | Mark confidence; allow manual override |
| LLM generates hallucinated citations | High | Never auto-trust LLM citations; mark all as needs-verification |
| Semantic Scholar rate limits | Low | Add exponential backoff; cache responses |
| GitHub API rate limits (unauthenticated) | Medium | Support GITHUB_TOKEN; cache tree responses |
| Paper text may be behind paywall | Medium | Document clearly; accept user-provided PDFs |
| LLM API cost at scale | High | Batch requests; cache outputs; Stage 2 cost controls |
|