Spaces:
Running
Running
Commit Β·
5fd7c4e
1
Parent(s): 0ea2446
docs: update README.md to reflect V2 architecture (Qdrant, Hybrid Search, new features)
Browse files
README.md
CHANGED
|
@@ -9,15 +9,208 @@ pinned: false
|
|
| 9 |
|
| 10 |
# π ΰΈΰΈ£ΰΈ°ΰΉΰΈΰΈ£ΰΈΰΈ΄ΰΈΰΈ ΰΈΰΈΰΈ±ΰΈΰΈ‘ΰΈ«ΰΈ²ΰΈΰΈΈΰΈ¬ΰΈ²ΰΈ₯ΰΈΰΈΰΈ£ΰΈΰΈ£ΰΈ²ΰΈΰΈ§ΰΈ΄ΰΈΰΈ’ΰΈ²ΰΈ₯ΰΈ±ΰΈ’
|
| 11 |
|
| 12 |
-
Tipitaka
|
| 13 |
|
| 14 |
-
|
| 15 |
-
- Backend: FastAPI (Python)
|
| 16 |
-
- Database: SQLite (loaded in-memory)
|
| 17 |
-
- Search: ChromaDB with Qwen3-Embedding-0.6B
|
| 18 |
-
- AI: DeepSeek API (deepseek-chat / deepseek-reasoner)
|
| 19 |
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
-
|
| 23 |
-
- DB + vector index downloaded from HF bucket at runtime
|
|
|
|
| 9 |
|
| 10 |
# π ΰΈΰΈ£ΰΈ°ΰΉΰΈΰΈ£ΰΈΰΈ΄ΰΈΰΈ ΰΈΰΈΰΈ±ΰΈΰΈ‘ΰΈ«ΰΈ²ΰΈΰΈΈΰΈ¬ΰΈ²ΰΈ₯ΰΈΰΈΰΈ£ΰΈΰΈ£ΰΈ²ΰΈΰΈ§ΰΈ΄ΰΈΰΈ’ΰΈ²ΰΈ₯ΰΈ±ΰΈ’
|
| 11 |
|
| 12 |
+
**Tipitaka Web App** β ΰΉΰΈ§ΰΉΰΈΰΈΰΉΰΈ²ΰΈΰΈΰΈ£ΰΈ°ΰΉΰΈΰΈ£ΰΈΰΈ΄ΰΈΰΈ ΰΈΰΈΰΈ±ΰΈ ΰΈ‘ΰΈΰΈ£. ΰΉΰΈΰΈΰΈΰΈ΄ΰΈΰΈ§ΰΈ΄ΰΉΰΈ§ΰΈ ΰΈΰΈ£ΰΉΰΈΰΈ‘ AI ΰΈΰΈΉΰΉΰΈΰΉΰΈ§ΰΈ’ΰΈΰΈ±ΰΈΰΈΰΈ£ΰΈ΄ΰΈ’ΰΈ°
|
| 13 |
|
| 14 |
+
> **Stack:** React + Vite | FastAPI + Python | SQLite FTS5 | Qdrant Vector | ONNX Reranker | DeepSeek API
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
## β¨ Features
|
| 19 |
+
|
| 20 |
+
| Feature | Description |
|
| 21 |
+
|---------|-------------|
|
| 22 |
+
| **π Reader** | 45+ ΰΉΰΈ₯ΰΉΰΈ‘ΰΈΰΈ£ΰΈ°ΰΉΰΈΰΈ£ΰΈΰΈ΄ΰΈΰΈ β ΰΈ«ΰΈΰΉΰΈ²-ΰΈΰΉΰΈ-ΰΈ«ΰΈΰΉΰΈ², ΰΈΰΈ£ΰΈ±ΰΈΰΈΰΈΰΈ²ΰΈΰΈΰΈ±ΰΈ§ΰΈΰΈ±ΰΈΰΈ©ΰΈ£, 3 themes (dark/light/classic) |
|
| 23 |
+
| **π Hybrid Search** | FTS5 + Vector (Qdrant) + ONNX Reranker β ΰΈΰΈΉΰΈΰΈΰΉΰΈΰΈΰΈΰΈ±ΰΉΰΈ keyword ΰΉΰΈ₯ΰΈ°ΰΈΰΈ§ΰΈ²ΰΈ‘ΰΈ«ΰΈ‘ΰΈ²ΰΈ’ |
|
| 24 |
+
| **π€ AI Assistant** | Floating chat ΰΈΰΈ£ΰΉΰΈΰΈ‘ RAG context, ΰΈ£ΰΈΰΈΰΈ£ΰΈ±ΰΈ DeepSeek-chat / DeepSeek-reasoner |
|
| 25 |
+
| **π Cross-Volume RAG** | ΰΈΰΉΰΈΰΈΰΉΰΈ²ΰΈ‘ΰΉΰΈ₯ΰΉΰΈ‘ΰΈΰΈ²ΰΈ Qdrant vector store + jina-embeddings-v3, rerank ΰΈΰΉΰΈ§ΰΈ’ jina-reranker-v2 |
|
| 26 |
+
| **π± Responsive** | Desktop + Mobile, swipe gesture, keyboard nav |
|
| 27 |
+
| **π¨ Markdown Rendering** | AI ΰΈΰΈΰΈΰΉΰΈΰΉΰΈ markdown ΰΈΰΈ£ΰΉΰΈΰΈ‘ prose-invert ΰΈͺΰΈ³ΰΈ«ΰΈ£ΰΈ±ΰΈ dark mode |
|
| 28 |
+
| **βοΈ Resizable Panel** | ΰΈ₯ΰΈ²ΰΈΰΈΰΈΰΈΰΈΰΈ§ΰΈ²ΰΉΰΈΰΈ·ΰΉΰΈΰΈΰΈ£ΰΈ±ΰΈΰΈΰΈΰΈ²ΰΈ AI chat β persist ΰΈΰΈΰΈ²ΰΈΰΉΰΈ§ΰΉ |
|
| 29 |
+
| **π·οΈ Pali Autocorrect** | ΰΉΰΈΰΉΰΈΰΈ³ΰΈΰΈ²ΰΈ₯ΰΈ΅ΰΈΰΈ΄ΰΈΰΉΰΈΰΈ ΰΈΰΉΰΈΰΈΰΈͺΰΉΰΈΰΈΰΉΰΈΰΈ«ΰΈ² β PyThaiNLP + custom dict |
|
| 30 |
+
|
| 31 |
+
---
|
| 32 |
+
|
| 33 |
+
## ποΈ Architecture (V2)
|
| 34 |
+
|
| 35 |
+
```
|
| 36 |
+
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 37 |
+
β Frontend (Vite + React) β
|
| 38 |
+
β AppShell β NavDrawer | ReaderPanel | RightToolbar | AIPopup β
|
| 39 |
+
β Zustand (5 stores) Β· Framer Motion Β· Tailwind CSS v4 β
|
| 40 |
+
β react-markdown + remark-gfm Β· Lucide React β
|
| 41 |
+
ββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββ
|
| 42 |
+
β HTTP / SSE
|
| 43 |
+
ββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββ
|
| 44 |
+
β Backend (FastAPI Python 3.11) β
|
| 45 |
+
β β
|
| 46 |
+
β ββββββββββββββ ββββββββββββββ ββββββββββββββ βββββββββββββ β
|
| 47 |
+
β β Page β β Search β β RAG β β LLM β β
|
| 48 |
+
β β Service β β Service β β Service β β Service β β
|
| 49 |
+
β βββββββ¬βββββββ βββββββ¬βββββββ βββββββ¬βββββββ βββββββ¬ββββββ β
|
| 50 |
+
β β β β β β
|
| 51 |
+
β ββββββΌβββββ βββββββΌββββββ ββββββΌβββββ ββββββΌββββββ β
|
| 52 |
+
β β SQLite β β SQLite β β Qdrant β β DeepSeek β β
|
| 53 |
+
β β FTS5 β β LIKE β β Vector β β API β β
|
| 54 |
+
β β (memory)β β fallback β β + ONNX β β Streamingβ β
|
| 55 |
+
β βββββββββββ βββββββββββββ β Rerankerβ ββββββββββββ β
|
| 56 |
+
β βββββββββββ β
|
| 57 |
+
β Services: page_service Β· search_service Β· pali_utils β
|
| 58 |
+
β rag_service Β· onnx_reranker Β· llm_service β
|
| 59 |
+
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
### π Hybrid Search Pipeline
|
| 63 |
+
|
| 64 |
+
```
|
| 65 |
+
ΰΈΰΈ³ΰΈΰΈ²ΰΈ‘ β Pali Autocorrect β FTS5 (SQLite, 30 results)
|
| 66 |
+
ββ Qdrant Vector (jina-embeddings-v3, 30 results)
|
| 67 |
+
β Merge & Deduplicate
|
| 68 |
+
β ONNX Rerank (jina-reranker-v2)
|
| 69 |
+
β Top N β AI context
|
| 70 |
+
```
|
| 71 |
+
|
| 72 |
+
---
|
| 73 |
+
|
| 74 |
+
## ποΈ Project Structure
|
| 75 |
+
|
| 76 |
+
```
|
| 77 |
+
RAG/
|
| 78 |
+
βββ webapp/
|
| 79 |
+
β βββ tipitaka-api/ # FastAPI backend
|
| 80 |
+
β β βββ app/
|
| 81 |
+
β β β βββ main.py # FastAPI entry + lifespan
|
| 82 |
+
β β β βββ config.py # pydantic-settings
|
| 83 |
+
β β β βββ schemas.py # Pydantic models
|
| 84 |
+
β β β βββ routers/ # pages, search, ai, health
|
| 85 |
+
β β β βββ services/ # 6 service modules
|
| 86 |
+
β β β βββ database/ # sqlite_db.py
|
| 87 |
+
β β βββ tests/ # 35 tests
|
| 88 |
+
β β βββ models/ # ONNX reranker
|
| 89 |
+
β β βββ startup.sh # HF Spaces entry
|
| 90 |
+
β β
|
| 91 |
+
β βββ tipitaka-web/ # React frontend
|
| 92 |
+
β βββ src/
|
| 93 |
+
β βββ components/ # layout, reader, toolbar, ai
|
| 94 |
+
β βββ stores/ # Zustand (5 stores)
|
| 95 |
+
β βββ hooks/ # swipe, keyboard
|
| 96 |
+
β βββ lib/ # api.ts
|
| 97 |
+
β
|
| 98 |
+
βββ qdrant_storage/ # Local Qdrant data
|
| 99 |
+
βββ snapshots/ # Qdrant snapshot files
|
| 100 |
+
βββ Dockerfile # Multi-stage build
|
| 101 |
+
βββ TIPITAKA_WEB_ARCHITECTURE_V2.md # Full architecture doc
|
| 102 |
+
```
|
| 103 |
+
|
| 104 |
+
---
|
| 105 |
+
|
| 106 |
+
## π Quick Start
|
| 107 |
+
|
| 108 |
+
### Prerequisites
|
| 109 |
+
- Python 3.11+
|
| 110 |
+
- Node.js 20+
|
| 111 |
+
- Ollama (for embedding β **jina-embeddings-v3**)
|
| 112 |
+
- Qdrant (optional β auto-detect localhost:6333)
|
| 113 |
+
- DeepSeek API key
|
| 114 |
+
|
| 115 |
+
### 1. Setup Backend
|
| 116 |
+
```bash
|
| 117 |
+
cd webapp/tipitaka-api
|
| 118 |
+
python -m venv .venv
|
| 119 |
+
.venv\Scripts\activate # Windows
|
| 120 |
+
pip install -r requirements.txt
|
| 121 |
+
|
| 122 |
+
# Copy .env.example β .env and fill DEEPSEEK_API_KEY
|
| 123 |
+
python -m app.main # starts on :8000
|
| 124 |
+
```
|
| 125 |
+
|
| 126 |
+
### 2. Setup Frontend
|
| 127 |
+
```bash
|
| 128 |
+
cd webapp/tipitaka-web
|
| 129 |
+
npm install
|
| 130 |
+
npm run dev # starts on :5173
|
| 131 |
+
```
|
| 132 |
+
|
| 133 |
+
### 3. Setup Embedding
|
| 134 |
+
```bash
|
| 135 |
+
# Download jina-embeddings-v3 GGUF
|
| 136 |
+
ollama pull hf.co/second-state/jina-embeddings-v3-GGUF:jina-embeddings-v3-Q4_K_M.gguf
|
| 137 |
+
```
|
| 138 |
+
|
| 139 |
+
### 4. Run Tests
|
| 140 |
+
```bash
|
| 141 |
+
cd webapp/tipitaka-api
|
| 142 |
+
python -m pytest tests/ -v # 35 tests, all passing
|
| 143 |
+
```
|
| 144 |
+
|
| 145 |
+
---
|
| 146 |
+
|
| 147 |
+
## π‘ API Endpoints
|
| 148 |
+
|
| 149 |
+
| Method | Path | Description |
|
| 150 |
+
|--------|------|-------------|
|
| 151 |
+
| `GET` | `/api/pages/volumes` | ΰΈ£ΰΈ²ΰΈ’ΰΈΰΈ²ΰΈ£ΰΉΰΈ₯ΰΉΰΈ‘ΰΈΰΈ£ΰΈ°ΰΉΰΈΰΈ£ΰΈΰΈ΄ΰΈΰΈ |
|
| 152 |
+
| `GET` | `/api/pages/volumes/{id}/toc` | ΰΈͺΰΈ²ΰΈ£ΰΈΰΈ±ΰΈ (dedup + noise filter) |
|
| 153 |
+
| `GET` | `/api/pages/{vol}/{page}` | ΰΉΰΈΰΈ·ΰΉΰΈΰΈ«ΰΈ²ΰΈ«ΰΈΰΉΰΈ² |
|
| 154 |
+
| `GET` | `/api/search?q=&limit=&offset=` | ΰΈΰΉΰΈΰΈ«ΰΈ² FTS5 + LIKE + highlight |
|
| 155 |
+
| `GET` | `/api/search/suggestions?q=` | Autocomplete |
|
| 156 |
+
| `POST` | `/api/ask/stream` | AI ΰΈΰΈ²ΰΈ‘-ΰΈΰΈΰΈ (SSE streaming) |
|
| 157 |
+
| `POST` | `/api/ask` | AI ΰΈΰΈ²ΰΈ‘-ΰΈΰΈΰΈ (non-stream β Discord) |
|
| 158 |
+
| `GET` | `/api/ask/rag-status` | ΰΈͺΰΈΰΈ²ΰΈΰΈ° Qdrant |
|
| 159 |
+
| `GET` | `/health` | Health check |
|
| 160 |
+
|
| 161 |
+
---
|
| 162 |
+
|
| 163 |
+
## π§ͺ Test Suite
|
| 164 |
+
|
| 165 |
+
```
|
| 166 |
+
tests/
|
| 167 |
+
βββ test_pali_utils.py # 90 assertions β autocorrect, digits, similarity
|
| 168 |
+
βββ test_search_service.py # FTS5 + LIKE merge, suggestions
|
| 169 |
+
βββ test_rag_service.py # Embedding LRU cache
|
| 170 |
+
|
| 171 |
+
Total: 35 tests β
|
| 172 |
+
```
|
| 173 |
+
|
| 174 |
+
---
|
| 175 |
+
|
| 176 |
+
## π§° Tech Stack Detail
|
| 177 |
+
|
| 178 |
+
| Layer | Technology |
|
| 179 |
+
|-------|-----------|
|
| 180 |
+
| **Frontend** | React 18, TypeScript 5, Vite 6, Tailwind CSS v4 |
|
| 181 |
+
| **State** | Zustand (5 stores) |
|
| 182 |
+
| **Animation** | Framer Motion |
|
| 183 |
+
| **Icons** | Lucide React |
|
| 184 |
+
| **Markdown** | react-markdown + remark-gfm |
|
| 185 |
+
| **Backend** | FastAPI, Python 3.11 |
|
| 186 |
+
| **Database** | SQLite (FTS5, loaded to `:memory:` at startup) |
|
| 187 |
+
| **Vector Store** | Qdrant (local/server dual-mode, snapshot restore) |
|
| 188 |
+
| **Embedding** | jina-embeddings-v3 (Q4_K_M GGUF via Ollama, 1024 dims) |
|
| 189 |
+
| **Reranker** | jina-reranker-v2-base-multilingual (ONNX, CPU) |
|
| 190 |
+
| **AI** | DeepSeek API (OpenAI-compatible, SSE streaming) |
|
| 191 |
+
| **CI/CD** | HF Spaces (Docker multi-stage, auto-build) |
|
| 192 |
+
|
| 193 |
+
---
|
| 194 |
+
|
| 195 |
+
## π Documents
|
| 196 |
+
|
| 197 |
+
| File | Description |
|
| 198 |
+
|------|-------------|
|
| 199 |
+
| `DESIGN.md` | Design system β colors, typography, components, themes |
|
| 200 |
+
| `PRODUCT.md` | Product vision, users, principles |
|
| 201 |
+
| `TIPITAKA_WEB_ARCHITECTURE_V2.md` | Full architecture V2 document |
|
| 202 |
+
| `TIPITAKA_WEB_ARCHITECTURE.md` | Original V1 architecture (legacy reference) |
|
| 203 |
+
|
| 204 |
+
---
|
| 205 |
+
|
| 206 |
+
## π Acknowledgements
|
| 207 |
+
|
| 208 |
+
- **ΰΈ‘ΰΈ«ΰΈ²ΰΈΰΈΈΰΈ¬ΰΈ²ΰΈ₯ΰΈΰΈΰΈ£ΰΈΰΈ£ΰΈ²ΰΈΰΈ§ΰΈ΄ΰΈΰΈ’ΰΈ²ΰΈ₯ΰΈ±ΰΈ’ (MCU)** β ΰΉΰΈΰΈ·ΰΉΰΈΰΈ«ΰΈ²ΰΈΰΈ£ΰΈ°ΰΉΰΈΰΈ£ΰΈΰΈ΄ΰΈΰΈ
|
| 209 |
+
- **Qdrant** β Vector database
|
| 210 |
+
- **jina.ai** β Embedding + Reranker models
|
| 211 |
+
- **Ollama** β Local model serving
|
| 212 |
+
- **Hugging Face Spaces** β Deployment platform
|
| 213 |
+
|
| 214 |
+
---
|
| 215 |
|
| 216 |
+
*"ΰΈΰΈΈΰΈΰΈ‘ΰΈ²ΰΉΰΈ«ΰΈ‘ΰΈ·ΰΈΰΈΰΉΰΈͺΰΈΰΉΰΈΰΈ΅ΰΈ’ΰΈΰΉΰΈΰΈ«ΰΉΰΈΰΈΰΈ‘ΰΈ·ΰΈ ΰΈΰΈ·ΰΈΰΈΰΈ±ΰΈΰΈΰΈ²ΰΈΰΈ³ΰΈ₯ΰΈ²ΰΈ’ΰΈΰΈ§ΰΈ΄ΰΈΰΈΰΈ²"*
|
|
|