Vasanth6 commited on
Commit
ce33ef6
Β·
1 Parent(s): 5fbafd7

changes for readme file and UI changes

Browse files
README.md CHANGED
@@ -49,35 +49,39 @@ Visualize and compare **5 chunking strategies** side-by-side:
49
  ## πŸ—οΈ Architecture
50
 
51
  ```mermaid
52
- graph TB
53
- subgraph Frontend ["Frontend (Vanilla HTML/CSS/JS)"]
54
- UI[index.html] --> JS[app.js]
55
- UI --> CSS[styles.css]
56
- JS -->|Canvas 2D API| Canvas[Vector Space Renderer]
57
- JS -->|DOM Manipulation| XRay[Document X-Ray Viewer]
58
- end
59
-
60
- subgraph Backend ["Backend (FastAPI + Python)"]
61
- API[FastAPI Server] --> ChunkRouter["/api/chunk"]
62
- API --> RetrievalRouter["/api/retrieve"]
63
-
64
- ChunkRouter --> ChunkEngine[Chunking Engine]
65
- ChunkRouter --> EmbedEngine[Embedding Engine]
66
- ChunkRouter --> Reducer[UMAP Reducer]
67
- ChunkRouter --> VStore[Vector Store]
68
-
69
- RetrievalRouter --> EmbedEngine
70
- RetrievalRouter --> VStore
71
-
72
- ChunkEngine -->|5 Strategies| Splitters[LangChain + NLTK Splitters]
73
- EmbedEngine -->|HTTP| Ollama[Ollama API :11434]
74
- Reducer --> UMAP[umap-learn]
75
- VStore --> ChromaDB[(ChromaDB)]
76
- end
77
-
78
- JS -->|fetch /api/chunk| ChunkRouter
79
- JS -->|fetch /api/retrieve| RetrievalRouter
80
- API -->|Static Files| UI
 
 
 
 
81
  ```
82
 
83
  ### Data Flow
@@ -279,13 +283,7 @@ Returns the list of available chunking strategies.
279
  | **Vector Database** | ChromaDB (persistent) | Cosine similarity search with HNSW index |
280
  | **Package Manager** | uv | Dependency management & virtual environments |
281
 
282
- ## πŸ“ License
283
-
284
- This project is for educational and personal use.
285
-
286
- ---
287
-
288
- ## πŸ™ Acknowledgements
289
 
290
  - [Ollama](https://ollama.com/) β€” Local LLM inference
291
  - [ChromaDB](https://www.trychroma.com/) β€” Open-source vector database
 
49
  ## πŸ—οΈ Architecture
50
 
51
  ```mermaid
52
+ flowchart LR
53
+
54
+ %% Ingestion Flow
55
+ DOC["πŸ“„ Input Document"] --> CHUNK["βœ‚οΈ Chunking Engine<br/>5 Chunking Strategies"]
56
+
57
+ CHUNK --> SPLIT["πŸ“ LangChain + NLTK<br/>Text Splitters"]
58
+
59
+ SPLIT --> EMBED["🧠 Embedding Engine<br/>Generate Semantic Vectors"]
60
+
61
+ EMBED --> OLLAMA["πŸ€– Ollama API<br/>Embedding Model"]
62
+
63
+ EMBED --> UMAP["πŸ“‰ UMAP<br/>2D Vector Projection"]
64
+
65
+ UMAP --> DB["πŸ—„οΈ ChromaDB<br/>Vector Storage"]
66
+
67
+ %% Retrieval Flow
68
+ USER["πŸ‘€ User Query"] --> QEMBED["🧠 Query Embedding"]
69
+
70
+ QEMBED --> OLLAMA
71
+
72
+ QEMBED --> SEARCH["πŸ” Similarity Search"]
73
+
74
+ SEARCH --> DB
75
+
76
+ DB --> RESULTS["πŸ“š Relevant Chunks"]
77
+
78
+ %% Visualization
79
+ RESULTS --> XRAY["πŸ”¬ Document X-Ray Viewer"]
80
+
81
+ UMAP --> VIS["πŸ“Š Vector Space Renderer"]
82
+
83
+ XRAY --> UI["🌐 Interactive Frontend"]
84
+ VIS --> UI
85
  ```
86
 
87
  ### Data Flow
 
283
  | **Vector Database** | ChromaDB (persistent) | Cosine similarity search with HNSW index |
284
  | **Package Manager** | uv | Dependency management & virtual environments |
285
 
286
+ ## Technologies
 
 
 
 
 
 
287
 
288
  - [Ollama](https://ollama.com/) β€” Local LLM inference
289
  - [ChromaDB](https://www.trychroma.com/) β€” Open-source vector database
backend/engines/chunking.py CHANGED
@@ -75,7 +75,7 @@ def construct_chunk_node(text, chunks, tokenizer):
75
 
76
  # 2. If exact find fails, try a clean stripped version
77
  if start == -1:
78
- clean_anchor = chunk.strip()[:40]
79
  if clean_anchor:
80
  start = text.find(clean_anchor, current_position)
81
 
 
75
 
76
  # 2. If exact find fails, try a clean stripped version
77
  if start == -1:
78
+ clean_anchor = chunk.strip()
79
  if clean_anchor:
80
  start = text.find(clean_anchor, current_position)
81
 
backend/storage/vector_store.py CHANGED
@@ -11,7 +11,12 @@ class VectorStore:
11
 
12
  def get_collection(self, name: str):
13
  return self.client.get_or_create_collection(
14
- name=name, metadata={"hnsw:space": "cosine"}
 
 
 
 
 
15
  )
16
 
17
  async def upsert(self, collection_name, ids, documents, embeddings, metadatas=None):
 
11
 
12
  def get_collection(self, name: str):
13
  return self.client.get_or_create_collection(
14
+ name=name,
15
+ metadata={
16
+ "hnsw:space": "cosine",
17
+ "hnsw:construction_ef": 100,
18
+ "hnsw:M": 16,
19
+ },
20
  )
21
 
22
  async def upsert(self, collection_name, ids, documents, embeddings, metadatas=None):
frontend/app.js CHANGED
@@ -1071,9 +1071,6 @@ function showTooltip(chunk, x, y) {
1071
  chunk.text.length > 80 ? chunk.text.slice(0, 80) + "..." : chunk.text;
1072
  const coordsStr = `[${chunk.coords_2d[0].toFixed(2)}, ${chunk.coords_2d[1].toFixed(2)}]`;
1073
 
1074
- tooltip.style.display = "block";
1075
- tooltip.style.left = `${x + 15}px`;
1076
- tooltip.style.top = `${y + 15}px`;
1077
  tooltip.innerHTML = `
1078
  <div class="vector-tooltip-title">
1079
  <span>${chunk.id}</span>
@@ -1084,6 +1081,21 @@ function showTooltip(chunk, x, y) {
1084
  </div>
1085
  <div class="vector-tooltip-text">${escapeHtml(snippet)}</div>
1086
  `;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1087
  }
1088
 
1089
  function hideTooltip() {
 
1071
  chunk.text.length > 80 ? chunk.text.slice(0, 80) + "..." : chunk.text;
1072
  const coordsStr = `[${chunk.coords_2d[0].toFixed(2)}, ${chunk.coords_2d[1].toFixed(2)}]`;
1073
 
 
 
 
1074
  tooltip.innerHTML = `
1075
  <div class="vector-tooltip-title">
1076
  <span>${chunk.id}</span>
 
1081
  </div>
1082
  <div class="vector-tooltip-text">${escapeHtml(snippet)}</div>
1083
  `;
1084
+ tooltip.style.display = "block";
1085
+
1086
+ let left = x + 15;
1087
+ let top = y + 15;
1088
+ const canvasRect = dom.vectorCanvas.getBoundingClientRect();
1089
+
1090
+ if (left + tooltip.offsetWidth > canvasRect.width) {
1091
+ left = x - tooltip.offsetWidth - 15;
1092
+ }
1093
+ if (top + tooltip.offsetHeight > canvasRect.height) {
1094
+ top = y - tooltip.offsetHeight - 15;
1095
+ }
1096
+
1097
+ tooltip.style.left = `${left}px`;
1098
+ tooltip.style.top = `${top}px`;
1099
  }
1100
 
1101
  function hideTooltip() {