api/ml/doc_store.json

{
  "0": "--- name: jurnalku-dataset-push description: \"Use this skill whenever the user wants to push, upload, or index documents into the Jurnalku AI Knowledge Base. Covers uploading PDF, DOCX, TXT, and MD files to the FastAPI endpoint at /api/ml/dataset/upload, which chunks the text and stores it in the FAISS vector index (faiss_index.bin) via Sentence-Transformers (all-MiniLM-L6-v2, 384-dim). Also covers batch pushing, validating upload responses, inspecting the FAISS index state, and resetting the knowledge base. Use this skill for any task involving the RAG pipeline's ingestion layer. Do NOT use for querying/searching the knowledge base (that is the retrieval side) or for modifying the rag_engine.py logic.\" license: Private — project-specific skill for Jurnalku ---  # Jurnalku Dataset Push — AI Knowledge Base Ingestion  ## Overview  The ingestion pipeline works in this sequence:  ``` File (PDF/DOCX/TXT/MD)   → POST /api/ml/dataset/upload   [FastAPI, api/index.py]   → extract_text_from_*()         [text extraction per format]   → split_into_chunks()           [api/processor/chunker.py, size=1000, overlap=200]   → model.encode()                [all-MiniLM-L6-v2, 384-dim vectors]   → index.add()                   [FAISS IndexFlatL2]   → save_state()                  [faiss_index.bin + doc_store.json] ```  Persistence files live at: - `api/ml/faiss_index.bin` — the FAISS vector index - `api/ml/doc_store.json` — plain-text chunk store (id → text)  ---  ## 1. Single file upload (Python)  Use this for pushing one document at a time via the FastAPI endpoint directly. ```python import httpx  def push_file(filepath: str, base_url: str = \"http://127.0.0.1:5328\") -> dict:     \"\"\"     Push a single PDF/DOCX/TXT/MD file to the knowledge base. Returns the JSON response with chunks_added count. \"\"\"     with open(filepath, \"rb\") as f:         filename = filepath.split(\"/\")[-1]         # Determine MIME type by extension         mime_map = {             \".pdf\":  \"application/pdf\",             \".docx\": \"application/vnd.openxmlformats-officedocument.wordprocessingml.document\",             \".txt\":  \"text/plain\",             \".md\":   \"text/markdown\",         }         ext = \".\" + filename.rsplit(\".\", 1)[-1].lower()         mime = mime_map.get(ext, \"application/octet-stream\")          response = httpx.post(             f\"{base_url}/api/ml/dataset/upload\",             files={\"file\": (filename, f, mime)},             timeout=120.0,  # large files need more time for embedding         )      response.raise_for_status()     result = response.json()     print(f\"[OK] {filename} → {result['chunks_added']} chunks added\")     return result  # Usage push_file(\"dokumen_riset.pdf\") push_file(\"laporan.docx\") push_file(\"catatan.md\") ```  ---  ## 2. Batch upload (multiple files)  Push a whole folder or list of files sequentially. FAISS `IndexFlatL2` is not thread-safe for concurrent writes — always push one file at a time. ```python import os import httpx import time  SUPPORTED_EXTENSIONS = {\".pdf\", \".docx\", \".txt\", \".md\"}  def push_folder(folder_path: str, base_url: str = \"http://127.0.0.1:5328\"):     \"\"\"Push all supported files in a folder to the knowledge base.\"\"\"     files = [         os.path.join(folder_path, f)         for f in os.listdir(folder_path)         if os.path.splitext(f)[1].lower() in SUPPORTED_EXTENSIONS     ]      print(f\"Found {len(files)} file(s) to push...\")     total_chunks = 0     failed = []      for filepath in files:         try:             result = push_file(filepath, base_url)             total_chunks += result.get(\"chunks_added\", 0)             time.sleep(0.5)  # brief pause between files — avoid overwhelming the server         except Exception as e:             print(f\"[FAIL] {filepath}: {e}\")             failed.append(filepath)      print(f\"\\nDone. Total chunks added: {total_chunks}\")     if failed:         print(f\"Failed ({len(failed)}): {failed}\")  # Usage push_folder(\"./dataset/papers/\") ```  ---  ## 3. Upload via Next.js proxy (from frontend/TypeScript)  The Next.js app proxies uploads — requests to `/api/ml/dataset/upload` from the browser go through Next.js to the FastAPI backend at port 5328. ```typescript // src/lib/knowledge.ts  export async function pushDocument(file: File): Promise<{   success: boolean;   chunks_added: number;   message: string; }> {   const formData = new FormData();   formData.append(\"file\", file);    const res = await fetch(\"/api/ml/dataset/upload\", {     method: \"POST\",     body: formData,     // Do NOT set Content-Type header manually — browser sets it with     // the correct multipart boundary automatically   });    if (!res.ok) {     const err = await res.json().catch(() => ({ detail: res.statusText }));     throw new Error(err.detail ?? \"Upload failed\");   }    return res.json(); }  // Usage in a React component async function handleFileChange(e: React.ChangeEvent<HTMLInputElement>) {   const file = e.target.files?.[0];   if (!file) return;    try {     const result = await pushDocument(file);     console.log(`Indexed ${result.chunks_added} chunks from ${file.name}`);   } catch (err) {     console.error(\"Upload failed:\", err);   } } ```  ---  ## 4. Validate the knowledge base state  After pushing, verify that the FAISS index and doc_store are in sync. ```python # scripts/validate_kb.py  import faiss import json import os  INDEX_PATH = \"api/ml/faiss_index.bin\" STORE_PATH = \"api/ml/doc_store.json\"  def validate():     if not os.path.exists(INDEX_PATH):         print(\"[ERROR] faiss_index.bin not found — nothing has been indexed yet.\")         return      index = faiss.read_index(INDEX_PATH)     print(f\"FAISS index: {index.ntotal} vectors, dimension={index.d}\")      with open(STORE_PATH, encoding=\"utf-8\") as f:         store = json.load(f)     print(f\"Doc store:   {len(store)} chunks\")      if index.ntotal != len(store):         print(\"[WARN] Mismatch! Index and doc_store are out of sync.\")         print(\"       Run reset_kb() then re-push all documents.\")     else:         print(\"[OK] Index and doc_store are in sync.\")      # Preview first 3 chunks     print(\"\\nSample chunks:\")     for i, (k, v) in enumerate(list(store.items())[:3]):         print(f\"  [{k}] {v[:120].strip()}...\")  validate() ```  ---  ## 5. Reset the knowledge base  Wipe all indexed documents and start fresh. Use when re-indexing from scratch or when index/doc_store are out of sync. ```python # scripts/reset_kb.py  import faiss import json import os  INDEX_PATH = \"api/ml/faiss_index.bin\" STORE_PATH = \"api/ml/doc_store.json\" DIMENSION  = 384  # must match all-MiniLM-L6-v2  def reset_kb(confirm: bool = False):     if not confirm:         print(\"Pass confirm=True to actually reset. This deletes ALL indexed data.\")         return      # Overwrite with a fresh empty index     index = faiss.IndexFlatL2(DIMENSION)     faiss.write_index(index, INDEX_PATH)      with open(STORE_PATH, \"w\", encoding=\"utf-8\") as f:         json.dump({}, f)      print(\"[OK] Knowledge base reset. faiss_index.bin and doc_store.json cleared.\")  reset_kb(confirm=True) ```  ---  ## 6. Inspect chunks for a specific document  After pushing, check what was actually stored for a given file by searching for a known phrase from its content. ```python import json  def search_store(query: str, store_path: str = \"api/ml/doc_store.json\", top_n: int = 5):     \"\"\"Simple keyword scan of stored chunks — useful for debugging ingestion.\"\"\"     with open(store_path, encoding=\"utf-8\") as f:         store = json.load(f)      hits = [         (k, v) for k, v in store.items()         if query.lower() in v.lower()     ][:top_n]      if not hits:         print(f\"No chunks found containing '{query}'\")         return      for chunk_id, text in hits:         print(f\"--- Chunk {chunk_id} ---\")         print(text[:300])         print()  # Usage — find chunks from a specific paper search_store(\"candidiasis\") search_store(\"machine learning\") ```  ---  ## 7.",
  "1": "Use when re-indexing from scratch or when index/doc_store are out of sync. ```python # scripts/reset_kb.py  import faiss import json import os  INDEX_PATH = \"api/ml/faiss_index.bin\" STORE_PATH = \"api/ml/doc_store.json\" DIMENSION  = 384  # must match all-MiniLM-L6-v2  def reset_kb(confirm: bool = False):     if not confirm:         print(\"Pass confirm=True to actually reset. This deletes ALL indexed data.\")         return      # Overwrite with a fresh empty index     index = faiss.IndexFlatL2(DIMENSION)     faiss.write_index(index, INDEX_PATH)      with open(STORE_PATH, \"w\", encoding=\"utf-8\") as f:         json.dump({}, f)      print(\"[OK] Knowledge base reset. faiss_index.bin and doc_store.json cleared.\")  reset_kb(confirm=True) ```  ---  ## 6. Inspect chunks for a specific document  After pushing, check what was actually stored for a given file by searching for a known phrase from its content. ```python import json  def search_store(query: str, store_path: str = \"api/ml/doc_store.json\", top_n: int = 5):     \"\"\"Simple keyword scan of stored chunks — useful for debugging ingestion.\"\"\"     with open(store_path, encoding=\"utf-8\") as f:         store = json.load(f)      hits = [         (k, v) for k, v in store.items()         if query.lower() in v.lower()     ][:top_n]      if not hits:         print(f\"No chunks found containing '{query}'\")         return      for chunk_id, text in hits:         print(f\"--- Chunk {chunk_id} ---\")         print(text[:300])         print()  # Usage — find chunks from a specific paper search_store(\"candidiasis\") search_store(\"machine learning\") ```  ---  ## 7. Common errors and fixes  | Error | Cause | Fix | |-------|-------|-----| | `400 Unsupported format` | File extension not in `.pdf .docx .txt .md` | Convert to supported format first | | `400 File kosong` | Text extraction returned empty string | Check if PDF is scanned (image-only) — needs OCR first | | `500 Internal Server Error` | `dev:py` not running, or dependency missing | Run `npm run dev:py` and check terminal for Python errors | | `WARN Mismatch index/store` | `save_state()` interrupted mid-write | Reset KB and re-push all documents | | Very slow embedding | Large file with many chunks | Normal — `all-MiniLM-L6-v2` on CPU does ~100–500 chunks/sec | | `connection refused port 5328` | FastAPI server not running | `npm run dev:py` in a separate terminal |  ---  ## 8. Chunking parameters reference  Defined in `api/processor/chunker.py`, called with: ```python chunks = split_into_chunks(text, chunk_size=1000, overlap=200) ```  | Parameter | Value | Effect | |-----------|-------|--------| | `chunk_size` | 1000 chars | Each chunk ≈ 150–200 words. Good for paragraph-level retrieval. | | `overlap` | 200 chars | 20% overlap prevents context loss at chunk boundaries. |  To tune for denser academic papers (longer context needed): ```python # In api/index.py — modify the call: chunks = split_into_chunks(text, chunk_size=1500, overlap=300) ```  ---  ## Quick reference  ```bash # Push one file (Python CLI) python -c \"from scripts.push import push_file; push_file('file.pdf')\"  # Push entire folder python -c \"from scripts.push import push_folder; push_folder('./dataset/')\"  # Validate KB state python scripts/validate_kb.py  # Reset KB (destructive!) python scripts/reset_kb.py  # Check FastAPI is running curl http://127.0.0.1:5328/api/ml/dataset/upload -X POST # Expected: 422 Unprocessable Entity (missing file field) = server is up ```",
  "2": "--- name: jurnalku-dataset-push description: \"Use this skill whenever the user wants to push, upload, or index documents into the Jurnalku AI Knowledge Base. Covers uploading PDF, DOCX, TXT, and MD files to the FastAPI endpoint at /api/ml/dataset/upload, which chunks the text and stores it in the FAISS vector index (faiss_index.bin) via Sentence-Transformers (all-MiniLM-L6-v2, 384-dim). Also covers batch pushing, validating upload responses, inspecting the FAISS index state, and resetting the knowledge base. Use this skill for any task involving the RAG pipeline's ingestion layer. Do NOT use for querying/searching the knowledge base (that is the retrieval side) or for modifying the rag_engine.py logic.\" license: Private — project-specific skill for Jurnalku ---  # Jurnalku Dataset Push — AI Knowledge Base Ingestion  ## Overview  The ingestion pipeline works in this sequence:  ``` File (PDF/DOCX/TXT/MD)   → POST /api/ml/dataset/upload   [FastAPI, api/index.py]   → extract_text_from_*()         [text extraction per format]   → split_into_chunks()           [api/processor/chunker.py, size=1000, overlap=200]   → model.encode()                [all-MiniLM-L6-v2, 384-dim vectors]   → index.add()                   [FAISS IndexFlatL2]   → save_state()                  [faiss_index.bin + doc_store.json] ```  Persistence files live at: - `api/ml/faiss_index.bin` — the FAISS vector index - `api/ml/doc_store.json` — plain-text chunk store (id → text)  ---  ## 1. Single file upload (Python)  Use this for pushing one document at a time via the FastAPI endpoint directly. ```python import httpx  def push_file(filepath: str, base_url: str = \"http://127.0.0.1:5328\") -> dict:     \"\"\"     Push a single PDF/DOCX/TXT/MD file to the knowledge base. Returns the JSON response with chunks_added count. \"\"\"     with open(filepath, \"rb\") as f:         filename = filepath.split(\"/\")[-1]         # Determine MIME type by extension         mime_map = {             \".pdf\":  \"application/pdf\",             \".docx\": \"application/vnd.openxmlformats-officedocument.wordprocessingml.document\",             \".txt\":  \"text/plain\",             \".md\":   \"text/markdown\",         }         ext = \".\" + filename.rsplit(\".\", 1)[-1].lower()         mime = mime_map.get(ext, \"application/octet-stream\")          response = httpx.post(             f\"{base_url}/api/ml/dataset/upload\",             files={\"file\": (filename, f, mime)},             timeout=120.0,  # large files need more time for embedding         )      response.raise_for_status()     result = response.json()     print(f\"[OK] {filename} → {result['chunks_added']} chunks added\")     return result  # Usage push_file(\"dokumen_riset.pdf\") push_file(\"laporan.docx\") push_file(\"catatan.md\") ```  ---  ## 2. Batch upload (multiple files)  Push a whole folder or list of files sequentially. FAISS `IndexFlatL2` is not thread-safe for concurrent writes — always push one file at a time. ```python import os import httpx import time  SUPPORTED_EXTENSIONS = {\".pdf\", \".docx\", \".txt\", \".md\"}  def push_folder(folder_path: str, base_url: str = \"http://127.0.0.1:5328\"):     \"\"\"Push all supported files in a folder to the knowledge base.\"\"\"     files = [         os.path.join(folder_path, f)         for f in os.listdir(folder_path)         if os.path.splitext(f)[1].lower() in SUPPORTED_EXTENSIONS     ]      print(f\"Found {len(files)} file(s) to push...\")     total_chunks = 0     failed = []      for filepath in files:         try:             result = push_file(filepath, base_url)             total_chunks += result.get(\"chunks_added\", 0)             time.sleep(0.5)  # brief pause between files — avoid overwhelming the server         except Exception as e:             print(f\"[FAIL] {filepath}: {e}\")             failed.append(filepath)      print(f\"\\nDone. Total chunks added: {total_chunks}\")     if failed:         print(f\"Failed ({len(failed)}): {failed}\")  # Usage push_folder(\"./dataset/papers/\") ```  ---  ## 3. Upload via Next.js proxy (from frontend/TypeScript)  The Next.js app proxies uploads — requests to `/api/ml/dataset/upload` from the browser go through Next.js to the FastAPI backend at port 5328. ```typescript // src/lib/knowledge.ts  export async function pushDocument(file: File): Promise<{   success: boolean;   chunks_added: number;   message: string; }> {   const formData = new FormData();   formData.append(\"file\", file);    const res = await fetch(\"/api/ml/dataset/upload\", {     method: \"POST\",     body: formData,     // Do NOT set Content-Type header manually — browser sets it with     // the correct multipart boundary automatically   });    if (!res.ok) {     const err = await res.json().catch(() => ({ detail: res.statusText }));     throw new Error(err.detail ?? \"Upload failed\");   }    return res.json(); }  // Usage in a React component async function handleFileChange(e: React.ChangeEvent<HTMLInputElement>) {   const file = e.target.files?.[0];   if (!file) return;    try {     const result = await pushDocument(file);     console.log(`Indexed ${result.chunks_added} chunks from ${file.name}`);   } catch (err) {     console.error(\"Upload failed:\", err);   } } ```  ---  ## 4. Validate the knowledge base state  After pushing, verify that the FAISS index and doc_store are in sync. ```python # scripts/validate_kb.py  import faiss import json import os  INDEX_PATH = \"api/ml/faiss_index.bin\" STORE_PATH = \"api/ml/doc_store.json\"  def validate():     if not os.path.exists(INDEX_PATH):         print(\"[ERROR] faiss_index.bin not found — nothing has been indexed yet.\")         return      index = faiss.read_index(INDEX_PATH)     print(f\"FAISS index: {index.ntotal} vectors, dimension={index.d}\")      with open(STORE_PATH, encoding=\"utf-8\") as f:         store = json.load(f)     print(f\"Doc store:   {len(store)} chunks\")      if index.ntotal != len(store):         print(\"[WARN] Mismatch! Index and doc_store are out of sync.\")         print(\"       Run reset_kb() then re-push all documents.\")     else:         print(\"[OK] Index and doc_store are in sync.\")      # Preview first 3 chunks     print(\"\\nSample chunks:\")     for i, (k, v) in enumerate(list(store.items())[:3]):         print(f\"  [{k}] {v[:120].strip()}...\")  validate() ```  ---  ## 5. Reset the knowledge base  Wipe all indexed documents and start fresh. Use when re-indexing from scratch or when index/doc_store are out of sync. ```python # scripts/reset_kb.py  import faiss import json import os  INDEX_PATH = \"api/ml/faiss_index.bin\" STORE_PATH = \"api/ml/doc_store.json\" DIMENSION  = 384  # must match all-MiniLM-L6-v2  def reset_kb(confirm: bool = False):     if not confirm:         print(\"Pass confirm=True to actually reset. This deletes ALL indexed data.\")         return      # Overwrite with a fresh empty index     index = faiss.IndexFlatL2(DIMENSION)     faiss.write_index(index, INDEX_PATH)      with open(STORE_PATH, \"w\", encoding=\"utf-8\") as f:         json.dump({}, f)      print(\"[OK] Knowledge base reset. faiss_index.bin and doc_store.json cleared.\")  reset_kb(confirm=True) ```  ---  ## 6. Inspect chunks for a specific document  After pushing, check what was actually stored for a given file by searching for a known phrase from its content. ```python import json  def search_store(query: str, store_path: str = \"api/ml/doc_store.json\", top_n: int = 5):     \"\"\"Simple keyword scan of stored chunks — useful for debugging ingestion.\"\"\"     with open(store_path, encoding=\"utf-8\") as f:         store = json.load(f)      hits = [         (k, v) for k, v in store.items()         if query.lower() in v.lower()     ][:top_n]      if not hits:         print(f\"No chunks found containing '{query}'\")         return      for chunk_id, text in hits:         print(f\"--- Chunk {chunk_id} ---\")         print(text[:300])         print()  # Usage — find chunks from a specific paper search_store(\"candidiasis\") search_store(\"machine learning\") ```  ---  ## 7.",
  "3": "Use when re-indexing from scratch or when index/doc_store are out of sync. ```python # scripts/reset_kb.py  import faiss import json import os  INDEX_PATH = \"api/ml/faiss_index.bin\" STORE_PATH = \"api/ml/doc_store.json\" DIMENSION  = 384  # must match all-MiniLM-L6-v2  def reset_kb(confirm: bool = False):     if not confirm:         print(\"Pass confirm=True to actually reset. This deletes ALL indexed data.\")         return      # Overwrite with a fresh empty index     index = faiss.IndexFlatL2(DIMENSION)     faiss.write_index(index, INDEX_PATH)      with open(STORE_PATH, \"w\", encoding=\"utf-8\") as f:         json.dump({}, f)      print(\"[OK] Knowledge base reset. faiss_index.bin and doc_store.json cleared.\")  reset_kb(confirm=True) ```  ---  ## 6. Inspect chunks for a specific document  After pushing, check what was actually stored for a given file by searching for a known phrase from its content. ```python import json  def search_store(query: str, store_path: str = \"api/ml/doc_store.json\", top_n: int = 5):     \"\"\"Simple keyword scan of stored chunks — useful for debugging ingestion.\"\"\"     with open(store_path, encoding=\"utf-8\") as f:         store = json.load(f)      hits = [         (k, v) for k, v in store.items()         if query.lower() in v.lower()     ][:top_n]      if not hits:         print(f\"No chunks found containing '{query}'\")         return      for chunk_id, text in hits:         print(f\"--- Chunk {chunk_id} ---\")         print(text[:300])         print()  # Usage — find chunks from a specific paper search_store(\"candidiasis\") search_store(\"machine learning\") ```  ---  ## 7. Common errors and fixes  | Error | Cause | Fix | |-------|-------|-----| | `400 Unsupported format` | File extension not in `.pdf .docx .txt .md` | Convert to supported format first | | `400 File kosong` | Text extraction returned empty string | Check if PDF is scanned (image-only) — needs OCR first | | `500 Internal Server Error` | `dev:py` not running, or dependency missing | Run `npm run dev:py` and check terminal for Python errors | | `WARN Mismatch index/store` | `save_state()` interrupted mid-write | Reset KB and re-push all documents | | Very slow embedding | Large file with many chunks | Normal — `all-MiniLM-L6-v2` on CPU does ~100–500 chunks/sec | | `connection refused port 5328` | FastAPI server not running | `npm run dev:py` in a separate terminal |  ---  ## 8. Chunking parameters reference  Defined in `api/processor/chunker.py`, called with: ```python chunks = split_into_chunks(text, chunk_size=1000, overlap=200) ```  | Parameter | Value | Effect | |-----------|-------|--------| | `chunk_size` | 1000 chars | Each chunk ≈ 150–200 words. Good for paragraph-level retrieval. | | `overlap` | 200 chars | 20% overlap prevents context loss at chunk boundaries. |  To tune for denser academic papers (longer context needed): ```python # In api/index.py — modify the call: chunks = split_into_chunks(text, chunk_size=1500, overlap=300) ```  ---  ## Quick reference  ```bash # Push one file (Python CLI) python -c \"from scripts.push import push_file; push_file('file.pdf')\"  # Push entire folder python -c \"from scripts.push import push_folder; push_folder('./dataset/')\"  # Validate KB state python scripts/validate_kb.py  # Reset KB (destructive!) python scripts/reset_kb.py  # Check FastAPI is running curl http://127.0.0.1:5328/api/ml/dataset/upload -X POST # Expected: 422 Unprocessable Entity (missing file field) = server is up ```",
  "4": "name: jurnalku-dataset-push description: \"Use this skill whenever the user wants to push, upload, or index documents into the Jurnalku AI Knowledge Base. Covers uploading PDF, DOCX, TXT, and MD files to the FastAPI endpoint at /api/ml/dataset/upload, which chunks the text and stores it in the FAISS vector index (faiss_index.bin) via Sentence-Transformers (all-MiniLM-L6-v2, 384-dim). Also covers batch pushing, validating upload responses, inspecting the FAISS index state, and resetting the knowledge base. Use this skill for any task involving the RAG pipeline's ingestion layer. Do NOT use for querying/searching the knowledge base (that is the retrieval side) or for modifying the rag_engine.py logic.\" license: Private — project-specific skill for Jurnalku Jurnalku Dataset Push — AI Knowledge Base Ingestion Overview The ingestion pipeline works in this sequence: File (PDF/DOCX/TXT/MD)   → POST /api/ml/dataset/upload   [FastAPI, api/index.py]   → extract_text_from_*()         [text extraction per format]   → split_into_chunks()           [api/processor/chunker.py, size=1000, overlap=200]   → model.encode()                [all-MiniLM-L6-v2, 384-dim vectors]   → index.add()                   [FAISS IndexFlatL2]   → save_state()                  [faiss_index.bin + doc_store.json] Persistence files live at: api/ml/faiss_index.bin — the FAISS vector index api/ml/doc_store.json — plain-text chunk store (id → text) 1. Single file upload (Python) Use this for pushing one document at a time via the FastAPI endpoint directly. import httpx def push_file(filepath: str, base_url: str = \"http://127.0.0.1:5328\") -> dict:     \"\"\"     Push a single PDF/DOCX/TXT/MD file to the knowledge base. Returns the JSON response with chunks_added count. \"\"\"     with open(filepath, \"rb\") as f:         filename = filepath.split(\"/\")[-1]         # Determine MIME type by extension         mime_map = {             \".pdf\":  \"application/pdf\",             \".docx\": \"application/vnd.openxmlformats-officedocument.wordprocessingml.document\",             \".txt\":  \"text/plain\",             \".md\":   \"text/markdown\",         }         ext = \".\" + filename.rsplit(\".\", 1)[-1].lower()         mime = mime_map.get(ext, \"application/octet-stream\")         response = httpx.post(             f\"{base_url}/api/ml/dataset/upload\",             files={\"file\": (filename, f, mime)},             timeout=120.0,  # large files need more time for embedding         )     response.raise_for_status()     result = response.json()     print(f\"[OK] {filename} → {result['chunks_added']} chunks added\")     return result # Usage push_file(\"dokumen_riset.pdf\") push_file(\"laporan.docx\") push_file(\"catatan.md\") 2. Batch upload (multiple files) Push a whole folder or list of files sequentially. FAISS IndexFlatL2 is not thread-safe for concurrent writes — always push one file at a time. import os import httpx import time SUPPORTED_EXTENSIONS = {\".pdf\", \".docx\", \".txt\", \".md\"} def push_folder(folder_path: str, base_url: str = \"http://127.0.0.1:5328\"):     \"\"\"Push all supported files in a folder to the knowledge base.\"\"\"     files = [         os.path.join(folder_path, f)         for f in os.listdir(folder_path)         if os.path.splitext(f)[1].lower() in SUPPORTED_EXTENSIONS     ]     print(f\"Found {len(files)} file(s) to push...\")     total_chunks = 0     failed = []     for filepath in files:         try:             result = push_file(filepath, base_url)             total_chunks += result.get(\"chunks_added\", 0)             time.sleep(0.5)  # brief pause between files — avoid overwhelming the server         except Exception as e:             print(f\"[FAIL] {filepath}: {e}\")             failed.append(filepath)     print(f\"\\nDone. Total chunks added: {total_chunks}\")     if failed:         print(f\"Failed ({len(failed)}): {failed}\") # Usage push_folder(\"./dataset/papers/\") 3. Upload via Next.js proxy (from frontend/TypeScript) The Next.js app proxies uploads — requests to /api/ml/dataset/upload from the browser go through Next.js to the FastAPI backend at port 5328. // src/lib/knowledge.ts export async function pushDocument(file: File): Promise<{   success: boolean;   chunks_added: number;   message: string; }> {   const formData = new FormData();   formData.append(\"file\", file);   const res = await fetch(\"/api/ml/dataset/upload\", {     method: \"POST\",     body: formData,     // Do NOT set Content-Type header manually — browser sets it with     // the correct multipart boundary automatically   });   if (!res.ok) {     const err = await res.json().catch(() => ({ detail: res.statusText }));     throw new Error(err.detail ?? \"Upload failed\");   }   return res.json(); } // Usage in a React component async function handleFileChange(e: React.ChangeEvent<HTMLInputElement>) {   const file = e.target.files?.[0];   if (!file) return;   try {     const result = await pushDocument(file);     console.log(`Indexed ${result.chunks_added} chunks from ${file.name}`);   } catch (err) {     console.error(\"Upload failed:\", err);   } } 4. Validate the knowledge base state After pushing, verify that the FAISS index and doc_store are in sync. # scripts/validate_kb.py import faiss import json import os INDEX_PATH = \"api/ml/faiss_index.bin\" STORE_PATH = \"api/ml/doc_store.json\" def validate():     if not os.path.exists(INDEX_PATH):         print(\"[ERROR] faiss_index.bin not found — nothing has been indexed yet.\")         return     index = faiss.read_index(INDEX_PATH)     print(f\"FAISS index: {index.ntotal} vectors, dimension={index.d}\")     with open(STORE_PATH, encoding=\"utf-8\") as f:         store = json.load(f)     print(f\"Doc store:   {len(store)} chunks\")     if index.ntotal != len(store):         print(\"[WARN] Mismatch! Index and doc_store are out of sync.\")         print(\"       Run reset_kb() then re-push all documents.\")     else:         print(\"[OK] Index and doc_store are in sync.\")     # Preview first 3 chunks     print(\"\\nSample chunks:\")     for i, (k, v) in enumerate(list(store.items())[:3]):         print(f\"  [{k}] {v[:120].strip()}...\") validate() 5. Reset the knowledge base Wipe all indexed documents and start fresh. Use when re-indexing from scratch or when index/doc_store are out of sync. # scripts/reset_kb.py import faiss import json import os INDEX_PATH = \"api/ml/faiss_index.bin\" STORE_PATH = \"api/ml/doc_store.json\" DIMENSION  = 384  # must match all-MiniLM-L6-v2 def reset_kb(confirm: bool = False):     if not confirm:         print(\"Pass confirm=True to actually reset. This deletes ALL indexed data.\")         return     # Overwrite with a fresh empty index     index = faiss.IndexFlatL2(DIMENSION)     faiss.write_index(index, INDEX_PATH)     with open(STORE_PATH, \"w\", encoding=\"utf-8\") as f:         json.dump({}, f)     print(\"[OK] Knowledge base reset. faiss_index.bin and doc_store.json cleared.\") reset_kb(confirm=True) 6. Inspect chunks for a specific document After pushing, check what was actually stored for a given file by searching for a known phrase from its content. import json def search_store(query: str, store_path: str = \"api/ml/doc_store.json\", top_n: int = 5):     \"\"\"Simple keyword scan of stored chunks — useful for debugging ingestion.\"\"\"     with open(store_path, encoding=\"utf-8\") as f:         store = json.load(f)     hits = [         (k, v) for k, v in store.items()         if query.lower() in v.lower()     ][:top_n]     if not hits:         print(f\"No chunks found containing '{query}'\")         return     for chunk_id, text in hits:         print(f\"--- Chunk {chunk_id} ---\")         print(text[:300])         print() # Usage — find chunks from a specific paper search_store(\"candidiasis\") search_store(\"machine learning\") 7. Common errors and fixes 8. Chunking parameters reference Defined in api/processor/chunker.py, called with: chunks = split_into_chunks(text, chunk_size=1000, overlap=200) To tune for denser academic papers (longer context needed): # In api/index.py — modify the call: chunks = split_into_chunks(text, chunk_size=1500, overlap=300) Quick reference # Push one file (Python CLI) python -c \"from scripts.push import push_file; push_file('file.pdf')\" # Push entire folder python -c \"from scripts.push import push_folder; push_folder('./dataset/')\" # Validate KB state python scripts/validate_kb.py # Reset KB (destructive!) python scripts/reset_kb.py # Check FastAPI is running curl http://127.0.0.1:5328/api/ml/dataset/upload -X POST # Expected: 422 Unprocessable Entity (missing file field) = server is up"
}