README.md

metadata

title: KnowYourRepo
sdk: docker
app_port: 7860

Document Search System

React + FastAPI demo app for indexing document sources, searching them semantically, opening the original source file, and chatting with the retrieved document context.

Supported demo sources:

• Public GitHub repository URLs, such as https://github.com/owner/repo
• Public Google Drive file and folder links
• Manual uploads through the web UI
• Local demo files in data/raw

Current Google Drive note: public folder ingestion uses gdown, which is good for demos. Private folders or per-user permissions should use the Google Drive API with OAuth.

Setup

Install dependencies:

pip install -r requirements.txt

Run Ollama and pull the models you want to use:

ollama pull bge-m3
ollama pull llama3.2:3b

Optional environment overrides:

EMBEDDING_PROVIDER=ollama
EMBEDDING_MODEL=bge-m3:567m
EMBEDDING_DIMENSION=1024
CHAT_MODEL=llama3.2:3b
VECTOR_DB_BACKEND=chroma

For Zilliz Cloud, put this in .env:

VECTOR_DB_BACKEND=zilliz
ZILLIZ_URI=your-zilliz-endpoint
ZILLIZ_TOKEN=your-zilliz-token
COLLECTION_NAME=vectorEMBD
SUPABASE_URL=your-supabase-project-url
SUPABASE_ANON_KEY=your-supabase-anon-key
ANONYMOUS_REPO_LIMIT_MB=100
EMBEDDING_PROVIDER=huggingface
EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
EMBEDDING_DIMENSION=384
CHAT_PROVIDER=groq
CHAT_MODEL=llama-3.1-8b-instant
GROQ_API_KEY=your-groq-api-key

.env is ignored by git because it contains secrets.

Start the app:

uvicorn app.api.main:app --reload --host 127.0.0.1 --port 8010

Run the React frontend during local development:

cd frontend
npm install
npm run dev

Build React for FastAPI to serve:

cd frontend
npm run build

Scheduled Vector Cleanup

Expired vectors are ignored during search, and the deployed cleanup endpoint physically deletes expired rows:

POST /api/cleanup-expired-vectors

GitHub Actions calls the Hugging Face Space cleanup endpoint every 15 minutes from:

.github/workflows/cleanup-expired-vectors.yml

For protected cleanup, set the same secret in both places:

• Hugging Face Space secret: CLEANUP_SECRET
• GitHub repository secret: CLEANUP_SECRET

Verify Vector Storage

Check which vector store is active and how many chunks are stored:

python scripts/check_vector_store.py

Run a quick search against the active vector store:

python scripts/check_vector_store.py --query "machine learning"

How It Works

1. A user provides a source link or uploads files.
2. The app extracts supported documents.
3. Text is chunked and embedded with Ollama.
4. Chunks and metadata are stored in ChromaDB.
5. Search returns relevant chunks grouped by original document.
6. The UI shows excerpts and an Open source or Download file action.
7. The chat panel answers follow-up questions using the most recent retrieved chunks.

User Isolation

Supabase Auth is used for sign in/sign up. Every ingested chunk is tagged with:

user_id
source_id
document_id

Search filters by user_id, so each signed-in user only retrieves their own indexed chunks.

Anonymous users can index GitHub repositories up to ANONYMOUS_REPO_LIMIT_MB. Larger repositories require sign-in. Anonymous indexing uses a temporary browser-session user ID, so it is intended for short-lived exploration rather than persistent workspaces.

Migration

This project is being moved toward a deployable source-connected demo. See MIGRATION.md for the current architecture, cloud backend plan, and the next vector database migration step.