PROJECT_DOCUMENTATION.md

ChatBoog: Space Satellite Assistant - Project Documentation

1) Problem Statement

The goal of this project is simple and public-facing: make satellite data from every country accessible through a chat interface. Users should be able to ask questions in natural language and get accurate, grounded answers.

We chose Gunter's Space Page as the source because it is comprehensive, open, and consistently structured:

• Source: https://space.skyrocket.de/directories/sat_c.htm

2) Why Scraping (Not API / Dataset)

We evaluated three ways to collect the data:

1. Public API - none available for this exact dataset.
2. Pre-built dataset - not complete and often outdated or locked.
3. Web scraping - free, reliable for this site, and under our control.

We selected scraping because it is cost-effective, repeatable, and gives us the most complete coverage.

3) Website Structure (How We Navigate the Data)

The site exposes satellite data through a 4-level structure. Understanding this structure is the key to correct scraping.

1. Main directory (all countries)
   • Contains every country name and link.
2. Country page
   • Lists all satellite categories for that country.
3. Category page
   • Lists satellites (names, sometimes operators) with links to the satellite pages.
4. Satellite page
   • Contains the full detail: description, specifications, launch history, and metadata.

This structure informed the exact scraper design.

4) Data Collection Strategy (Step-by-Step)

We intentionally built the pipeline in small, reliable steps first, then combined it into a fast final scraper.

Step 1 - Countries

We first scraped all countries and their links. This gives the root of the tree.

Step 2 - Categories

For the target country (initially China), we scraped all categories and their links.

Step 3 - Satellites

For each category, we captured:

• Satellite name
• Operator (if available)
• Link to the satellite details page

Step 4 - Satellite Details

We then scraped each satellite detail page to collect:

• Description
• Specifications table
• Launch history
• Images

This breakdown made the process understandable and easier to validate.

5) Selenium First, Then BeautifulSoup

We initially used Selenium to explore and verify the structure. Selenium is heavy but good for discovery:

• Opens a real browser
• Validates what is visible
• Helps debug layout and missing fields

Once the structure was clear, we switched to BeautifulSoup for production because:

• Faster
• Lightweight
• More stable for bulk scraping

The experimental and step scripts are kept in development_logs/ for reference and auditing.

6) Why SQLite for Intermediate Storage

We store scraped links and metadata in SQLite because:

• It is simple and fast for local use
• Easy to query and inspect
• Great for pipeline checkpoints
• No external database required

What is stored in SQLite

We store:

• Countries and their links
• Categories and their links per country
• Satellites with name, category, operator, and detail page URL

This allows us to restart scraping without repeating earlier steps.

7) Final Scraper (Production Grade)

The final scraper is in src/full_scraper.py. It:

• Reads all satellite links from SQLite
• Fetches each satellite page using a persistent requests session
• Extracts structured data reliably
• Stores everything into one clean JSON file

What the final JSON contains

Each satellite record includes:

• id, name, country, category, operator, url
• description
• specifications (parsed from the #satdata table)
• launch_history (parsed from the #satlist table)
• images

This is the core dataset used for the RAG pipeline.

8) Why JSON for Final Output

We store all satellite details in a single JSON file because:

• The dataset is small enough to load quickly
• It is portable and easy to parse
• It integrates cleanly with RAG chunking tools

This is the final source of truth for embedding and retrieval.

9) RAG Pipeline (Chunking + Embeddings)

Once we have clean satellite data, we convert it into a knowledge base for semantic search.

9.1 Document Formatting

In src/build_rag_index.py, each satellite is converted into a structured Markdown document:

• Title includes satellite name
• Country and operator are included at the top
• Description section
• Specifications section (with explicit key re-labeling for search clarity)
• Launch history section

Why this matters:

• Markdown adds clear structure for chunking
• Injecting the satellite name into each spec line improves semantic matching
• Launch details are normalized into consistent sentences for retrieval

9.2 Chunking Strategy

We split each Markdown document using:

• RecursiveCharacterTextSplitter
• chunk_size=1000, chunk_overlap=200
• Separators: headings, paragraphs, lines, spaces

Why this matters:

• Prevents context overflow
• Keeps related information together
• Improves recall during retrieval

9.3 Embeddings

We use BAAI/bge-small-en-v1.5 because:

• Strong semantic search performance
• Lightweight and fast for local use
• Normalized embeddings improve similarity search

9.4 Vector Storage (ChromaDB)

We store embeddings in ChromaDB with local persistence:

• Collection name: satellites
• Vector size: 384 (matches BGE-small)
• Distance: cosine

Why ChromaDB:

• Native integration with LangChain
• tailored for local/embedded usage
• Stability: Handles file locking better than Qdrant in stateless/ephemeral environments like Hugging Face Spaces.

10) Chatbot Logic (Semantic Retrieval + LLM)

The chatbot runs in Streamlit (src/app.py) and uses:

• ChromaDB retriever for relevant chunks
• Groq LLM (llama-3.3-70b-versatile) for answer generation

Prompt Design (Why it works)

The prompt explicitly enforces:

• Precision with numbers and technical fields
• Honest fallback if data is missing
• Use of provided context only (avoid hallucinations)

This keeps answers grounded and accurate.

11) Testing and Quality Checks

We include tests/test_rag.py to validate:

• Model initialization
• Retrieval quality
• Hallucination resistance for out-of-scope questions

This provides a repeatable sanity check for the RAG system.

12) Deployment and Reproducibility

We support containerized deployment with a "Build-on-Start" strategy to handle large data files:

1. Lazy Indexing (Self-Healing):

   • The application (src/app.py -> src/rag_engine.py) automatically checks if the ChromaDB index exists/is empty on startup.
   • If empty (first run on cloud), it triggers src/build_rag_index.py to rebuild the index from the JSON data.
   • This bypasses the need to push large binary database files (.sqlite3, .bin) to git, avoiding Git LFS quotas and rejection errors.

2. Container Setup:

   • Dockerfile sets up the environment, including start.sh handling permissions.
   • .gitignore explicitly excludes data/chroma_db to ensure a clean slate for deployment.
   • .github/workflows/sync_to_huggingface.yml handles the sync to Hugging Face Spaces.

13) End-to-End Flow (Project Diagram)

Main Page (countries)
|- Country Page
|  |- Category Page
|     |- Satellite List (name + operator + link)
|        |- Satellite Detail Page (full data)
|        |- JSON Output
|     |- SQLite index for tracking

14) What We Have Achieved

We now have:

• A verified scraper pipeline (BeautifulSoup)
• Clean, structured satellite JSON data
• A reproducible RAG pipeline (chunking + embeddings)
• A working Streamlit chat UI
• Docker and Hugging Face deployment readiness

15) Why This Approach Works

• Scalable: We can add more countries easily
• Reliable: Stored checkpoints in SQLite
• Cost-effective: No paid APIs
• Accurate: Data comes directly from the source
• RAG-ready: JSON -> chunking -> embeddings -> ChromaDB

16) Next Steps (Optional)

• Expand scraping from China to all countries
• Add scheduled refresh jobs
• Add evaluation metrics for RAG accuracy
• Add UI filters (country, category)

---

Files Referenced

• src/full_scraper.py
• src/build_rag_index.py (includes build_index entry point)
• src/rag_engine.py (lazy indexing logic)
• src/app.py
• tests/test_rag.py
• Dockerfile
• start.sh
• README.md
• development_logs/ (Contains legacy scripts: diagnose.py, evaluate_rag.py, etc.)

---

If you want, I can also generate a shorter version for README or a slide-friendly summary.