Spaces:

shankarsain
/

build-small-hackathon

Sleeping

App Files Files Community

build-small-hackathon / README.md

shankarsain

docs: add Hugging Face Space metadata to README.md

c13e599 about 1 month ago

preview code

Raw

History Blame Contribute Delete

16.9 kB

metadata

title: VidyaBot Gradio
emoji: 📚
colorFrom: yellow
colorTo: green
sdk: docker
app_port: 7860

VidyaBot Gradio Edition — Offline AI Study Partner

Offline-first, cost-optimized AI tutor for rural Indian students powered by local Ollama inference and an advanced 5-Stage Context Pruning pipeline.

🎖️ Build Small 2026 Merit Badges

We have successfully earned all 5 merit badges for the Build Small 2026 hackathon:

🔌 Off the Grid — No cloud APIs used; runs fully offline with local Ollama/FastAPI backend
🦙 Llama Champion — Runs via standard llama.cpp runtime embedded in local Ollama instance
🎨 Off-Brand — Completely custom Gradio frontend with Indian flag aesthetics and responsive layouts
📓 Field Notes — Complete 2,000-word engineering retro published at docs/field_notes.md
🎯 Well-Tuned — Mistral 7B fine-tuned on 103 student Q&A pairs via Modal A10G GPU (LoRA/QLoRA)

🎯 Model: Fine-Tuned Mistral 7B (`mistral-vidyabot`)

VidyaBot uses Mistral 7B Instruct fine-tuned on student Q&A pairs from NCERT Class 10 curriculum. The fine-tuned model is served 100% offline via Ollama (llama.cpp runtime), maintaining the Off the Grid badge while improving answer quality.

Fine-Tuning Details

Detail	Value
Base model	`mistralai/Mistral-7B-Instruct-v0.1`
Training data	103 hand-crafted + synthetic NCERT Q&A pairs
Method	QLoRA (4-bit quantization + LoRA adapters)
LoRA config	r=8, alpha=16, targets: q/v/k/o_proj
Hardware	Modal A10G GPU (24GB VRAM)
Training time	~1–2 hours
Estimated cost	~$3–5 from $250 Modal credits
Inference	GGUF Q4_K_M via Ollama (CPU-only, ~4GB RAM)
Ollama model name	`mistral-vidyabot`

Why Fine-Tune on Educational Q&A?

Base mistral:latest is a strong general-purpose model, but fine-tuning on NCERT-aligned Q&A pairs produces:

✅ More structured answers — consistent 2-4 sentence format
✅ Better NCERT terminology — uses the exact textbook language students recognise
✅ Curriculum-aware responses — references chapter context and exam-relevant concepts
✅ Bilingual support — trained on Hindi-language Q&A pairs

Fine-Tuning Pipeline

Student Q&A Data (103 pairs)
    ↓
[Modal A10G GPU]
    QLoRA: Mistral-7B-Instruct + LoRA adapters (r=8)
    3 epochs, DataCollatorForLanguageModeling
    ↓ Merge LoRA into base (merge_and_unload)
    ↓ Save full merged model → Modal Volume
    ↓
[local: modal_convert_gguf.py]
    Convert HF safetensors → GGUF (llama.cpp)
    Q4_K_M quantization (~4GB)
    ↓
[Ollama]
    ollama create mistral-vidyabot -f Modelfile
    → Offline inference at 4-8 tokens/sec (CPU)

Reproduce the Fine-Tuning

# Step 1: Generate dataset (needs Ollama running with mistral:latest)
python data/finetuning/generate_synthetic_qa.py

# Step 2: Submit to Modal (needs modal account + credits)
modal run modal_finetune.py

# Step 3: Download + convert to GGUF + register in Ollama
python modal_convert_gguf.py

# Step 4: Test the fine-tuned model
ollama run mistral-vidyabot "What is photosynthesis?"

🎯 Problem Statement

Over 200 million Indian students use textbooks from national and state boards (NCERT, CBSE, SSLC, etc.), but face:

Limited or unstable internet connectivity in small towns and villages
High cost of cloud APIs ($0.77+ per question using naive RAG baselines)
Language barriers (need for Hindi, Kannada, Telugu, Tamil, Marathi, etc.)
Need for absolute hardware resilience (must run on older 8GB-16GB RAM CPU laptops)

VidyaBot solves this by wrapping a local quantized LLM with a 5-Stage Context Pruning pipeline that achieves 88.2% input token reduction, allowing CPU inference to run in less than 2 seconds with $0.00 API costs.

🏗️ Architecture Diagram

                              STUDENT QUERY
                                    │
                                    ▼
                     ┌──────────────────────────────┐
                     │ STAGE 0: Curriculum Router   │
                     │  - Eliminates 70% chapters   │
                     │  - Zero cost | Latency <1ms  │
                     └──────────────┬───────────────┘
                                    │
                                    ▼
                     ┌──────────────────────────────┐
                     │ STAGE 1: BM25 Filter         │
                     │  - Keyword pre-filtering     │
                     │  - Top-30 candidate chunks   │
                     └──────────────┬───────────────┘
                                    │
                                    ▼
                     ┌──────────────────────────────┐
                     │ STAGE 2: Cross-Encoder       │
                     │  - ms-marco-MiniLM-L-6-v2    │
                     │  - Joint scoring | Top-5     │
                     └──────────────┬───────────────┘
                                    │
                                    ▼
                     ┌──────────────────────────────┐
                     │ STAGE 3: Token Budget        │
                     │  - Hard 512-token context cap│
                     └──────────────┬───────────────┘
                                    │
                                    ▼
                     ┌──────────────────────────────┐
                     │ STAGE 4: Sentence Pruner     │
                     │  - Similarity-based trimming │
                     │  - 30-50% text reduction     │
                     └──────────────┬───────────────┘
                                    │
                                    ▼
                    ┌───────────────────────────────┐
                    │  OLLAMA LOCAL INFERENCE (CPU)  │
                    │  - Model: mistral-vidyabot    │
                    │  - Cost: $0.00 | TTFT: <2s    │
                    └───────────────┬───────────────┘
                                    │
                                    ▼
                             STUDENT ANSWER

💾 Tech Stack

Layer	Technology	Why
Frontend	Gradio (Blocks UI)	Premium Indian themed interface with streaming and dashboards
Backend	Python 3.11 + FastAPI	Lightweight asynchronous API server
Inference Engine	Ollama (`llama.cpp` runtime)	Fast local inference on consumer CPU hardware
Embeddings	sentence-transformers (`all-MiniLM-L6-v2`)	384D, CPU-only, 22MB model
Reranker	Cross-Encoder (`ms-marco-MiniLM-L-6-v2`)	80MB model, joint scoring for 15-25% more precision
PDF Processing	pdfplumber + PyMuPDF	Robust layout-aware textbook text extraction
Vector Search	FAISS (`IndexFlatIP`)	Sub-millisecond local semantic search
Database	SQLite	Single `.db` file for student metadata and caching
Translation	deep-translator	Multi-language support (free tier)

📁 Project Structure

vidyabot/
├── backend/
│   ├── main.py                      # FastAPI entry point & routers
│   ├── config.py                    # Settings & env loading
│   ├── database.py                  # SQLite schema & DTOs
│   │
│   ├── ingestion/
│   │   ├── pdf_parser.py            # PDF -> structured text
│   │   ├── chunker.py               # Semantic chunking
│   │   └── embedder.py              # MiniLM embeddings generator
│   │
│   ├── retrieval/
│   │   ├── bm25_index.py            # Stage 1: BM25 indexer
│   │   ├── vector_store.py          # Stage 2: FAISS vector store
│   │   ├── reranker.py              # Stage 2: Cross-Encoder reranker
│   │   ├── sentence_pruner.py       # Stage 4: Sentence trimmer
│   │   └── context_pruner.py        # 5-stage orchestrator (CORE)
│   │
│   ├── llm/
│   │   ├── ollama_client.py         # Local Ollama client (offline)
│   │   └── prompt_builder.py        # Prompt formatting
│   │
│   └── cache/
│       └── semantic_cache.py        # FAISS-based query cache
│
├── docs/
│   ├── field_notes.md               # 2000-word engineering retrospective
│   └── social_post.md               # Social media post drafts
│
├── data/                            # Local databases and PDF storage
├── gradio_app.py                    # Gradio blocks application layout
├── app.py                           # Unified Gradio + FastAPI launcher
├── space_requirements.txt           # HF Space requirements file
└── README.md                        # This file

🚀 Quick Start (Running Offline)

1. Pre-requisites

Python 3.11+
Ollama installed on your machine.

Download the local target model:

ollama serve
ollama pull llama3.2:latest

2. Clone & Setup

git clone https://github.com/shankarsai000/Paradox-vidyabot.git
cd Paradox-vidyabot

# Create and activate virtual environment
python -m venv venv
venv\Scripts\activate  # Unix: source venv/bin/activate

# Install dependencies
pip install -r backend/requirements.txt

3. Configure Env

Create a .env file in the root directory:

LLM_BACKEND=ollama
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=mistral-vidyabot

4. Start Unified Application

$env:PYTHONPATH="."
python app.py

*Gradio interface will launch at http://localhost:7860. FastAPI routes will run at /api.*


✅ Ready to use! Ask questions about your textbooks.

---

## 📊 API Reference

### Ingestion

**POST /api/ingest** — Upload & process PDF

```bash
curl -F "file=@textbook.pdf" \
     -F "board=CBSE" \
     -F "subject=Biology" \
     -F "grade=10" \
     -F "title=Biology Class 10" \
     http://localhost:8000/api/ingest

Response:

{
  "status": "success",
  "textbook_id": 1,
  "total_chunks": 442,
  "processing_time_seconds": 28
}

GET /api/textbooks — List available textbooks

{
  "textbooks": [
    {
      "id": 1,
      "title": "Biology Class 10",
      "board": "CBSE",
      "subject": "Biology",
      "grade": "10",
      "total_pages": 256,
      "total_chunks": 442
    }
  ]
}

Query & LLM

POST /api/query — Answer a question

curl -X POST http://localhost:8000/api/query \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What is photosynthesis?",
    "textbook_id": 1,
    "language": "english",
    "mode": "answer"
  }'

Response:

{
  "answer": "Photosynthesis is the process by which plants...",
  "tokens_used": 387,
  "baseline_tokens": 2000,
  "tokens_saved": 1613,
  "cost_usd": 0.000097,
  "cost_saved_usd": 0.000403,
  "cache_hit": false,
  "pruning_ratio": 0.807,
  "time_ms": 1250,
  "source_pages": "45,46"
}

Analytics

GET /api/stats — Cumulative cost dashboard

{
  "total_queries": 1547,
  "cache_hits": 621,
  "cache_hit_rate": 0.401,
  "total_tokens_used": 598818,
  "total_baseline_tokens": 3094000,
  "total_tokens_saved": 2495182,
  "total_cost_usd": 0.1497,
  "baseline_cost_usd": 0.7735,
  "total_savings_usd": 0.6238,
  "savings_percentage": 80.7,
  "avg_tokens_per_query": 387,
  "textbooks_ingested": 3
}

🎓 How Cost Savings Work

Baseline (Full Textbook to LLM)

Input: Entire chapter (~2000 tokens)
Cost: 2000 tokens × ($0.25/1M) = $0.0005/query
Per 1000 queries: $0.50

VidyaBot (Pruned Context)

Input: Relevant chunks only (~400 tokens)
Stages:
1. BM25 filter: top-30 chunks (0ms, free)
2. Semantic rerank: top-10 chunks (5ms, local MiniLM)
3. Token budget: top-3 chunks (0ms, local logic)
Cost: 400 tokens × ($0.25/1M) = $0.0001/query
Per 1000 queries: $0.10

Result

Savings = $0.50 - $0.10 = $0.40 per 1000 queries
Percentage = (0.40 / 0.50) × 100 = 80% reduction

At scale: Serving 100,000 students each asking 10 questions = $20,000 saved vs cloud alternatives.

🧪 Running Tests

# Install pytest
pip install pytest

# Run all tests
pytest tests/ -v

# Run specific test file
pytest tests/test_pruning.py -v

# Run with coverage
pytest tests/ --cov=backend

Test Coverage

✅ PDF parsing & chunking
✅ 3-stage pruning pipeline
✅ Semantic cache deduplication
✅ Edge cases (empty inputs, long texts, etc.)

🌍 Languages Supported

VidyaBot works with Indian languages via deep-translator:

English (default)
हिंदी (Hindi)
ಕನ್ನಡ (Kannada)
తెలుగు (Telugu)
தமிழ் (Tamil)
मराठी (Marathi)
বাংলা (Bengali)

How it works:

Student asks in their language
Question translated to English (free Google Translate)
Answer fetched from English textbook
Answer translated back to student's language

🔐 Security & Privacy

✅ All data stays local:

SQLite DB stored locally (./data/vidyabot.db)
No user data sent to VidyaBot servers
Only LLM prompt + context sent to Anthropic

✅ Offline-first:

Service worker caches app shell
Can answer repeat questions offline
No tracking or analytics

✅ API key protection:

Never exposed in browser
Backend-only communication with Anthropic

📝 Adding New Textbooks

Via Web UI

Navigate to "📤 Upload" tab
Select PDF file
Fill in metadata
Click "Upload & Process"
Done! (Takes ~30 seconds per 300-page book)

Via CLI

python -c "
from backend.ingestion.pdf_parser import PDFParser
from backend.ingestion.chunker import Chunker
from backend.ingestion.embedder import Embedder

parser = PDFParser('path/to/book.pdf')
pages = parser.parse()

chunker = Chunker()
chunks = chunker.chunk_by_section(pages, textbook_id=1)

embedder = Embedder()
embedder.embed_chunks([c.content for c in chunks])
"

🛠️ Deployment

Local Development

To run the unified application (Gradio fronted + FastAPI backend):

$env:PYTHONPATH="."
python app.py

🐛 Troubleshooting

Issue	Solution
"Ollama connection refused"	Make sure the Ollama desktop application is open or `ollama serve` is running.
"Ollama model not found"	Run `ollama pull llama3.2:latest` (or model name specified in `.env`).
"No textbooks loaded"	Navigate to the "Upload Textbook" tab in the UI or use the API ingest route.
"Slow first query"	First query compiles indexes (~10-20s). Subsequent queries are extremely fast.
"PDF upload fails"	Ensure the uploaded PDF is a digital text-based document (not scanned images).
"Out of memory"	Quantized models (3B/7B) run safely inside 8GB RAM. Ensure other heavy applications are closed.

📚 Acceptance Criteria ✅

✅ POST /api/ingest returns total_chunks > 0 in <60 seconds
✅ POST /api/query returns answer with tokens_used < 600
✅ tokens_saved consistently >1000 (proving ~80% reduction)
✅ Second identical query returns cache_hit: true with tokens_used: 0
✅ Frontend loads, shows textbook selector, displays answer + savings badge
✅ GET /api/stats shows cumulative savings
✅ All tests pass (pytest tests/ -v)

📄 License

MIT License — Free for educational use.

🙏 Contributing

Contributions welcome! Focus areas:

Additional Indian languages
Mobile app (React Native)
Handwriting recognition for math
Teacher dashboard
Offline video integration

Made with ❤️ for education access across rural India.

"Not all children have access to tutors, but they should have access to knowledge."