README.md

metadata

title: IJNet Assistant
emoji: 🌍
colorFrom: blue
colorTo: indigo
sdk: streamlit
sdk_version: 1.57.0
app_file: app.py
pinned: false

🌍 IJNet Chatbot Assistant

A LangChain-powered RAG chatbot that helps journalists find training opportunities, fellowships, grants, tools, and expert guidance from the IJNet knowledge base.

Built as a prototype for IJNet's chatbot assistance system, using entirely free and open-source tools.

---

Features

• Hybrid Retrieval: Combines semantic search (FAISS) with keyword search (BM25) using Reciprocal Rank Fusion for robust document retrieval
• Smart Query Classification: Automatically detects query intent (deadline-based, region-filtered, topic-specific, newsletter, etc.) and applies appropriate retrieval strategies
• Metadata-Aware Filtering: Post-retrieval filtering on deadlines, regions, opportunity types, and topics
• Source Attribution: Every response cites its sources with titles, URLs, and metadata
• Multi-Turn Conversation: Maintains conversation history for follow-up questions
• Debug Mode: Toggle to inspect retrieval details (semantic scores, BM25 scores, query classification)
• Evaluation Suite: Automated tests for retrieval accuracy and query classification

---

Architecture

User Query
    │
    ▼
┌──────────────────┐
│ Query Classifier  │  → Detects intent, extracts filters, identifies boost keywords
└────────┬─────────┘
         │
    ┌────┴────┐
    ▼         ▼
┌───────┐ ┌──────┐
│ FAISS │ │ BM25 │    → Parallel semantic + keyword retrieval
│(top-8)│ │(top-8│
└───┬───┘ └──┬───┘
    │        │
    ▼        ▼
┌──────────────────┐
│  Reciprocal Rank │  → Fuses both rankings without tuned weights
│     Fusion       │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│ Metadata Filter  │  → Reranks by deadline proximity, region match, type match
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│   Groq LLM       │  → Llama 3.3 70B generates grounded response with citations
│  (Llama 3.3)     │
└────────┬─────────┘
         │
         ▼
   Answer + Sources

---

Tech Stack (All Free)

Component	Tool	Why
LLM	Groq (Llama 3.3 70B)	Free API, fast inference, high quality
Embeddings	HuggingFace all-MiniLM-L6-v2	Runs locally, no API cost, good quality
Vector Store	FAISS	In-memory, no server needed, fast similarity search
Keyword Search	BM25 (rank-bm25)	Complements semantic search for exact-term matching
Framework	LangChain	Industry-standard RAG orchestration
Frontend	Streamlit	Fast prototyping, built-in chat UI

---

Quick Start

1. Clone and install

git clone <repo-url>
cd ijnet-assistant
pip install -r requirements.txt

2. Get a Groq API key (free)

1. Go to console.groq.com
2. Sign up for a free account
3. Create an API key

3. Set your API key

export GROQ_API_KEY="gsk_your_key_here"

4. Build the vector index

python -m src.ingest

This loads the knowledge base, chunks documents, generates embeddings, and saves the FAISS index.

5. Run the evaluation (optional)

python evaluate.py

Validates retrieval accuracy (17 tests covering classification + retrieval).

6. Launch the chatbot

streamlit run app.py

Open http://localhost:8501 in your browser and start asking questions.

---

Example Queries

Query	What it tests
"What opportunities are available for investigative journalists in Africa?"	Region + topic filtering
"Find fellowships with deadlines in the next 30 days"	Deadline extraction + type filtering
"What resources does IJNet have on AI tools for journalists?"	Topic-based retrieval across articles + opportunities
"Can you summarize the latest opportunities for product/design people in newsrooms?"	Cross-document synthesis
"Which IJNet newsletter should I subscribe to?"	Specific article retrieval
"What grants are available for data journalism projects?"	Type + topic filtering
"Tell me about digital security tools for journalists"	Article retrieval

---

Project Structure

ijnet-assistant/
├── app.py                    # Streamlit chat interface
├── evaluate.py               # Retrieval accuracy tests
├── scraper.py                # IJNet public page scraper (optional)
├── requirements.txt
├── README.md
├── data/
│   ├── knowledge_base.json   # Mock IJNet corpus (20 opportunities + 6 articles)
│   └── faiss_index/          # Persisted FAISS vector store (generated)
└── src/
    ├── __init__.py
    ├── ingest.py             # Document loading, chunking, embedding
    ├── retriever.py          # Hybrid retrieval (FAISS + BM25 + RRF)
    └── chain.py              # RAG chain connecting retriever to Groq LLM

---

Design Decisions

Why Hybrid Retrieval?

Pure semantic search misses exact keyword matches (e.g., "Pulitzer Center" might not be the top semantic match for "reporting grants"). Pure keyword search misses meaning (e.g., "accountability journalism" wouldn't match "investigative reporting"). Combining both with Reciprocal Rank Fusion gives the best of both worlds without needing to tune relative weights.

Why One-Chunk-Per-Opportunity?

Each opportunity is a self-contained record (~200 words). Splitting it would lose context (e.g., separating eligibility from the description). Keeping it whole means the LLM always gets complete opportunity details.

Why Query Classification?

A naive "embed and retrieve" approach fails on queries like "fellowships with deadlines in the next 30 days" — the embedding doesn't understand calendar math. The classifier extracts structured filters (deadline window, region, type) and applies them post-retrieval, giving much better precision.

Why Low Temperature (0.1)?

This is a factual Q&A system. Higher temperatures increase creativity but also hallucination risk. At 0.1, the LLM sticks closely to the provided context.

---

Limitations & Future Improvements

Current Limitations

• Static corpus: The knowledge base is a fixed JSON file; in production, this would connect to IJNet's CMS/database
• Deadline reasoning: Deadlines are compared numerically; the LLM doesn't do real date arithmetic
• No re-ranking model: A cross-encoder re-ranker (e.g., ms-marco-MiniLM) would improve precision further
• Single language: Currently English-only; IJNet serves 8 languages

Potential Improvements

• Live data pipeline: Connect to IJNet's RSS feeds or API for real-time opportunity updates
• Cross-encoder re-ranking: Add a re-ranker between retrieval and generation for better precision
• Multi-language support: Use multilingual embeddings (paraphrase-multilingual-MiniLM-L12-v2) and add language detection
• Structured queries: For deadline/region filters, use SQL-like queries on a structured database alongside the vector search
• User personalization: Let users set preferences (region, topics, career stage) for personalized recommendations
• Evaluation on real queries: Collect user queries to build a proper evaluation set

---

License

MIT — built as a prototype for IJNet chatbot assistance.