README.md

metadata

title: BPL RAG Spring 2026
emoji: 📚
colorFrom: blue
colorTo: green
sdk: streamlit
sdk_version: 1.32.0
app_file: app.py
pinned: false
license: mit

Boston Public Library - RAG Search System (Spring 2026)

Natural language search for the BPL Digital Commonwealth collection using Retrieval-Augmented Generation.

🎯 Project Overview

This system allows users to search through the Boston Public Library's digital collections using natural language queries. Built as part of the BU Spark! DS549 course in Spring 2026.

Key Features

• Semantic Search: Natural language queries across 445K+ BPL items
• Full-Text Search: Search within document content, not just metadata
• Metadata Filtering: Time-based, location-based, and type-based filtering
• Evaluation Framework: Built-in testing with gold-standard datasets
• PostgreSQL + pgvector: Scalable vector database backend

🚀 Quick Start

1. Enter your query in natural language (e.g., "What were important events in Boston in 1919?")
2. View retrieved documents with relevance scores
3. Read AI-generated contextual explanations

📊 Example Queries

• "Find pictures of JFK's house on Cape Cod"
• "Are there any maps of Worcester, MA from the 18th century?"
• "Show me depictions of indigenous Americans"
• "What were important historical events in Boston in 1919?"

🛠️ Technical Stack

• Frontend: Streamlit
• Database: PostgreSQL with pgvector extension
• Embeddings: Sentence Transformers
• LLM: OpenAI GPT-4 / Anthropic Claude
• Retrieval: BM25 + Vector Search with metadata filtering
• Evaluation: DeepEval framework

📁 Project Structure

current_spring2026/
├── app.py                      # Main Streamlit application
├── pipeline.py                 # RAG pipeline orchestration
├── config.py                   # Configuration management
├── database/                   # Database connection & queries
├── embedding/                  # Vector embeddings
├── retrieval/                  # Document retrieval logic
├── generation/                 # LLM response generation
├── evaluation/                 # Testing & metrics
├── ingestion/                  # Data processing pipeline
└── scripts/                    # Utility scripts

🔐 Environment Variables

Required secrets (set in Space Settings → Repository secrets):

OPENAI_API_KEY=your_openai_key
DATABASE_URL=postgresql://user:pass@host:port/dbname

📈 Spring 2026 Improvements

Building on Fall 2025 work, this version adds:

• ✅ Full-text document search (not just metadata)
• ✅ Gold-standard evaluation dataset
• ✅ Structured logging of queries and metrics
• ✅ Improved retrieval metrics (Precision, Recall, MRR)
• ✅ Enhanced UI with developer debug view

📚 Data

• Source: Digital Commonwealth (BPL subset)
• Records: ~445,000 items
• Types: Photographs, maps, newspapers, manuscripts, books
• Full-text: 210K items with OCR (~1.5M pages)

👥 Team

Spring 2026 Spark! Team

• Boston University Data Science (DS549)
• Client: Eben English, Boston Public Library

Previous Semesters

• Fall 2025: Infrastructure & pgvector migration
• Fall 2024: Initial RAG prototype

📄 License

MIT License

🔗 Links

• Digital Commonwealth
• BPL Digital Repository
• Project Documentation

---

Built with ❤️ by BU Spark! for Boston Public Library