docs(planning): Finalize project plans to reflect strategic pivots

Overhauls the initial and implementation plans to serve as final, annotated deliverables. This commit transforms the plans from simple task lists into a strategic narrative that documents the project's evolution and key decision-making processes.

The primary changes include:

- **Annotate Strategic Pivot:** Explicitly documents the critical decision to move from complex programmatic PDF extraction to a manually curated JSON knowledge base. This annotation explains the rationale, tying the decision to the "Bias for Action" principle and the goal of de-risking the project to focus on RAG quality.

- **Detail AI Collaboration Workflow:** Replaces the generic "AI as a Co-pilot" section with the specific, human-directed methodology used. This includes the multi-model planning process, the core technique of active context management, and the concept of the "virtuous cycle of code quality."

- **Reflect Stack Evolution:** Updates the technical stack to accurately represent the final implementation (e.g., direct implementation vs. LangChain) and clarifies the deployment path (Gradio app on Hugging Face Spaces vs. a formal REST API).

- **Improve Technical Precision:** Refines language to be more technically accurate, such as distinguishing between an "interactive web application" and a formal "REST API."

These finalized documents now provide a clear, professional, and honest account of the project's journey from initial concept to final implementation.

docs/implementation_log.md

@@ -0,0 +1,124 @@
+# FOT Intervention Recommender
+## Final Implementation Plan (Annotated)
+
+---
+
+## Overview
+
+This implementation plan documents the executable phases, tasks, and deliverables used to build the working proof-of-concept for the Freshman On-Track Intervention Recommender. This plan was updated from its original version to reflect the final, successful implementation path.
+
+**Primary Deliverable**: A working RAG system, deployed as an interactive web application, that provides evidence-based intervention recommendations.
+
+---
+
+## Phase 0: Environment Setup & Resource Gathering
+**Goal**: Establish a lean development environment and prepare all source materials.
+
+### Tasks
+
+#### 0.1 Development Environment Setup
+- [✅] Create local project structure (`pyproject.toml`, `src/`, `tests/`).
+- [✅] Configure a modern, fast dependency manager (`uv`).
+- [✅] Install core libraries: `torch`, `sentence-transformers`, `faiss-cpu`, `google-generativeai`, `gradio`.
+- [✅] Create a `.env` file for secure management of API keys.
+
+#### 0.2 Source Material Collection
+- [✅] Identify and download the primary source document (NCS FOT Toolkit) and five additional high-quality, evidence-based articles.
+- [✅] Manually extract and structure all relevant interventions from these sources into a clean, high-quality `knowledge_base_raw.json` file.
+- [✅] Create a `citations.json` file to store metadata for all source documents.
+
+### Success Criteria
+- ✅ Local development environment running with all simplified dependencies.
+- ✅ `knowledge_base_raw.json` and `citations.json` files are created, validated, and located in `data/processed/`.
+
+---
+
+## Phase 1: Knowledge Base Construction
+**Goal**: Load and semantically chunk the curated knowledge base to prepare it for embedding.
+
+> **_Strategic Pivot: From Programmatic Extraction to Curated Knowledge Base_**
+>
+> *   **Initial Approach:** My original plan detailed a complex pipeline to programmatically extract text and tables from source PDFs using tools like PyMuPDF and pdfplumber.
+> *   **Challenge & Insight:** I quickly identified this approach as a significant project risk. The complexity and unreliability of PDF parsing could easily consume the majority of development time, detracting from the core task: building a high-quality RAG system. The ultimate goal is to provide relevant recommendations, which depends entirely on the *quality* and *cleanliness* of the knowledge base, not the sophistication of the extraction method.
+> *   **Decision & Rationale:** In line with the "Bias for Action" and "Startup Urgency" principles, I made a strategic decision to pivot. I manually curated a high-quality `knowledge_base_raw.json` file, a process accelerated by using an LLM as a co-pilot. This action de-risked the project, guaranteed the highest possible quality for the RAG pipeline's input, and allowed me to focus my efforts on the more critical tasks of semantic chunking, embedding, and retrieval logic.
+> *   **Result:** This pivot resulted in a more robust and effective PoC. The final system is built on a foundation of clean, reliable data, directly leading to more relevant and trustworthy recommendations.
+
+### Tasks
+
+#### 1.1 Content Loading
+- [✅] Implement logic in `scripts/build_knowledge_base.py` to load the `knowledge_base_raw.json` file.
+
+#### 1.2 Content Processing & Chunking
+- [✅] Create a `src/fot_recommender/semantic_chunker.py` module to group related items from the raw JSON file.
+- [✅] Implement a `chunk_by_concept` strategy that combines page-based items into larger, topic-based chunks.
+
+#### 1.3 Knowledge Base Structuring
+- [✅] Define a final chunk structure, ensuring the output is a clean list of dictionaries, each containing `title`, `source_document`, `fot_pages`, and a combined `content_for_embedding` string.
+- [✅] Save the processed data as `knowledge_base_final_chunks.json`.
+
+### Success Criteria
+- ✅ `knowledge_base_raw.json` successfully loaded into the build script.
+- ✅ Semantic chunking logic correctly combines related pages into fewer, more coherent chunks.
+- ✅ A final `knowledge_base_final_chunks.json` file is produced and validated for quality.
+
+---
+
+## Phase 2: RAG Pipeline Implementation
+**Goal**: Build and test the core Retrieval-Augmented Generation functionality.
+
+### Tasks
+
+#### 2.1 Vector Embedding Setup
+- [✅] **Initialize embedding model**: In `rag_pipeline.py`, initialize `all-MiniLM-L6-v2` using the `sentence-transformers` library.
+- [✅] **Create embeddings**: Implement a function to create embeddings from the `content_for_embedding` field of each chunk.
+- [✅] **Set up FAISS vector database**: Implement `create_vector_db` to build an `IndexFlatIP` index from the embeddings and save it to `faiss_index.bin`.
+
+#### 2.2 Retrieval System
+- [✅] **Implement semantic search**: Create a `search_interventions` function that takes a query, embeds it, and uses the FAISS index to retrieve the top-k most relevant chunks.
+- [✅] **Test retrieval**: Validate that sample queries return relevant and high-scoring results.
+
+#### 2.3 Response Generation
+- [✅] **Implement Generative Model**: Use the `google-generativeai` library to call the Gemini API.
+- [✅] **Create persona-based prompts**: In `prompts.py`, create distinct, detailed prompts for 'teacher', 'parent', and 'principal' personas.
+- [✅] **Synthesize response**: Create a `generate_recommendation_summary` function that formats the retrieved chunks and user query into the selected persona's prompt and sends it to the Gemini API.
+
+### Success Criteria
+- ✅ Vector database successfully created with all intervention embeddings.
+- ✅ Semantic search returns relevant results for test queries.
+- ✅ Response generation successfully synthesizes retrieved chunks into a coherent, persona-specific recommendation.
+
+---
+
+## Phase 3: System Integration & Application Deployment
+**Goal**: Build a user-facing application, create a full test suite, and deploy the system.
+
+### Tasks
+- [✅] **Create an Interactive Web Application**: In `app.py`, build an interactive UI using Gradio.
+- [✅] **Integrate RAG Pipeline**: Connect the UI components to the full RAG pipeline (embedding, search, generation).
+- [✅] **Add Examples and UI Polish**: Include example scenarios and helper functions to improve user experience.
+- [✅] **Implement Access Key**: Add a simple password field to protect the demo.
+- [✅] **Deploy to Hugging Face Spaces**: Configure the repository for deployment and launch the live application.
+- [✅] **Create a Full Test Suite**: In the `tests/` directory, write unit tests using `pytest` for key logic, including semantic chunking and RAG pipeline functions.
+
+### Success Criteria
+- ✅ End-to-end pipeline is fully functional within the Gradio application.
+- ✅ Application successfully deployed and accessible via a public URL.
+- ✅ Core logic is validated with passing unit tests, ensuring system resilience.
+
+---
+
+## Phase 4: Documentation & Presentation
+**Goal**: Create clear, comprehensive documentation for the project.
+
+### Tasks
+- [✅] **Write a comprehensive `README.md`**:
+    - Include project goals, features, and system architecture.
+    - Provide clear, step-by-step instructions for local setup and execution.
+    - Add a link to the live deployed application.
+- [✅] **Document code**: Add docstrings and inline comments to all major functions and modules.
+- [✅] **Prepare for presentation**: Create a logical flow for a 5-minute video demonstration, walking through the project's "why," the PoC notebook, and the final live application.
+
+### Success Criteria
+- ✅ `README.md` is professional, clear, and comprehensive.
+- ✅ Code is well-documented and easy to understand.
+- ✅ A clear plan for the final presentation is established.

docs/implementation_plan.md

@@ -1,279 +0,0 @@
-# FOT Intervention Recommender
-## Detailed Implementation Plan (Revision 2)
-
----
-
-## Overview
-
-This implementation plan transforms the strategic project plan into executable phases, with specific tasks, deliverables, and success criteria for building the working proof-of-concept.
-
-***Note on Strategic Pivot:*** *We have shifted from programmatic PDF extraction to using a manually curated, high-quality JSON knowledge base (`knowledge_base_raw.json`). This decision was made to bypass the complexities and unreliability of PDF parsing and to focus directly on the core RAG pipeline development.*
-
-**Primary Deliverable**: A working RAG system application that provides intervention recommendations.
-
----
-
-## Phase 0: Environment Setup & Resource Gathering
-**Goal**: Establish a lean development environment and use the pre-processed source materials.
-
-### Tasks
-
-#### 0.1 Development Environment Setup
-- [✅] Create local project structure.
-- [✅] ~~Install required libraries in first cell:~~
-  ```python
-  ~~!pip install sentence-transformers faiss-cpu langchain pandas pymupdf pdfplumber transformers~~
-  ```
-- [✅] **Install simplified libraries:**
-  ```bash
-  uv pip install langchain sentence-transformers faiss-cpu transformers torch
-  ```
-- [✅] Import necessary libraries and test basic functionality.
-
-#### 0.2 Source Material Collection
-- [✅] ~~**Extract FOT Toolkit pages 43-68**~~
-- [✅] ~~**Download 5 external sources**~~
-- [✅] **Prepare `knowledge_base_raw.json`**: Manually (or with LLM assistance) extract and structure all relevant interventions from the FOT Toolkit into a clean JSON file. This file becomes our single source of truth.
-
-#### 0.3 Quick Content Reconnaissance
-- [✅] ~~Scan each document to identify complexity~~
-- [✅] ~~Create a "document complexity map" for processing strategy~~
-
-### Success Criteria
-- ✅ Local development environment running with all simplified dependencies.
-- ✅ `knowledge_base_raw.json` file is created, validated, and located in `data/processed/`.
-- ✅ ~~Basic understanding of each document's structure and complexity~~
-
----
-
-## Phase 1: Knowledge Base Construction
-**Goal**: ~~Extract, process, and structure content~~ **Load and semantically chunk the pre-processed knowledge base.**
-
-### Tasks
-
-#### 1.1 Content Extraction (Hybrid Approach)
-- [✅] ~~**Implement PyMuPDF extraction**~~
-- [✅] ~~**Implement pdfplumber for tables**~~
-- [✅] ~~**Manual extraction for complex pages**~~
-- [✅] **Load Pre-processed Knowledge Base**: Implement logic in `main.py` to load the `knowledge_base_raw.json` file.
-
-#### 1.2 Content Processing & Standardization
-- [ ] ~~**Create intervention extraction function**~~
-- [ ] ~~**Process each document**~~
-- [ ] **Implement Semantic Chunker**: Create a `semantic_chunker.py` module that combines related page-based items from the raw JSON into larger, topic-based chunks (e.g., group all pages about "Intervention Evaluation Flowchart" into one chunk).
-
-#### 1.3 Knowledge Base Structuring
-- [ ] ~~**Create standardized intervention format**~~
-- [ ] ~~**Implement semantic chunking**~~
-- [ ] **Define Final Chunk Structure**: Ensure the output of the semantic chunker is a clean list of dictionaries, each containing `title`, `fot_pages`, and a combined `content` string.
-
-### Success Criteria
-- ✅ `knowledge_base_raw.json` successfully loaded into the application.
-- ✅ Semantic chunking logic correctly combines related pages into fewer, more coherent chunks.
-- ✅ A final `knowledge_base_final_chunks.json` file is produced and validated for quality.
-
----
-
-## Phase 2: RAG Pipeline Implementation
-**Goal**: Build and test the core RAG functionality.
-
-### Tasks
-
-#### 2.1 Vector Embedding Setup
-- [✅] **Initialize embedding model**:
-```python
-from sentence_transformers import SentenceTransformer
-model = SentenceTransformer('all-MiniLM-L6-v2')
-```
-
-- [✅] **Create embeddings for knowledge base**:
-```python
-def create_embeddings(intervention_chunks):
-    embeddings = model.encode(intervention_chunks)
-    return embeddings
-```
-
-- [✅] **Set up FAISS vector database**:
-```python
-import faiss 
-def create_vector_db(embeddings):
-    dimension = embeddings.shape[1]
-    index = faiss.IndexFlatIP(dimension)
-    index.add(embeddings)
-    return index
-```
-
-#### 2.2 Retrieval System
-- [✅] **Implement semantic search**:
-```python
-def search_interventions(query, index, intervention_data, k=3):
-    query_embedding = model.encode([query])
-    scores, indices = index.search(query_embedding, k)
-    return [(intervention_data[i], scores[0][idx]) for idx, i in enumerate(indices[0])]
-```
-
-- [ ] **Test retrieval with sample queries**:
-  - "Student failing core classes and missing school"
-  - "Attendance problems and behavioral issues"
-  - "Low credits earned, needs academic support"
-
-#### 2.3 Response Generation
-- [ ] **Create educator-friendly formatter**:
-```python
-def format_recommendations(retrieved_interventions, student_profile):
-    formatted_response = []
-    for intervention, score in retrieved_interventions:
-        recommendation = {
-            "intervention_name": intervention["name"],
-            "rationale": f"Recommended because: {explain_match(intervention, student_profile)}",
-            "implementation_steps": intervention["implementation_steps"],
-            "source": intervention["source_document"],
-            "confidence_score": score
-        }
-        formatted_response.append(recommendation)
-    return formatted_response
-```
-
-### Success Criteria
-- ✅ Vector database successfully created with all intervention embeddings
-- ✅ Semantic search returns relevant results for test queries
-- ✅ Response format is educator-friendly with clear implementation guidance
-- ✅ Source citations are properly maintained throughout pipeline
-
----
-
-## Phase 3: System Integration & Testing
-**Goal**: End-to-end testing with provided student profile
-
-### Tasks
-
-#### 3.1 End-to-End Pipeline Integration
-- [ ] **Create main recommendation function**:
-```python
-def get_fot_recommendations(student_profile_narrative):
-    # 1. Process student narrative
-    # 2. Perform semantic search  
-    # 3. Retrieve top 3 interventions
-    # 4. Format for educators
-    # 5. Return structured recommendations
-    pass
-```
-
-#### 3.2 Testing with Sample Student Profile
-- [ ] **Test with provided profile**:
-```python
-sample_student = """This student is struggling to keep up with coursework, 
-having failed one core class and earning only 2.5 credits out of 4 credits 
-expected for the semester. Attendance is becoming a concern at 88% for an 
-average annual target of 90%, and they have had one behavioral incident. 
-The student needs targeted academic and attendance support to get back on 
-track for graduation."""
-
-recommendations = get_fot_recommendations(sample_student)
-```
-
-#### 3.3 Quality Validation & Refinement
-- [ ] **Evaluate recommendation quality**:
-  - Do recommendations address student's specific risk factors?
-  - Are implementation steps clear and actionable?
-  - Are source citations accurate and helpful?
-
-- [ ] **Refine retrieval if needed**:
-  - Adjust embedding model parameters
-  - Modify chunking strategy if results are poor
-  - Fine-tune response formatting
-
-### Success Criteria
-- ✅ End-to-end pipeline processes student profile successfully
-- ✅ Returns exactly 3 relevant intervention recommendations
-- ✅ Each recommendation includes implementation steps and source citation
-- ✅ Recommendations directly address student's risk factors (credits, attendance, behavior)
-
----
-
-## Phase 4: Documentation & Presentation Preparation
-**Goal**: Create clear notebook documentation and prepare for video presentation
-
-### Tasks
-
-#### 4.1 Colab Notebook Documentation
-- [ ] **Add comprehensive markdown cells**:
-  - Project overview and goals
-  - Knowledge base composition and rationale
-  - Technical architecture explanation
-  - Step-by-step process documentation
-
-- [ ] **Code documentation**:
-  - Add docstrings to all functions
-  - Include inline comments for complex logic
-  - Add example usage for key functions
-
-#### 4.2 Demonstration Preparation
-- [ ] **Create demonstration workflow**:
-  - Show knowledge base construction process
-  - Demonstrate search functionality with different queries
-  - Walk through the sample student profile analysis
-  - Display formatted recommendations
-
-- [ ] **Prepare talking points for video**:
-  - Project value proposition (30 seconds)
-  - Technical approach overview (60 seconds)
-  - Live demonstration (2 minutes)
-  - Next steps and product vision (90 seconds)
-
-### Success Criteria
-- ✅ Notebook is well-documented with clear explanations
-- ✅ All code cells execute successfully from top to bottom
-- ✅ Demonstration workflow is smooth and highlights key features
-- ✅ Ready for 5-minute video recording
-
----
-
-## Phase 5: Bonus Features (Optional)
-**Goal**: Implement advanced features to differentiate the solution
-
-### Option A: API Microservice (Bonus 1)
-- [ ] **Create FastAPI application**:
-```python
-from fastapi import FastAPI
-app = FastAPI(title="FOT Intervention Recommender")
-
-@app.post("/recommend")
-async def get_recommendations(student_narrative: str):
-    return get_fot_recommendations(student_narrative)
-```
-
-- [ ] **Containerize with Docker**
-- [ ] **Create deployment documentation**
-
-### Option B: Persona-Based Recommendations (Bonus 2)
-- [ ] **Implement persona-specific prompts**:
-```python
-def generate_persona_recommendations(interventions, persona):
-    # Teacher: Classroom-focused, actionable steps
-    # Parent: Supportive language, home-based strategies  
-    # Principal: Resource requirements, systemic approach
-    pass
-```
-
-### Success Criteria (if attempted)
-- ✅ Bonus feature fully functional and demonstrated
-- ✅ Added value is clear and well-articulated
-- ✅ Implementation quality matches core system standards
-
----
-
-## Risk Mitigation Strategies
-
-### Technical Risks
-- **~~Complex PDF extraction fails~~**: **(RESOLVED)** This risk has been completely eliminated by pivoting to a manually curated JSON file.
-- **Poor embedding quality**: Test alternative models (e.g., `all-mpnet-base-v2`).
-- **Retrieval returns irrelevant results**: Adjust chunking strategy or add filtering.
-- **New Risk - Poor Manual Extraction**: The quality of the RAG system now depends entirely on the quality of the `knowledge_base_raw.json`. Mitigation: Manually review and edit the JSON for clarity, accuracy, and completeness.
-
-### Time Management Risks
-- **~~Document processing takes too long~~**: **(RESOLVED)** This risk is eliminated.
-- **Perfectionism trap**: Focus on working MVP first, refinements second.
-- **Scope creep**: Stick to core deliverables, save enhancements for bonus phase.
-
----

docs/initial_plan.md

@@ -1,150 +0,0 @@
-# Freshman On-Track Intervention Recommender
-## Project Plan & Technical Design (Revision 3)
-
----
-
-## Problem Understanding
-
-**Core Problem**: Freshman year performance is the strongest predictor of high school graduation, yet educators lack systematic tools to match at-risk 9th graders with evidence-based interventions. Currently, intervention selection relies on educator intuition rather than proven best practices, leading to inconsistent support for struggling students.
-
-**Goal of this PoC**: Build a Retrieval-Augmented Generation (RAG) system that takes a student's on-track indicators (credits, attendance, behavioral flags) and automatically recommends the most relevant, evidence-based intervention strategies from a curated knowledge base of proven FOT practices.
-
-**Value Proposition**: This system transforms scattered research into actionable guidance, enabling educators to quickly identify targeted interventions without requiring deep expertise in educational research. By democratizing access to best practices, we can systematically improve outcomes for at-risk freshmen.
-
----
-
-## Proposed RAG Architecture
-
-### Technical Stack & Rationale
-
-**Programming Language**: Python
-- Industry standard for ML/AI development
-- Rich ecosystem of libraries for RAG implementation
-- Rapid prototyping capabilities align with "bias for action" principle
-
-**Core Libraries**:
-- **LangChain**: Framework for RAG pipeline orchestration and prompt management
-- **Sentence Transformers**: High-quality semantic embeddings optimized for educational content
-- **FAISS**: Fast, in-memory vector search for PoC (Facebook AI Similarity Search)
-- **Simplified Stack**: Focus on `langchain`, `sentence-transformers`, `faiss-cpu`, `torch`, and `transformers` to directly support the core RAG pipeline, removing dependencies for direct PDF processing.
-
-**Vector Embeddings**: `all-MiniLM-L6-v2` model
-- Optimized for semantic similarity tasks
-- Balanced performance vs. computational efficiency
-- Strong performance on educational/instructional text
-
-**Cloud Services** (Production Path):
-- **Google Cloud Run**: Serverless, auto-scaling container deployment
-- **Pinecone/Weaviate**: Managed vector database for production scale
-- **Google Cloud Storage**: Document storage and versioning
-
-### RAG Pipeline Architecture
-
-1.  **Knowledge Base Ingestion**: Load and process a manually curated, high-quality JSON knowledge base (`knowledge_base_raw.json`). This bypasses unreliable PDF parsing to focus on core RAG functionality.
-2.  **Chunking Strategy**: Semantic chunking by intervention type and implementation steps.
-3.  **Vector Embedding**: Transform text chunks into searchable vector representations.
-4.  **Retrieval**: Take the `narrative_summary_for_embedding` from the student profile as the query. Perform semantic search against the vector database to retrieve the top 3 most relevant intervention chunks.
-5.  **Synthesis**: Generate educator-friendly recommendations with source citations.
-
-### Alignment with Architectural Principles
-
-- **RAG as Core**: Semantic search ensures recommendations are grounded in evidence-based research.
-- **Actionable for Educators**: Output format prioritizes clear, implementable steps over raw research.
-- **Startup Scale**: FAISS for PoC, cloud-native services for production scalability.
-- **Bias for Action**: Minimal viable architecture focused on core functionality first.
-
----
-
-## Knowledge Base & Data Processing Strategy
-
-### Selected Best-Practice Documents
-
-The knowledge base is built from the primary source document provided and is complemented by five additional high-quality, evidence-based resources to provide specific, actionable "playbooks" for educators.
-
-**Primary Source Document:**
-1.  **Freshman On‑Track Toolkit (2nd Edition)** (Network for College Success, 2017)
-    -   ***Primary Focus Area***: **Tool Set C: Developing and Tracking Interventions (Pages 43-68)**, which provides the core framework for intervention planning, tracking, and evaluation.
-
-**Additional Curated Sources:**
-2.  **17 Quick Tips for Your Credit Recovery Program** (Edmentum, 2024)
-    -   *Focus*: Actionable strategies for designing and implementing effective credit recovery programs at both the district and school levels.
-3.  **Handout: Strategies to Address Chronic Absenteeism** (Institute of Education Sciences, REL Southwest, 2025)
-    -   *Focus*: Evidence-based interventions for chronic absenteeism, including Early Warning Systems, Mentoring, and Check & Connect.
-4.  **High-Quality Tutoring: An Evidence-Based Strategy to Tackle Learning Loss** (Institute of Education Sciences, 2021)
-    -   *Focus*: Defines the characteristics of effective, high-impact tutoring to accelerate student learning.
-5.  **WWC Intervention Report: Check & Connect** (Institute of Education Sciences, What Works Clearinghouse, 2015)
-    -   *Evidence Level*: A detailed report on a key dropout prevention program with positive effects on keeping students in school.
-6.  **Early Intervention Strategies: Using Teams to Monitor and Identify Students in Need of Support** (Attendance Works, 2019)
-    -   *Focus*: A multi-tiered team-based approach to monitoring attendance data and implementing early interventions.
-
-### Data Processing Strategy
-
-~~**Content Extraction** (Hybrid Strategy):~~
-- ~~**Tier 1**: PyMuPDF (fitz) for rapid extraction of simple, single-column text pages~~
-- ~~**Tier 2**: pdfplumber for structured tabular data to preserve relational integrity~~
-- ~~**Tier 3**: Nougat (Meta AI) layout-aware model for complex multi-column layouts and flowcharts~~
-- ~~**Quality Assurance**: Manual review and validation of extracted content accuracy~~
-
-**Pivoted Content Extraction Strategy:**
-- **Manual Curation**: Bypassed programmatic PDF extraction due to complexity and unreliability. Instead, key interventions were manually extracted (with LLM assistance) from all source documents into a single, high-quality `knowledge_base_raw.json` file. This ensures maximum quality and allows direct focus on the RAG pipeline.
-
-**Chunking Approach**:
-- **Semantic Chunking**: Break documents by intervention type, not arbitrary word limits.
-- **Chunk Size**: 300-500 words to maintain context while enabling precise retrieval.
-- **Overlap Strategy**: 50-word overlap to preserve cross-boundary context.
-- **Metadata Tagging**: Source document, intervention category, target indicators.
-
-**Content Preparation**:
-- Standardize intervention descriptions with consistent format.
-- Extract key implementation steps and required resources.
-- Tag interventions by target risk factors (attendance, credits, behavior).
-- Create intervention summaries optimized for educator consumption.
-
----
-
-## AI as a Co-pilot Strategy
-
-### Development Acceleration
-
-**GitHub Copilot**:
-- Code generation for standard RAG pipeline components.
-- Boilerplate reduction for data processing and API endpoints.
-- Test case generation for validation scenarios.
-
-**Large Language Models (GPT-4/Claude)**:
-- **Knowledge Base Curation**: Accelerated the manual extraction process by summarizing dense academic PDFs and structuring the content into the clean `knowledge_base_raw.json` format.
-- **Prompt Engineering**: Optimize prompts for educator-specific output formatting.
-- **Content Synthesis**: Transform academic language into practitioner-friendly recommendations.
-- **Code Review**: Architecture validation and optimization suggestions.
-
-### Problem-Solving Workflow
-
-1.  **Research Phase**: Use LLMs to quickly synthesize intervention research and identify gaps.
-2.  **Architecture Design**: Validate technical approach against startup scaling requirements.
-3.  **Implementation**: Leverage Copilot for rapid prototype development.
-4.  **Testing**: AI-assisted generation of diverse student profile test cases.
-5.  **Optimization**: LLM-powered analysis of retrieval quality and recommendation relevance.
-
-### Quality Assurance
-
-- **Prompt Validation**: Use AI to generate edge cases for robust testing.
-- **Content Review**: AI-assisted verification that academic content translates to actionable guidance.
-- **Bias Detection**: Systematic review of recommendations for potential equity issues.
-
----
-
-## Success Metrics & Next Steps
-
-**PoC Success Criteria**:
-- Accurate retrieval of top 3 relevant interventions for sample student profile.
-- Educator-friendly output format with clear implementation guidance.
-- Sub-2 second response time for typical queries.
-- Proper source citation for all recommendations.
-
-**Production Evolution Path**:
-1.  **Enhanced Knowledge Base**: Scale to 50+ intervention documents.
-2.  **Persona-Based Outputs**: Tailored recommendations for teachers, parents, principals.
-3.  **API Microservice**: RESTful service for integration with SIS platforms.
-4.  **Analytics Dashboard**: Track intervention effectiveness and usage patterns.
-
-This PoC establishes the foundation for a scalable, evidence-based intervention recommendation system that can transform how educators support at-risk freshmen nationwide.

docs/project_plan_and_design.md

@@ -0,0 +1,111 @@
+# Freshman On-Track Intervention Recommender
+## Project Plan & Technical Design (Final)
+
+---
+
+## 1. Problem Understanding
+
+**Core Problem**: Freshman year performance is the strongest predictor of high school graduation, yet educators lack systematic tools to match at-risk 9th graders with evidence-based interventions. Currently, intervention selection often relies on educator intuition rather than proven best practices, leading to inconsistent support for struggling students.
+
+**Goal of this PoC**: Build a Retrieval-Augmented Generation (RAG) system that takes a simple narrative about a student's challenges and automatically recommends the most relevant, evidence-based intervention strategies from a curated knowledge base of proven FOT practices.
+
+**Value Proposition**: This system transforms scattered educational research into actionable guidance, enabling educators to quickly identify targeted interventions. By democratizing access to best practices, we can systematically improve outcomes for at-risk freshmen.
+
+---
+
+## 2. Proposed RAG Architecture
+
+### Technical Stack & Rationale
+
+The final technical stack was chosen to prioritize development speed, robustness, and alignment with modern AI engineering practices.
+
+**Programming Language**: Python 3.12
+- *Rationale*: Industry standard for AI/ML, rich library ecosystem, and rapid prototyping capabilities.
+
+**Core Libraries**:
+- **Sentence Transformers & FAISS**: For high-quality semantic search. `all-MiniLM-L6-v2` offers an excellent balance of performance and efficiency. FAISS provides a fast, in-memory vector store ideal for a PoC.
+- **Google Generative AI**: To leverage the powerful `gemini-1.5-flash` model for the "Generation" step, synthesizing evidence into actionable, persona-based advice.
+- **Gradio**: To rapidly build and deploy a user-friendly, interactive web application for the demo.
+- **uv**: A modern, high-speed project and environment manager used to ensure fast, reliable dependency installation and management.
+
+> **_Stack Evolution_**
+> My initial plan considered using `LangChain` for pipeline orchestration. However, for a focused PoC, implementing the RAG logic directly provided greater control, transparency into the prompts, and avoided an additional dependency. The successful pivot away from programmatic PDF extraction also eliminated the need for data manipulation libraries like `Pandas`, resulting in a leaner and more focused final stack.
+
+### Deployment & Production Path
+
+- **PoC Deployment**: The application was successfully deployed to **Hugging Face Spaces**.
+    - *Rationale*: This platform is ideal for hosting interactive Gradio applications, providing a public URL for live demonstrations and stakeholder feedback without complex infrastructure setup.
+
+- **Production Path**:
+    - The core logic would be packaged into a formal **REST API microservice** using a framework like FastAPI and containerized with Docker.
+    - This API would be deployed on a scalable, serverless platform like **Google Cloud Run** or **AWS Lambda** for cost-effective, high-availability serving.
+    - The FAISS index would be replaced by a managed vector database like **Pinecone** or **Weaviate** to handle a larger knowledge base and higher query volumes.
+
+### RAG Pipeline Architecture
+
+1.  **Knowledge Base Curation**: A high-quality JSON file (`knowledge_base_raw.json`) was manually curated from source documents to ensure maximum data quality.
+2.  **Chunking & Indexing (Build Time)**: A build script processes the raw JSON, performs semantic chunking by concept, and creates a FAISS vector index (`faiss_index.bin`).
+3.  **Retrieval (Runtime)**: A user's narrative is embedded and used to perform a semantic search against the FAISS index, retrieving the most relevant intervention chunks.
+4.  **Synthesis (Runtime)**: The retrieved chunks and the original query are formatted into a persona-specific prompt and sent to the Gemini API.
+5.  **Output**: The API generates a synthesized, actionable recommendation tailored to a teacher, parent, or principal.
+
+---
+
+## 3. Knowledge Base & Data Processing Strategy
+
+### Selected Best-Practice Documents
+The knowledge base was built from the primary source and five additional high-quality, evidence-based resources:
+
+1.  **Freshman On‑Track Toolkit (2nd Edition)** (Network for College Success, 2017)
+2.  **17 Quick Tips for Your Credit Recovery Program** (Edmentum, 2024)
+3.  **Handout: Strategies to Address Chronic Absenteeism** (IES, REL Southwest, 2025)
+4.  **High-Quality Tutoring: An Evidence-Based Strategy...** (IES, 2021)
+5.  **WWC Intervention Report: Check & Connect** (IES, What Works Clearinghouse, 2015)
+6.  **Early Intervention Strategies...** (Attendance Works, 2019)
+
+### Data Processing Strategy
+
+> **_Strategic Pivot Summary_**
+> My initial plan involved complex programmatic PDF extraction. I pivoted to a manually curated `knowledge_base_raw.json` file, a decision driven by the "Bias for Action" principle. This approach guaranteed high-quality data, de-risked the project, and allowed me to focus on building a more effective core RAG pipeline.
+
+**Final Processing Approach**:
+- **Manual Curation**: Key interventions were manually extracted from all source documents into a single, high-quality `knowledge_base_raw.json`.
+- **Semantic Chunking**: A script groups the raw data by `concept` (e.g., "Intervention: Mentoring") to create meaningful, coherent chunks for embedding. This is more effective than chunking by arbitrary word counts.
+- **Content Preparation**: The title is prepended to the content for each chunk (`"Title: {concept}. Content: {content}"`) to improve the contextual richness of the embeddings.
+
+---
+
+## 4. AI as a Co-pilot: A Human-Directed Workflow
+
+My approach to AI collaboration treats large language models as strategic partners, with the human acting as the director and critical thinker. This involves a structured, iterative dialogue rather than simple prompting.
+
+### Strategic Planning & Ideation
+The project and implementation plans were developed through an iterative, multi-model process.
+- **Initial Drafting:** I engaged both Gemini Pro and Claude Sonnet to generate independent approaches for structuring the project and tackling the core tasks.
+- **Iterative Refinement:** I then orchestrated a dialogue between the models, using prompts like *"What do you think of this take?"* to have each model critique the other's feedback. This iterative loop allowed me to synthesize their strengths and converge on a robust, detailed plan.
+
+### Core Technique: Active Context Management
+Throughout the collaboration, I carefully managed the conversational context to ensure high-quality, relevant outputs.
+- **Preventing Confusion:** I actively curated the chat history to avoid "muddying the results." If multiple versions of code or text were present, I would remove obsolete versions from the context window to prevent the model from referring to the wrong information.
+- **Tool-Specific Workflows:** I leveraged the unique features of different browser-based interfaces. **Claude.ai's** "Artifacts" feature was invaluable for creating and editing planning documents. **Google's AI Studio** offered a significant advantage with its ability to fork conversations and delete individual messages from the context window, enabling precise context control.
+
+### Foundational Best Practices
+This AI-driven workflow is built on a foundation of solid engineering hygiene. The project was initiated from a pre-configured template with standard tooling (`ruff`, `uv`, `.gitignore`), ensuring a clean and maintainable codebase from the start.
+
+This creates a **virtuous cycle of code quality**. By starting with and consistently adding well-structured code, that uses best practices, the context window provided to the AI is enriched with high-quality exemplars. The model's in-context learning capabilities mean it, in turn, generates new code that adheres to these established patterns, further elevating the quality baseline. This compounding effect is a deliberate strategy to maintain a high standard of maintainability and robustness throughout the development lifecycle.
+
+---
+
+## 5. Success Metrics & Production Path
+
+**PoC Success Criteria (Achieved)**:
+- ✅ Accurate retrieval of the most relevant interventions for sample student narratives.
+- ✅ Persona-based output is clear, actionable, and tailored to the audience.
+- ✅ Sub-second response time for the entire RAG pipeline.
+- ✅ All recommendations are grounded in evidence, with source documents and relevance scores displayed.
+- ✅ Project is fully documented, tested, and deployed as a live, interactive web application.
+
+**Production Evolution Path**:
+1.  **Enhanced Knowledge Base**: Scale the knowledge base to include a wider range of interventions.
+2.  **Formal REST API**: Package the final logic into a production-ready REST API (e.g., using FastAPI and Docker) for robust integration with School Information Systems (SIS).
+3.  **Feedback Loop & Analytics**: Add a mechanism for educators to rate the usefulness of recommendations and build an analytics dashboard to track intervention effectiveness.