Update README.md

README.md

@@ -1,12 +1,293 @@
 ---
-title: Demo10
-emoji: 👁
-colorFrom: purple
+title: American University Academic Advisor
+emoji: 🎓
+colorFrom: blue
 colorTo: indigo
 sdk: docker
-sdk_version: 5.26.0
+sdk_version: 4.26.0
 app_file: app.py
 pinned: false
+license: cc-by-nc-4.0
+short_description: Research Chatbot for AU advising questions
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# American University Academic Advisor Chatbot
+
+**This is an ongoing student research project under development for academic purposes. The data focuses on mathematics, statistics, and data science programs and is incomplete. Users are cautioned that LLM may given incomplete or incorrect answers and they should check any responses independently using authoritative sources and their advisors before making any decisions.**
+
+A RAG (Retrieval-Augmented Generation) chatbot using Mistral 7B and ChromaDB to answer questions about American University academic programs and courses. This RAG chatbot uses information from a variety of American University public websites to provide context for user queries when they are passed to the generative AI chatbot model. The information is "retrieved" from a database of scrapped information and used to "augment" the query to the generative AI model.
+
+The database is populated with information from the following sources at American University for the 2024-2025 Academic year.
+
+- Academic program pages for undergraduate majors and masters degrees in the Mathematics and Statistics department.
+- Academic program pages for undergraduate minors in the Mathematics and Statistics department.
+- The Course Catalog for courses with identifiers: DATA, STAT, MATH, CSC, and ITEC.
+- Undergraduate and and Graduate Academic Regulations
+- Study Abroad introductory pages and program pages for majors in Data Science, Mathematics, Statistics, and Computer Science
+
+## Overview
+
+This chatbot application uses:
+
+- **Shiny**: For the web interface
+- **ChromaDB**: As vector database for storing and retrieving academic information
+- **Sentence-Transformers**: For text embeddings
+- **Mistral 7B**: For generating human-like responses
+- **Playwright**: For web scraping academic program and course information
+
+The application is structured to allow for scraping different types of content (academic programs, courses, minors, academic rules, study abroad options) and storing them in a vector database for efficient retrieval.
+
+## Project Structure
+
+```
+au_advisor/
+├── scrapers/                  # Web scrapers for different data sources
+│   ├── course_scraper.py      # Scraper for course catalog
+│   ├── program_scraper.py     # Scraper for academic programs
+│   ├── minor_scraper.py       # Scraper for minor programs
+│   ├── study_abroad_scraper.py # Scraper for study abroad options
+│   ├── academic_rules_scraper.py # Scraper for academic regulations
+│   └── extract_grad_regulations.py # Helper for graduate regulations
+├── utils/                     # Utility modules
+│   ├── chroma_utils.py        # ChromaDB operations
+│   ├── config_utils.py        # Configuration management
+│   ├── scraper_utils.py       # Common scraper utilities
+│   ├── metadata_utils.py      # Metadata enhancement utilities
+│   ├── db_manager.py          # Database management utilities
+│   ├── logging_utils.py       # Centralized logging
+│   ├── chroma_explorer.py     # Utility for exploring ChromaDB data
+│   ├── directory_scan.py      # Scan directory structures
+│   └── check_dependencies.py  # Dependency checker
+├── utils_deploy/              # Deployment utilities
+│   └── check_dependencies.py  # Dependency checker for deployment
+├── config/                    # Configuration files
+│   ├── scrapers_config.json   # Scraper configurations
+│   ├── models.txt             # Embedding model mappings
+│   ├── keys.txt               # Authentication keys (template)
+│   ├── repo_config.json       # Repository configuration
+│   ├── program_urls.txt       # URLs for program scraping
+│   ├── course_urls.txt        # URLs for course scraping
+│   ├── minors.txt             # URLs for minor scraping
+│   ├── regulations.txt        # URLs for academic rules
+│   └── study_abroad_urls.txt  # URLs for study abroad scraping
+├── app.py                     # Main Gradio application 
+├── app_shiny.py               # Shiny web application (alternative)
+├── chatbot.py                 # Core chatbot functionality
+├── collect_data.py            # Data collection script
+├── setup.py                   # Package setup configuration
+├── requirements.txt           # Dependencies
+├── runtime.txt                # Python version for deployment
+└── init.py                    # Environment setup script
+```
+
+## Setup Instructions
+
+### 1. Initialize the Environment
+
+The easiest way to set up is using the included initialization script, which supports both Conda and virtual environments:
+
+```bash
+# Run the initialization script
+python init.py
+```
+
+The script will:
+- Create the chosen environment type
+- Install dependencies
+- Install Playwright browser
+- Create necessary configuration files
+
+You can also specify your preferences via command line:
+```bash
+# For Conda environment
+python init.py --env-type conda
+
+# For virtual environment
+python init.py --env-type venv
+
+# With custom environment name
+python init.py --env-name my-chatbot-env
+
+# With specific Python version
+python init.py --python-version 3.12.7
+```
+
+### 2. Activate the Environment
+
+After initialization, activate your environment:
+
+For Conda:
+```bash
+conda activate chatbot_env  # or your custom name
+```
+
+For virtual environment:
+```bash
+# On Windows
+chatbot_env\Scripts\activate
+
+# On macOS/Linux
+source chatbot_env/bin/activate
+```
+
+### 3. Configuration
+
+1. Create a `.env` file with your Hugging Face API key:
+   ```
+   HF_API_KEY=your_api_key_here
+   ```
+
+2. (Optional) Configure the embedding model in `config.json`:
+   ```json
+   {
+     "embedding_model": {
+       "size": "e5"
+     }
+   }
+   ```
+
+### 4. Data Collection
+
+To gather academic information for the chatbot:
+
+```bash
+# Run all scrapers
+python collect_data.py
+
+# Run a specific scraper
+python collect_data.py --scraper courses
+python collect_data.py --scraper programs
+python collect_data.py --scraper minors
+
+# Enable debug mode to save detailed outputs
+python collect_data.py --debug --save-json
+
+# Use a specific embedding model
+python collect_data.py --model e5
+```
+
+The scrapers are configured in `config/scrapers_config.json` and use URL lists from the corresponding text files in the `config` directory.
+
+### 5. Running the Application
+
+Run the Gradio app:
+```bash
+python app.py
+```
+
+Or run the Shiny app (if installed):
+```bash
+shiny run app_shiny.py
+```
+
+## Embedding Models
+
+The system supports multiple embedding models for different use cases:
+
+| Model | Description | Dimensions | Best For |
+|-------|-------------|------------|----------|
+| small | sentence-transformers/all-MiniLM-L6-v2 | 384 | Fast performance, limited resources |
+| medium | sentence-transformers/all-mpnet-base-v2 | 768 | Good balance of quality and performance |
+| large | sentence-transformers/all-roberta-large-v1 | 1024 | Better quality, more resources |
+| multilingual | paraphrase-multilingual-MiniLM-L12-v2 | 384 | Content in multiple languages |
+| e5 | intfloat/e5-large-v2 | 1024 | Highest quality retrieval, requires more resources |
+
+You can select your preferred model in `config.json` or with command-line arguments.
+
+## ChromaDB Explorer Utility
+
+The project includes a utility for exploring and managing the data in ChromaDB:
+
+```bash
+# Start interactive mode
+python utils/chroma_explorer.py --interactive
+
+# Get statistics about your collection
+python utils/chroma_explorer.py stats
+
+# Export documents to JSON
+python utils/chroma_explorer.py export --output data/export.json
+
+# Create a human-readable text dump
+python utils/chroma_explorer.py dump --output debug/chroma_dump.txt
+
+# Search for specific content
+python utils/chroma_explorer.py search "data science program requirements" --results 10
+
+# Delete documents (use with caution!)
+python utils/chroma_explorer.py delete --type program
+```
+
+## Dependency Management
+
+The project includes a dependency checker to ensure all required packages are properly installed:
+
+```bash
+# Check dependencies in the current directory
+python utils/check_dependencies.py
+
+# Check dependencies in a specific directory
+python utils/check_dependencies.py /path/to/project
+```
+
+The checker will:
+1. Find all imports in Python files
+2. Compare them to packages in requirements.txt
+3. Identify missing or extra dependencies
+4. Generate a proposed requirements.txt with version pinning
+
+## Debug Mode
+
+All scrapers support a debug mode that provides detailed information during execution:
+
+```bash
+# Enable debug mode for program scraper
+python scrapers/program_scraper.py --url "https://www.american.edu/cas/mathstat/data-undergrad/" --debug
+
+# Enable debug mode for course scraper
+python scrapers/course_scraper.py --debug
+```
+
+In debug mode:
+- Detailed logs are saved to the `logs` directory
+- Screenshots of visited pages are taken
+- Raw HTML content is preserved
+- Extracted data is saved as JSON
+- More verbose console output is provided
+
+## Deployment
+
+### Deploying to Hugging Face Spaces
+
+1. Make sure you have a `runtime.txt` file specifying the Python version:
+   ```
+   python-3.12.7
+   ```
+
+2. Fork this repository to your GitHub account
+
+3. Create a new Hugging Face Space using the Gradio SDK
+
+4. Connect your GitHub repository to the Space
+
+5. Configure any necessary environment variables (HF_API_KEY)
+
+## Troubleshooting
+
+- **API Key Issues**: Ensure your Hugging Face API key is valid and has access to the Mistral 7B model
+- **ChromaDB Errors**: Make sure the `chroma_db` directory is writable
+- **Scraping Failures**: Check the scraping logs in `logs/` or detailed debug output in `debug/` 
+- **Dependency Issues**: Run the dependency checker to identify missing packages
+- **Model Compatibility**: If you encounter memory issues, try a smaller embedding model
+
+## License
+
+[![License: CC BY-NC 4.0](https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc/4.0/)
+
+This project is licensed under the Creative Commons Attribution-NonCommercial 4.0 International License.
+
+## Acknowledgments
+
+- American University for the course and program information.
+- Hugging Face for providing access to the Mistral 7B model.
+- The open source community for the various libraries used in this project.