| | --- |
| | title: LocalAGI AI Sommelier |
| | emoji: 🍸 |
| | colorFrom: indigo |
| | colorTo: purple |
| | sdk: gradio |
| | app_file: app.py |
| | pinned: false |
| | --- |
| | |
| | # 🍸 LocalAGI: The AI Sommelier |
| |
|
| | ## 📖 Overview |
| | LocalAGI is a multimodal Retrieval-Augmented Generation (RAG) application that acts as an intelligent, interactive bartender. By combining state-of-the-art computer vision with vector search, the application allows users to upload a photo of any liquor bottle and instantly receive curated cocktail recipes utilizing that specific spirit from a custom-ingested library. |
| |
|
| | Engineered to run entirely on CPU-bound cloud environments (like Hugging Face Spaces), this project showcases advanced optimization techniques, including dynamic image cropping, intelligent text-splitting, and dual-pass vision logic. |
| |
|
| | ## ✨ Key Features |
| | * **Visual Brand Recognition:** Utilizes a Vision-Language Model (VLM) to read labels and identify specific alcohol brands from user-uploaded photos, going beyond generic object categorization. |
| | * **Custom Knowledge Base (RAG):** Ingests raw `.txt` and `.pdf` recipe books, intelligently splitting them into discrete recipe chunks using RegEx and LangChain, and stores them in a local Chroma vector database. |
| | * **Smart Cropping Pipeline:** Implements YOLOv8 to locate bottles or glasses in an image, applying dynamic 25% padding to isolate the label and strip away background noise. |
| | * **Hardware-Optimized Processing:** Features custom logic to downscale images and restrict token generation limits, allowing complex 2-billion-parameter models to run efficiently on free-tier cloud CPUs. |
| | * **Interactive UI:** A Gradio interface featuring a conversational chat format, session state memory, and a hidden "Vision Debug" gallery for real-time insight into the AI's detection process. |
| |
|
| | ## 🛠️ Technical Stack |
| | * **Frontend/UI:** Gradio |
| | * **Computer Vision:** Ultralytics YOLOv8 (Object Detection) |
| | * **Vision-Language Model:** HuggingFaceTB/SmolVLM-Instruct (Label OCR & Context) |
| | * **Vector Database:** ChromaDB |
| | * **Embeddings:** `sentence-transformers/all-MiniLM-L6-v2` |
| | * **Orchestration:** LangChain (Document Loaders, Text Splitters) |
| |
|
| | ## 🧠 How It Works Under the Hood |
| | 1. **Document Ingestion:** The user uploads a recipe book. The system uses a strict "Hard Cut" method to split the document exactly at the start of every new recipe, ensuring clean data retrieval. |
| | 2. **Object Detection:** When a photo is uploaded, YOLOv8 scans the image for bottles (Class 39) or glasses (Class 40/41), creating a focused, padded crop of the object. |
| | 3. **Vision Processing:** The cropped image is aggressively downscaled (384x384) and passed to SmolVLM. The model is restricted to a 15-token output to rapidly extract just the brand name (e.g., "Absolut Vodka"). |
| | 4. **Fallback Logic:** If the VLM returns a generic term (e.g., just "Vodka") due to a bad crop, the system automatically triggers a secondary pass using the full, uncropped image to guarantee brand identification. |
| | 5. **Context Retrieval (RAG):** The extracted brand name is embedded and queried against the Chroma database, retrieving the top 4 most relevant, full-text recipes. |
| | 6. **Chat Output:** The system formats the retrieved recipes and returns them to the user via the conversational UI. |
| |
|
| | ## 🚀 Future Roadmap |
| | * Integration with hardware-accelerated APIs (Groq/Gemini) for sub-3-second vision processing. |
| | * User inventory tracking to suggest recipes based on a combination of multiple owned bottles. |
