README.md

metadata

title: LocalAGI AI Sommelier
emoji: 🍸
colorFrom: indigo
colorTo: purple
sdk: gradio
app_file: app.py
pinned: false

🍸 LocalAGI: The AI Sommelier

📖 Overview

LocalAGI is a multimodal Retrieval-Augmented Generation (RAG) application that acts as an intelligent, interactive bartender. By combining state-of-the-art computer vision with vector search, the application allows users to upload a photo of any liquor bottle and instantly receive curated cocktail recipes utilizing that specific spirit from a custom-ingested library.

Engineered to run entirely on CPU-bound cloud environments (like Hugging Face Spaces), this project showcases advanced optimization techniques, including dynamic image cropping, intelligent text-splitting, and dual-pass vision logic.

✨ Key Features

• Visual Brand Recognition: Utilizes a Vision-Language Model (VLM) to read labels and identify specific alcohol brands from user-uploaded photos, going beyond generic object categorization.
• Custom Knowledge Base (RAG): Ingests raw .txt and .pdf recipe books, intelligently splitting them into discrete recipe chunks using RegEx and LangChain, and stores them in a local Chroma vector database.
• Smart Cropping Pipeline: Implements YOLOv8 to locate bottles or glasses in an image, applying dynamic 25% padding to isolate the label and strip away background noise.
• Hardware-Optimized Processing: Features custom logic to downscale images and restrict token generation limits, allowing complex 2-billion-parameter models to run efficiently on free-tier cloud CPUs.
• Interactive UI: A Gradio interface featuring a conversational chat format, session state memory, and a hidden "Vision Debug" gallery for real-time insight into the AI's detection process.

🛠️ Technical Stack

• Frontend/UI: Gradio
• Computer Vision: Ultralytics YOLOv8 (Object Detection)
• Vision-Language Model: HuggingFaceTB/SmolVLM-Instruct (Label OCR & Context)
• Vector Database: ChromaDB
• Embeddings: sentence-transformers/all-MiniLM-L6-v2
• Orchestration: LangChain (Document Loaders, Text Splitters)

🧠 How It Works Under the Hood

1. Document Ingestion: The user uploads a recipe book. The system uses a strict "Hard Cut" method to split the document exactly at the start of every new recipe, ensuring clean data retrieval.
2. Object Detection: When a photo is uploaded, YOLOv8 scans the image for bottles (Class 39) or glasses (Class 40/41), creating a focused, padded crop of the object.
3. Vision Processing: The cropped image is aggressively downscaled (384x384) and passed to SmolVLM. The model is restricted to a 15-token output to rapidly extract just the brand name (e.g., "Absolut Vodka").
4. Fallback Logic: If the VLM returns a generic term (e.g., just "Vodka") due to a bad crop, the system automatically triggers a secondary pass using the full, uncropped image to guarantee brand identification.
5. Context Retrieval (RAG): The extracted brand name is embedded and queried against the Chroma database, retrieving the top 4 most relevant, full-text recipes.
6. Chat Output: The system formats the retrieved recipes and returns them to the user via the conversational UI.

🚀 Future Roadmap

• Integration with hardware-accelerated APIs (Groq/Gemini) for sub-3-second vision processing.
• User inventory tracking to suggest recipes based on a combination of multiple owned bottles.