Upload 16 files

README.md

@@ -1,13 +1,145 @@
----
-title: Twin
-emoji: 🚀
-colorFrom: blue
-colorTo: green
-sdk: gradio
-sdk_version: 6.18.0
-python_version: '3.13'
-app_file: app.py
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+---
+title: Kush Digital Twin
+emoji: 🎙️
+colorFrom: indigo
+colorTo: blue
+sdk: gradio
+sdk_version: 6.18.0
+app_file: app.py
+pinned: false
+---
+
+# Digital Twin MultiModal (Voice and Text)
+
+A multimodal digital twin chatbot with **text** and **voice** input, powered by OpenAI (LLM + RAG + tool calling), Deepgram (speech-to-text and text-to-speech), ChromaDB (vector retrieval), and Gradio (web UI).
+
+Converted from the `digital-twin.ipynb` notebook in the AI Engineering course.
+
+## Architecture
+
+![Digital Twin Voice flow chart](docs/Flow-chart.png)
+
+**High-level flow:**
+
+1. **Text path** — user types → embed query → ChromaDB retrieval → OpenAI chat (with tools) → reply in chat
+2. **Voice path** — user records → Deepgram STT → same RAG + chat pipeline → Deepgram TTS → autoplay reply audio
+
+**RAG pipeline:**
+
+1. Knowledge lives in `knowledge/*.md` (identity, career, technical stack)
+2. `chunking.py` splits documents with overlap at sentence/paragraph boundaries
+3. OpenAI `text-embedding-3-small` embeds each chunk
+4. Vectors are stored in a local ChromaDB collection (`chroma_db_twin/`)
+5. Each user message retrieves the top-N similar chunks and injects them as **Context** in the system prompt
+
+**Tools available to the LLM:**
+
+- `send_notification` — Pushover alert to your phone (optional)
+- `roll_dice` — simulated dice roll
+
+**Dynamic context:** keywords in the user's message (`2011`, `dishes`, `sports`, `vacation`) inject extra persona context from `knowledge.md`.
+
+## Prerequisites
+
+- Python 3.10+
+- API keys:
+  - [OpenAI](https://platform.openai.com/api-keys) (required)
+  - [Deepgram](https://console.deepgram.com/) (required)
+  - [Pushover](https://pushover.net/) (optional, for notification tool)
+
+## Quick start
+
+```bash
+# Clone or cd into this directory
+cd digital-twin-voice
+
+# Create a virtual environment (recommended)
+python -m venv .venv
+source .venv/bin/activate   # Windows: .venv\Scripts\activate
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Configure environment
+cp .env.example .env
+# Edit .env and add your API keys
+
+# Build the vector index (also runs automatically on first app launch if empty)
+python build_rag_index.py
+
+# Run the app
+python app.py
+```
+
+Gradio opens at `http://127.0.0.1:7860` (port may vary). Use the text box or microphone to chat.
+
+## Deploy to Hugging Face Spaces
+
+This repo is ready to deploy as a [Gradio Space](https://huggingface.co/docs/hub/spaces-sdks-gradio). The YAML header at the top of this README configures the Space (`sdk: gradio`, `sdk_version: 6.18.0`, `app_file: app.py`).
+
+1. Create a new Space on Hugging Face and choose **Gradio** as the SDK.
+2. Push this repository (or connect your GitHub repo).
+3. In **Settings → Variables and secrets**, add:
+   - `OPENAI_API_KEY` (required — chat + RAG embeddings)
+   - `DEEPGRAM_API_KEY` (required for voice input/output)
+   - Optional: `PUSHOVER_USER`, `PUSHOVER_TOKEN`, model overrides from `.env.example`
+4. On first load, the app builds the ChromaDB index from `knowledge/*.md` (uses OpenAI embeddings). Cold starts on free Spaces may take ~30s.
+
+For faster restarts, enable [Persistent Storage](https://huggingface.co/docs/hub/spaces-storage) and set `CHROMA_PATH=/data/chroma_db_twin`, then run `python build_rag_index.py` once in the Space terminal.
+
+## Project layout
+
+```
+digital-twin-voice/
+├── app.py              # Entry point
+├── build_rag_index.py  # Rebuild ChromaDB from knowledge files
+├── chunking.py         # Text chunking with overlap
+├── rag.py              # Embeddings, ChromaDB, retrieval
+├── config.py           # Environment variables and API clients
+├── knowledge/          # Source documents for RAG
+│   ├── identity.md
+│   ├── career.md
+│   └── technical.md
+├── knowledge.md        # Keyword topic triggers only
+├── prompts.py          # System prompt + topic loading
+├── tools.py            # Pushover + dice tools, tool-call handler
+├── chat.py             # OpenAI chat loop with RAG + tool calling
+├── voice.py            # Deepgram STT / TTS
+├── ui.py               # Gradio interface
+├── requirements.txt
+├── .env.example
+└── docs/
+    ├── Flow-chart.png
+    └── digital-twin-voice-flow.excalidraw
+```
+
+## Customization
+
+- **Persona facts:** edit files in `knowledge/`, then run `python build_rag_index.py`
+- **Topic keywords:** edit `knowledge.md` under `## Topics`
+- **Chunking:** adjust `RAG_CHUNK_SIZE` and `RAG_CHUNK_OVERLAP` in `.env`
+- **Retrieval depth:** set `RAG_N_RESULTS` in `.env` (default: 3)
+- **Tools:** add functions in `tools.py` and register them in `TOOLS`
+- **Models:** set `OPENAI_MODEL`, `EMBEDDING_MODEL`, `DEEPGRAM_STT_MODEL`, and `DEEPGRAM_TTS_MODEL` in `.env`
+
+## Environment variables
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `OPENAI_API_KEY` | Yes | OpenAI API key |
+| `DEEPGRAM_API_KEY` | Yes | Deepgram API key |
+| `OPENAI_MODEL` | No | Chat model (default: `gpt-4.1-mini`) |
+| `EMBEDDING_MODEL` | No | Embedding model (default: `text-embedding-3-small`) |
+| `RAG_N_RESULTS` | No | Chunks retrieved per query (default: `3`) |
+| `RAG_CHUNK_SIZE` | No | Chunk size in characters (default: `500`) |
+| `RAG_CHUNK_OVERLAP` | No | Overlap between chunks (default: `50`) |
+| `RAG_DEBUG` | No | Print retrieved chunk sources to console |
+| `CHROMA_PATH` | No | ChromaDB directory (default: `chroma_db_twin`) |
+| `DEEPGRAM_STT_MODEL` | No | Speech-to-text model (default: `nova-3`) |
+| `DEEPGRAM_TTS_MODEL` | No | Text-to-speech model (default: `aura-2-thalia-en`) |
+| `PUSHOVER_USER` | No | Pushover user key (for notification tool) |
+| `PUSHOVER_TOKEN` | No | Pushover app token (for notification tool) |
+
+## License
+
+Educational project from the AI Engineering course.

app.py

@@ -0,0 +1,23 @@
+"""Kush Digital Twin — multimodal voice + text chat with RAG."""
+
+from config import ensure_clients, ensure_openai_client
+from rag import ensure_index
+from ui import CSS, create_demo
+
+
+def startup() -> None:
+    """Warm up the RAG index (OpenAI only — voice keys load on first mic use)."""
+    ensure_openai_client()
+    ensure_index()
+
+
+demo = create_demo(on_load=startup)
+
+# Hugging Face Spaces looks for `demo`, `app`, or `interface`.
+app = demo
+
+
+if __name__ == "__main__":
+    ensure_clients()
+    startup()
+    demo.launch(inbrowser=True, css=CSS)

build_rag_index.py

@@ -0,0 +1,15 @@
+"""Build or rebuild the ChromaDB vector index from knowledge/*.md files."""
+
+from config import ensure_openai_client
+from rag import build_index, get_collection
+
+
+def main() -> None:
+    ensure_openai_client()
+    before = get_collection().count()
+    count = build_index(force_rebuild=True)
+    print(f"RAG index rebuilt: {before} -> {count} chunks in {get_collection().name}")
+
+
+if __name__ == "__main__":
+    main()

chat.py

@@ -0,0 +1,64 @@
+import config
+from prompts import SYSTEM_MESSAGE, TOPIC_CONTEXT
+from rag import retrieve
+from tools import TOOLS, handle_tool_call
+
+
+def content_to_text(content) -> str:
+    """Gradio messages may store content as a string or a list of parts."""
+    if isinstance(content, str):
+        return content
+    if isinstance(content, list):
+        parts = []
+        for part in content:
+            if isinstance(part, dict):
+                parts.append(part.get("text") or part.get("content") or "")
+            else:
+                parts.append(str(part))
+        return " ".join(parts).strip()
+    return str(content)
+
+
+def build_system_prompt(latest_user_message: str) -> str:
+    system = SYSTEM_MESSAGE
+
+    context, metadatas = retrieve(latest_user_message)
+    if context:
+        system += f"\n\nContext:\n\n{context}"
+
+    if config.RAG_DEBUG and metadatas:
+        print("retrieved chunks:")
+        for meta in metadatas:
+            print(f"  {meta['source']} — chunk {meta['chunk_index']}")
+
+    lowered = latest_user_message.lower()
+    for keyword, extra in TOPIC_CONTEXT.items():
+        if keyword in lowered:
+            system += f"\n\n{extra}"
+    return system
+
+
+def response_ai(history: list[dict]) -> str:
+    """history: list of {role, content} chat messages; returns the assistant reply text."""
+    config.ensure_clients()
+    client = config.ensure_openai_client()
+    msgs = [{"role": m["role"], "content": content_to_text(m["content"])} for m in history]
+    system = build_system_prompt(msgs[-1]["content"])
+    messages = [{"role": "system", "content": system}] + msgs
+
+    reply = client.chat.completions.create(
+        model=config.OPENAI_MODEL,
+        messages=messages,
+        tools=TOOLS,
+    ).choices[0].message
+
+    while reply.tool_calls:
+        messages.append(reply)
+        messages.extend(handle_tool_call(reply.tool_calls))
+        reply = client.chat.completions.create(
+            model=config.OPENAI_MODEL,
+            messages=messages,
+            tools=TOOLS,
+        ).choices[0].message
+
+    return reply.content

chunking.py

@@ -0,0 +1,19 @@
+def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
+    """Split text into overlapping chunks, preferring breaks at paragraph or sentence boundaries."""
+    chunks: list[str] = []
+    start = 0
+    while start < len(text):
+        end = min(start + chunk_size, len(text))
+        if end == len(text):
+            chunks.append(text[start:])
+            break
+        halfway = start + (end - start) // 2
+        cut = end
+        for sep in ("\n\n", "\n", ".", "!", "?", " "):
+            i = text.rfind(sep, halfway, end)
+            if i != -1:
+                cut = i + len(sep)
+                break
+        chunks.append(text[start:cut])
+        start = cut - overlap
+    return chunks

config.py

@@ -0,0 +1,60 @@
+import os
+from pathlib import Path
+
+from dotenv import load_dotenv
+from openai import OpenAI
+
+load_dotenv()
+
+ROOT = Path(__file__).parent
+
+OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
+DEEPGRAM_API_KEY = os.getenv("DEEPGRAM_API_KEY")
+PUSHOVER_USER = os.getenv("PUSHOVER_USER")
+PUSHOVER_TOKEN = os.getenv("PUSHOVER_TOKEN")
+
+OPENAI_MODEL = os.getenv("OPENAI_MODEL", "gpt-4.1-mini")
+EMBEDDING_MODEL = os.getenv("EMBEDDING_MODEL", "text-embedding-3-small")
+DEEPGRAM_STT_MODEL = os.getenv("DEEPGRAM_STT_MODEL", "nova-3")
+DEEPGRAM_TTS_MODEL = os.getenv("DEEPGRAM_TTS_MODEL", "aura-2-thalia-en")
+
+RAG_N_RESULTS = int(os.getenv("RAG_N_RESULTS", "3"))
+RAG_CHUNK_SIZE = int(os.getenv("RAG_CHUNK_SIZE", "500"))
+RAG_CHUNK_OVERLAP = int(os.getenv("RAG_CHUNK_OVERLAP", "50"))
+RAG_DEBUG = os.getenv("RAG_DEBUG", "").lower() in ("1", "true", "yes")
+
+_default_chroma = "chroma_db_twin"
+if os.getenv("SPACE_ID") and not os.getenv("CHROMA_PATH"):
+    _default_chroma = "/tmp/chroma_db_twin"
+CHROMA_PATH = Path(os.getenv("CHROMA_PATH", _default_chroma))
+if not CHROMA_PATH.is_absolute():
+    CHROMA_PATH = ROOT / CHROMA_PATH
+
+openai_client: OpenAI | None = None
+deepgram_client = None
+
+
+def ensure_openai_client() -> OpenAI:
+    global openai_client
+    if OPENAI_API_KEY is None:
+        raise RuntimeError("OPENAI_API_KEY is missing. Copy .env.example to .env and add your key.")
+    if openai_client is None:
+        openai_client = OpenAI(api_key=OPENAI_API_KEY)
+    return openai_client
+
+
+def ensure_deepgram_client():
+    global deepgram_client
+    if DEEPGRAM_API_KEY is None:
+        raise RuntimeError("DEEPGRAM_API_KEY is missing. Copy .env.example to .env and add your key.")
+    if deepgram_client is None:
+        from deepgram import DeepgramClient
+
+        deepgram_client = DeepgramClient(api_key=DEEPGRAM_API_KEY)
+    return deepgram_client
+
+
+def ensure_clients() -> None:
+    """Validate env vars and initialize API clients (called at app startup)."""
+    ensure_openai_client()
+    ensure_deepgram_client()

knowledge.md

@@ -0,0 +1,17 @@
+# Persona topic triggers
+
+Keyword-based context is injected when a user's message contains the keyword (case-insensitive). Full persona facts live in `knowledge/*.md` and are retrieved via RAG at query time.
+
+## Topics
+
+### 2011
+***In 2011, Kush was in high school in his early teens, focused on academics and extracurricular activities. Even then, he showed a growing interest in technology and programming.***
+
+### dishes
+***Kush is a big fan of Italian cuisine — pasta, pizza, and gelato. He enjoys experimenting with cooking at home and trying new recipes.***
+
+### sports
+***Kush enjoys playing and watching badminton. He has followed the sport since his youth and still tracks major leagues and international tournaments.***
+
+### vacation
+***Kush loves tropical vacations — sun, beaches, and local culture — and uses them to unplug and come back refreshed.***

knowledge/career.md

@@ -0,0 +1,21 @@
+## Career history (most recent first)
+
+**Symcor, Toronto** (Jan 2023–Sept 2024)
+At Symcor, Kush worked on a cheque image processing platform built in TypeScript and NestJS, integrated with Azure Cognitive Services for OCR and document understanding. The platform processed roughly 40,000 cheques per day, requiring reliable throughput, error handling, and audit trails for financial compliance.
+
+He also built an analyst review dashboard in Next.js with Recharts, Redux, and Shadcn/Tailwind, backed by Supabase SSR with row-level security. Around 200 analysts used the tooling daily across three separate review applications. He integrated Anthropic SDKs into analyst workflows, which cut manual document review time by roughly 40%. The team operated in an NX monorepo with GitLab CI/CD, and affected-only builds reduced pipeline time by about 55%.
+
+**Scotiabank, Toronto** (Aug–Dec 2022)
+During an earlier stint at Scotiabank, Kush built a cashback rewards dashboard in React and TypeScript with Recharts and Redux, serving more than 4 million cardholders. He developed the supporting Node.js, Express, and Postgres API with roughly 85% test coverage via Jest. He also integrated the OpenAI SDK for rewards transaction classification, automating about 70% of manual tagging work that analysts had previously done by hand.
+
+**IBM, Toronto** (July 2021–July 2022)
+At IBM, Kush helped build CIBC's open banking platform in Next.js and TypeScript, reaching over 6 million digital banking customers. He worked on Watson NLU chatbot flows spanning more than 90 intents for CIBC and Sun Life, and built an Elasticsearch-backed document management system handling 400,000+ documents for Lifeworks. He contributed a reusable Tailwind and Radix component library adopted across three internal products, and deployed services on Red Hat OpenShift Kubernetes with pod autoscaling for production workloads.
+
+**MeazureUp, Toronto** (Jan–July 2021)
+At MeazureUp, Kush worked on a restaurant audit and checklist platform deployed to more than 400 franchise locations nationwide. He built a Bull and Redis mass email scheduler that handled 80,000+ sends per week, with attachments and templates stored in AWS S3.
+
+**Sightline Innovation, Toronto** (July 2019–Dec 2020)
+At Sightline Innovation, Kush developed a precision agriculture IoT sensor platform with D3 dashboards monitoring 300+ field sensors. He introduced Elasticsearch and Redis caching that cut repeated query time by roughly 65%, making real-time field data usable for agronomists and operations teams.
+
+**Jumper.ai, Toronto** (June–Dec 2016)
+Early in his career at Jumper.ai, Kush built a Dialogflow payment chatbot with Node.js and Express webhook handlers, persisting conversation state and payment metadata in Firebase Firestore. This was his first production experience shipping conversational interfaces tied to real financial transactions.

knowledge/identity.md

@@ -0,0 +1,23 @@
+## Identity
+
+Kushagra Trivedi holds a Masters in Applied Computer Science from the University of Winnipeg, completed between 2017 and 2019. His graduate research focused on overlapping community detection in social networks, using Voronoi diagrams and tolerance rough sets under an unsupervised learning framework — work that sits at the intersection of graph theory, machine learning, and network analysis.
+
+He is based in Toronto, Canada, where he has built most of his professional career in fintech, banking, and enterprise software. During his master's program, he was awarded the Queen Elizabeth II Diamond Jubilee Scholarship, recognizing academic excellence and community contribution. He is also a published researcher in graph-based community detection, with peer-reviewed work that informed his later interest in data-heavy systems and analytics platforms.
+
+## Current role
+
+Kush is a Senior Software Developer at Scotiabank, a role he has held since October 2024. He is building a pre-authorized payment service in TypeScript and Node.js, backed by TypeORM, Postgres, and a distributed Redis cache. The service handles roughly 80,000 daily transactions, and his performance work reduced end-to-end latency from around 500ms to roughly 40ms on critical paths.
+
+On the frontend, he works in Next.js with React Server Components, Zustand for state, Tailwind and Radix UI for the design system, and SSE streaming for live transaction and status updates. Azure integrations include Blob Storage for document and artifact storage, Service Bus for async messaging between services, and Key Vault for secrets and certificate management.
+
+He also built an internal LLM service layer on top of the OpenAI SDK, designed for provider-agnostic switching, structured JSON output, and token-cost instrumentation so teams can track and control AI spend. He has led design reviews across the organization, and TypeScript patterns he established have been adopted by four downstream teams building on the same platform.
+
+## Personal context
+
+In 2011, Kush was in high school in his early teens, focused on academics and extracurricular activities. Even then, he showed a growing interest in technology and programming — tinkering with computers, exploring how software worked, and gravitating toward problem-solving that would later shape his career in computer science and software engineering.
+
+Kush is a big fan of Italian cuisine. He especially enjoys pasta, pizza, and gelato, and he likes exploring authentic trattorias when he travels. At home, he experiments with cooking — trying new recipes, refining sauces and doughs, and treating the kitchen as a creative outlet away from the screen.
+
+Badminton is a long-standing passion. Kush has played and watched the sport since his youth, and he still follows major leagues and international tournaments. It is both exercise and a way to stay connected to a competitive sport he grew up with.
+
+When he takes time off, Kush gravitates toward tropical destinations — places with sun, beaches, and rich local culture. He values the chance to unplug, swim, explore new food and scenery, and come back to work refreshed rather than burned out.

knowledge/technical.md

@@ -0,0 +1,27 @@
+## Core technical stack
+
+Languages: TypeScript is the primary language across most production work — used for strict typing, shared contracts between frontend and backend, and maintainability across large teams. JavaScript remains in play for scripts, legacy integrations, and rapid prototyping. Python is used for data processing, ML-adjacent work, and lightweight APIs with Flask or FastAPI. Bash covers automation, deployment scripts, and CI pipeline glue. SQL is used daily for Postgres queries, schema design, migrations, and performance tuning.
+
+Frontend: React is the default UI layer, often paired with Next.js for routing, SSR, and React Server Components. State is managed with Zustand for lightweight local and global state, and Redux where predictable action flows and middleware matter at scale. Styling uses Tailwind for utility-first layout, Radix UI and Shadcn for accessible primitives and consistent design systems. Data visualization leans on Recharts for standard charts, D3 for custom interactive visuals, and Chart.js where simpler chart needs fit.
+
+Backend: Node.js is the backbone for most services — NestJS for structured, modular APIs with dependency injection, Express for lean microservices and integrations. Python backends use Flask for smaller services and FastAPI for typed, async-friendly REST APIs. REST is the standard API style across teams, with clear versioning, error contracts, and OpenAPI documentation where applicable.
+
+Databases: Postgres is the primary relational store for transactional workloads, using TypeORM or raw SQL depending on complexity. MongoDB appears where flexible document schemas help. Elasticsearch powers full-text search, analytics, and log-heavy query patterns. Redis handles caching, pub/sub, session storage, and job queues in distributed setups.
+
+DevOps: Azure is heavily used — Blob Storage, Service Bus, Key Vault, and Azure Cognitive Services. Docker containerizes services for consistent dev/prod parity. Kubernetes and Red Hat OpenShift manage orchestration, autoscaling, and multi-environment deployments. GCP and AWS appear for specific integrations such as S3 and cloud-native ML. GitHub and GitLab CI/CD automate build, test, and deploy pipelines; Grafana monitors metrics and alerts in production.
+
+AI/LLM: OpenAI SDK and Anthropic SDK for production LLM calls — structured outputs, streaming, token tracking, and provider switching. Claude Code, MCP, and Copilot support developer workflows, agent tooling, and IDE-assisted development.
+
+Certs: Red Hat EX180 (containers and Kubernetes) and EX080 (DevOps practices) — validates hands-on container and pipeline work.
+
+## What you know well
+
+- Production LLM integration in financial workflows: wiring OpenAI and Anthropic into payment classification, document review, and analyst tooling with guardrails, structured JSON responses, cost instrumentation, and fallback when models fail or hit rate limits.
+
+- Modern frontend architecture: React Server Components for reducing client bundle size and improving time-to-first-byte, SSE streaming for live updates without polling, and real-time dashboards that stay responsive under heavy data refresh.
+
+- Building for scale: Redis and distributed caching to cut latency on hot paths, horizontal scaling patterns, database indexing and query optimization, and CI/CD pipelines with affected-only builds and monorepo tooling to keep feedback loops fast across large codebases.
+
+- Financial domain expertise: pre-authorized payments and high-volume transaction processing, cheque image processing and OCR pipelines, open banking APIs and consent flows, and rewards classification and cashback dashboards — all with compliance and audit expectations in mind.
+
+- TypeScript architecture and multi-team API design: shared types across services, module boundaries that downstream teams can adopt without breaking changes, and design reviews that establish patterns for error handling, logging, and auth replicated across four or more consuming teams.

prompts.py

@@ -0,0 +1,47 @@
+import warnings
+from pathlib import Path
+
+KNOWLEDGE_PATH = Path(__file__).parent / "knowledge.md"
+TOPICS_HEADER = "## Topics"
+
+SYSTEM_MESSAGE = """You are a digital twin of Kushagra Trivedi. When people talk to you, you respond AS Kush — in first person, using his voice, personality, and knowledge.
+Important: do not make things up. If you don't know an answer, say you don't know. The only factual information available to you is what's in this system message and any retrieved Context below. You cannot get any more facts about Kushagra from the internet or make them up.
+"""
+
+
+def _load_topics() -> dict[str, str]:
+    if not KNOWLEDGE_PATH.is_file():
+        warnings.warn(
+            f"knowledge.md not found at {KNOWLEDGE_PATH}; topic keyword context will be empty.",
+            stacklevel=2,
+        )
+        return {}
+
+    text = KNOWLEDGE_PATH.read_text(encoding="utf-8")
+    if TOPICS_HEADER not in text:
+        return {}
+    _, topics_section = text.split(TOPICS_HEADER, 1)
+    return _parse_topics(topics_section)
+
+
+def _parse_topics(section: str) -> dict[str, str]:
+    topics: dict[str, str] = {}
+    current_key: str | None = None
+    current_lines: list[str] = []
+
+    for line in section.splitlines():
+        if line.startswith("### "):
+            if current_key:
+                topics[current_key] = "\n".join(current_lines).strip()
+            current_key = line[4:].strip()
+            current_lines = []
+        elif current_key is not None:
+            current_lines.append(line)
+
+    if current_key:
+        topics[current_key] = "\n".join(current_lines).strip()
+
+    return topics
+
+
+TOPIC_CONTEXT = _load_topics()

rag.py

@@ -0,0 +1,133 @@
+"""ChromaDB-backed RAG: chunk knowledge files, embed, retrieve."""
+
+from __future__ import annotations
+
+import uuid
+from pathlib import Path
+
+import chromadb
+
+import config
+from chunking import chunk_text
+
+ROOT = Path(__file__).parent
+KNOWLEDGE_DIR = ROOT / "knowledge"
+COLLECTION_NAME = "kush_memo"
+
+DOCUMENTS = [
+    {"file": "identity.md", "source": "Identity and Personal Context"},
+    {"file": "career.md", "source": "Career History"},
+    {"file": "technical.md", "source": "Technical Stack"},
+]
+
+_chroma_client: chromadb.ClientAPI | None = None
+_collection: chromadb.Collection | None = None
+
+
+def _get_chroma_client() -> chromadb.ClientAPI:
+    global _chroma_client
+    if _chroma_client is None:
+        config.CHROMA_PATH.mkdir(parents=True, exist_ok=True)
+        _chroma_client = chromadb.PersistentClient(path=str(config.CHROMA_PATH))
+    return _chroma_client
+
+
+def get_collection() -> chromadb.Collection:
+    global _collection
+    if _collection is None:
+        _collection = _get_chroma_client().get_or_create_collection(COLLECTION_NAME)
+    return _collection
+
+
+def _load_documents() -> list[dict[str, str]]:
+    docs: list[dict[str, str]] = []
+    for spec in DOCUMENTS:
+        path = KNOWLEDGE_DIR / spec["file"]
+        if not path.is_file():
+            raise FileNotFoundError(f"Knowledge file not found: {path}")
+        docs.append({"text": path.read_text(encoding="utf-8"), "source": spec["source"]})
+    return docs
+
+
+def _chunk_documents(documents: list[dict[str, str]]) -> tuple[list[str], list[str], list[dict]]:
+    chunks: list[str] = []
+    ids: list[str] = []
+    metadatas: list[dict] = []
+
+    for doc in documents:
+        doc_chunks = chunk_text(
+            doc["text"],
+            chunk_size=config.RAG_CHUNK_SIZE,
+            overlap=config.RAG_CHUNK_OVERLAP,
+        )
+        ids.extend(str(uuid.uuid4()) for _ in doc_chunks)
+        metadatas.extend(
+            {"source": doc["source"], "chunk_index": i} for i in range(len(doc_chunks))
+        )
+        chunks.extend(doc_chunks)
+
+    return chunks, ids, metadatas
+
+
+def embed_texts(texts: list[str]) -> list[list[float]]:
+    client = config.ensure_openai_client()
+    response = client.embeddings.create(
+        input=texts,
+        model=config.EMBEDDING_MODEL,
+    )
+    return [item.embedding for item in response.data]
+
+
+def build_index(*, force_rebuild: bool = False) -> int:
+    """Index knowledge files into ChromaDB. Returns number of chunks stored."""
+    config.ensure_openai_client()
+    collection = get_collection()
+
+    if force_rebuild:
+        existing = collection.get()["ids"]
+        if existing:
+            collection.delete(ids=existing)
+
+    if collection.count() > 0 and not force_rebuild:
+        return collection.count()
+
+    documents = _load_documents()
+    chunks, ids, metadatas = _chunk_documents(documents)
+    if not chunks:
+        return 0
+
+    embeddings = embed_texts(chunks)
+    collection.add(
+        ids=ids,
+        documents=chunks,
+        embeddings=embeddings,
+        metadatas=metadatas,
+    )
+    return len(chunks)
+
+
+def ensure_index() -> None:
+    """Build the vector index when the collection is empty."""
+    if get_collection().count() == 0:
+        count = build_index()
+        print(f"Built RAG index with {count} chunks.")
+
+
+def retrieve(query: str, n_results: int | None = None) -> tuple[str, list[dict]]:
+    """Return joined context text and retrieval metadata for a user query."""
+    config.ensure_openai_client()
+    collection = get_collection()
+    if collection.count() == 0:
+        return "", []
+
+    n = n_results if n_results is not None else config.RAG_N_RESULTS
+    query_embedding = embed_texts([query])[0]
+    results = collection.query(
+        query_embeddings=[query_embedding],
+        n_results=min(n, collection.count()),
+    )
+
+    documents = results["documents"][0]
+    metadatas = results["metadatas"][0]
+    context = "\n\n".join(documents)
+    return context, metadatas

requirements.txt

@@ -0,0 +1,6 @@
+openai>=1.0.0
+python-dotenv>=1.0.0
+gradio>=6.0.0,<7.0.0
+deepgram-sdk>=4.0.0
+requests>=2.31.0
+chromadb>=0.5.0

tools.py

@@ -0,0 +1,75 @@
+import json
+import random
+
+import requests
+
+from config import PUSHOVER_TOKEN, PUSHOVER_USER
+
+PUSHOVER_URL = "https://api.pushover.net/1/messages.json"
+
+
+def send_notification(message: str) -> None:
+    if not PUSHOVER_USER or not PUSHOVER_TOKEN:
+        raise RuntimeError("PUSHOVER_USER and PUSHOVER_TOKEN must be set to use notifications.")
+    payload = {"user": PUSHOVER_USER, "token": PUSHOVER_TOKEN, "message": message}
+    requests.post(PUSHOVER_URL, data=payload, timeout=10)
+
+
+def dice_roll() -> int:
+    return random.randint(1, 6)
+
+
+SEND_NOTIFICATION_FUNCTION = {
+    "name": "send_notification",
+    "description": (
+        "Send a notification to real version of you (Kush) phone via Pushover. "
+        "Use this to alert Kush of important events, completed tasks, time-sensitive information, etc."
+    ),
+    "parameters": {
+        "type": "object",
+        "properties": {
+            "message": {
+                "type": "string",
+                "description": "The message to send",
+            }
+        },
+        "required": ["message"],
+    },
+}
+
+ROLL_DICE_FUNCTION = {
+    "name": "roll_dice",
+    "description": "Roll a dice and return the result",
+    "parameters": {
+        "type": "object",
+        "properties": {},
+    },
+}
+
+TOOLS = [
+    {"type": "function", "function": SEND_NOTIFICATION_FUNCTION},
+    {"type": "function", "function": ROLL_DICE_FUNCTION},
+]
+
+
+def handle_tool_call(tool_calls):
+    tool_results = []
+    for tool_call in tool_calls:
+        function_name = tool_call.function.name
+        args = json.loads(tool_call.function.arguments)
+
+        if function_name == "send_notification":
+            send_notification(args["message"])
+            content = f"Notification sent: {args['message']}"
+        elif function_name == "roll_dice":
+            result = dice_roll()
+            content = f"Dice rolled: {result}"
+        else:
+            content = f"Unknown function: {function_name}"
+
+        tool_results.append({
+            "role": "tool",
+            "content": content,
+            "tool_call_id": tool_call.id,
+        })
+    return tool_results

ui.py

@@ -0,0 +1,119 @@
+import gradio as gr
+
+from chat import response_ai
+from voice import speak_text, transcribe_audio
+
+CSS = """
+#record-box { background: rgb(39, 39, 42) !important; border-radius: 8px !important; padding: 8px !important; }
+#record-box #mic { width: 100% !important; background: transparent !important; }
+#record-box #mic .wrap, #record-box #mic .form, #record-box #mic .empty,
+#record-box #mic .audio-container { background: transparent !important; }
+#record-box #mic .empty, #record-box #mic .audio-container { min-height: 0 !important; }
+#reply-audio { position: absolute !important; width: 1px !important; height: 1px !important;
+  overflow: hidden !important; opacity: 0 !important; pointer-events: none !important; }
+"""
+
+RESET_AUDIO_JS = """
+() => {
+  const app = document.querySelector('gradio-app');
+  const root = (app && app.shadowRoot) ? app.shadowRoot : document;
+  const c = root.querySelector('#reply-audio');
+  if (!c) return;
+  const m = c.querySelector('audio, video');
+  if (!m) return;
+  const play = () => { try { m.currentTime = 0; } catch (e) {} const p = m.play(); if (p && p.catch) p.catch(() => {}); };
+  if (m.readyState >= 2) play(); else m.addEventListener('loadeddata', play, { once: true });
+}
+"""
+
+
+def chat_message(role: str, text: str) -> dict:
+    """Gradio 6 messages format with structured content blocks."""
+    return {"role": role, "content": [{"type": "text", "text": text}]}
+
+
+def voice_transcribe(audio, history):
+    """Step 1: transcribe speech, show it as the user message, and clear the mic."""
+    history = history or []
+    if audio is None:
+        return history, None
+    try:
+        transcript = transcribe_audio(audio)
+    except Exception as e:
+        return history + [chat_message("assistant", f"Error: {e}")], None
+    if not transcript:
+        return history, None
+    return history + [chat_message("user", transcript)], None
+
+
+def voice_reply(history):
+    """Step 2: LLM reply + autoplayed speech."""
+    history = history or []
+    if not history or history[-1]["role"] != "user":
+        yield history, None
+        return
+    try:
+        reply = response_ai(history)
+        audio_path = speak_text(reply)
+    except Exception as e:
+        yield history + [chat_message("assistant", f"Error: {e}")], None
+        return
+
+    yield history + [chat_message("assistant", reply)], audio_path
+
+
+def add_user_message(message, history):
+    """Step 1: show the user message and clear the textbox immediately."""
+    history = history or []
+    if not message or not message.strip():
+        return history, message
+    return history + [chat_message("user", message.strip())], ""
+
+
+def bot_reply(history):
+    """Step 2: generate the assistant reply in chat only (no TTS)."""
+    history = history or []
+    if not history or history[-1]["role"] != "user":
+        yield history
+        return
+    try:
+        reply = response_ai(history)
+    except Exception as e:
+        yield history + [chat_message("assistant", f"Error: {e}")]
+        return
+
+    yield history + [chat_message("assistant", reply)]
+
+
+def create_demo(on_load=None) -> gr.Blocks:
+    with gr.Blocks(title="Kush Digital Twin — Voice") as demo:
+        gr.Markdown("# MultiModal Chat")
+        chatbot = gr.Chatbot(show_label=False, height=420, autoscroll=True)
+        text_in = gr.Textbox(show_label=False, placeholder="Type a message…", container=False)
+        with gr.Column(elem_id="record-box"):
+            mic = gr.Audio(
+                sources=["microphone"],
+                type="filepath",
+                show_label=False,
+                container=False,
+                elem_id="mic",
+            )
+
+        audio_out = gr.Audio(autoplay=True, interactive=False, elem_id="reply-audio", buttons=[])
+
+        mic.stop_recording(
+            voice_transcribe, [mic, chatbot], [chatbot, mic], queue=False,
+        ).then(voice_reply, chatbot, [chatbot, audio_out])
+
+        text_in.submit(
+            add_user_message, [text_in, chatbot], [chatbot, text_in], queue=False,
+        ).then(bot_reply, chatbot, chatbot)
+
+        audio_out.change(None, None, None, js=RESET_AUDIO_JS)
+
+        if on_load is not None:
+            demo.load(on_load, None, None, queue=False)
+
+    demo.queue(default_concurrency_limit=1)
+    demo._deprecated_css = CSS
+    return demo

voice.py

@@ -0,0 +1,35 @@
+import tempfile
+import time
+
+import config
+
+
+def transcribe_audio(audio_path: str) -> str:
+    """Speech-to-text with Deepgram Nova."""
+    config.ensure_clients()
+    with open(audio_path, "rb") as f:
+        audio_bytes = f.read()
+    response = config.deepgram_client.listen.v1.media.transcribe_file(
+        request=audio_bytes,
+        model=config.DEEPGRAM_STT_MODEL,
+        smart_format=True,
+        punctuate=True,
+    )
+    return response.results.channels[0].alternatives[0].transcript.strip()
+
+
+def speak_text(text: str) -> str:
+    """Text-to-speech with Deepgram Aura; returns a unique WAV path (helps autoplay)."""
+    config.ensure_clients()
+    text = text[:2000]
+    stream = config.deepgram_client.speak.v1.audio.generate(
+        text=text,
+        model=config.DEEPGRAM_TTS_MODEL,
+        encoding="linear16",
+        container="wav",
+    )
+    audio_bytes = b"".join(stream)
+    out = tempfile.NamedTemporaryFile(delete=False, suffix=f"_reply_{time.time_ns()}.wav")
+    out.write(audio_bytes)
+    out.close()
+    return out.name