README.md

---
title: MemoryBridge AAC
emoji: 🧠
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: "6.14.0"
python_version: "3.10"
app_file: app.py
pinned: false
---

# MemoryBridge

Assistive AAC communication system for people with cerebral palsy and other motor/speech disabilities. Generates personalized, contextually relevant candidate responses by fusing text retrieval (knowledge graphs + semantic search) with real-time vision signals (gestures, affect, air-sign letters).

## Architecture

```
Partner Query
      │
      ▼
┌─────────────────────────────────────────────┐
│           PARALLEL EXECUTION BLOCK           │
│  ┌──────────────────┐  ┌──────────────────┐  │
│  │   TEXT PATH       │  │   VISION PATH    │  │
│  │  Intent → Router  │  │  MediaPipe Loop  │  │
│  │  KG + FAISS       │  │  Signal Buffer   │  │
│  │  Reranker         │  │                  │  │
│  └────────┬──────────┘  └────────┬─────────┘  │
│           └──────────┬───────────┘            │
│                      ▼                        │
│            WEIGHTED FUSION LAYER              │
│         (CoT reasoning, Qwen3-32B)            │
│                      │                        │
│                      ▼                        │
│           RESPONSE GENERATION                 │
│       3 candidates + 1 turnaround             │
└─────────────────────────────────────────────┘
```

**Latency target**: < 5 seconds to first token.

## Quickstart

```bash
# 1. Clone and install
pip install -r requirements.txt

# 2. Set up API keys
cp .env.example .env
# edit .env with your keys

# 3. Build persona data (one-time)
python scripts/build_persona.py --persona alex_rivera
python scripts/build_kg.py --persona alex_rivera
python scripts/build_faiss.py --persona alex_rivera

# 4. Launch UI
python -m memorybridge.ui.gradio_app

# 5. Run evaluation
python scripts/run_eval.py --mode S3
```

## Configuration

All models and thresholds are configurable in `config/settings.yaml`. Swap any model (including provider) with zero code changes:

```yaml
generation:
  provider: "anthropic"          # groq | together | openai | anthropic | ollama
  model: "claude-sonnet-4-6-20250514"
```

## Project Structure

```
memorybridge/
├── config/          # settings.yaml + persona configs
├── core/            # pipeline orchestrator, model registry, schemas
├── text_path/       # intent decomposition, KG + FAISS retrieval, reranking
├── vision_path/     # gesture, affect, air-sign detection, signal buffer
├── fusion/          # weighted fusion with CoT reasoning
├── generation/      # response generation, prompt builder, style scorer
├── feedback/        # learning from user selections
├── memory/          # profile loader, KG/FAISS builders, conversation history
├── data/            # persona content, indices, knowledge graphs
├── evaluation/      # RAGAS eval, latency tracking, ablation study
├── ui/              # Gradio interface with gaze selection
└── scripts/         # build and evaluation CLI scripts
```

## Personas

- **Alex Rivera** (deep persona): 34, Buffalo NY, spastic diplegia CP, Bills fan
- **Persona 2 / 3**: Shallow personas for generalizability testing

## Evaluation

Three ablation conditions:
- **S1** — No retrieval (baseline)
- **S2** — Profile-only retrieval
- **S3** — Full system (KG + FAISS + vision fusion)

Metrics: RAGAS groundedness/faithfulness, E2E latency, gesture-response alignment.