README.md

metadata

title: MemoryBridge AAC
emoji: 🧠
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 6.14.0
python_version: '3.10'
app_file: app.py
pinned: false

MemoryBridge

Assistive AAC communication system for people with cerebral palsy and other motor/speech disabilities. Generates personalized, contextually relevant candidate responses by fusing text retrieval (knowledge graphs + semantic search) with real-time vision signals (gestures, affect, air-sign letters).

Architecture

Partner Query
      │
      ▼
┌─────────────────────────────────────────────┐
│           PARALLEL EXECUTION BLOCK           │
│  ┌──────────────────┐  ┌──────────────────┐  │
│  │   TEXT PATH       │  │   VISION PATH    │  │
│  │  Intent → Router  │  │  MediaPipe Loop  │  │
│  │  KG + FAISS       │  │  Signal Buffer   │  │
│  │  Reranker         │  │                  │  │
│  └────────┬──────────┘  └────────┬─────────┘  │
│           └──────────┬───────────┘            │
│                      ▼                        │
│            WEIGHTED FUSION LAYER              │
│         (CoT reasoning, Qwen3-32B)            │
│                      │                        │
│                      ▼                        │
│           RESPONSE GENERATION                 │
│       3 candidates + 1 turnaround             │
└─────────────────────────────────────────────┘

Latency target: < 5 seconds to first token.

Quickstart

# 1. Clone and install
pip install -r requirements.txt

# 2. Set up API keys
cp .env.example .env
# edit .env with your keys

# 3. Build persona data (one-time)
python scripts/build_persona.py --persona alex_rivera
python scripts/build_kg.py --persona alex_rivera
python scripts/build_faiss.py --persona alex_rivera

# 4. Launch UI
python -m memorybridge.ui.gradio_app

# 5. Run evaluation
python scripts/run_eval.py --mode S3

Configuration

All models and thresholds are configurable in config/settings.yaml. Swap any model (including provider) with zero code changes:

generation:
  provider: "anthropic"          # groq | together | openai | anthropic | ollama
  model: "claude-sonnet-4-6-20250514"

Project Structure

memorybridge/
├── config/          # settings.yaml + persona configs
├── core/            # pipeline orchestrator, model registry, schemas
├── text_path/       # intent decomposition, KG + FAISS retrieval, reranking
├── vision_path/     # gesture, affect, air-sign detection, signal buffer
├── fusion/          # weighted fusion with CoT reasoning
├── generation/      # response generation, prompt builder, style scorer
├── feedback/        # learning from user selections
├── memory/          # profile loader, KG/FAISS builders, conversation history
├── data/            # persona content, indices, knowledge graphs
├── evaluation/      # RAGAS eval, latency tracking, ablation study
├── ui/              # Gradio interface with gaze selection
└── scripts/         # build and evaluation CLI scripts

Personas

• Alex Rivera (deep persona): 34, Buffalo NY, spastic diplegia CP, Bills fan
• Persona 2 / 3: Shallow personas for generalizability testing

Evaluation

Three ablation conditions:

• S1 — No retrieval (baseline)
• S2 — Profile-only retrieval
• S3 — Full system (KG + FAISS + vision fusion)

Metrics: RAGAS groundedness/faithfulness, E2E latency, gesture-response alignment.