DOCS/phase0_foundation.md

# Phase 0 — Project Foundation
**Status:** ✅ Complete | **Tests:** 58/58 passed | **Date:** March 2026

---

## Overview
Phase 0 establishes the complete project skeleton before any business logic is written. Every subsequent phase builds on top of this foundation — the directory structure, dependency manifest, centralized config, data contracts (Pydantic models), SQLite schema, and the 4-tab Gradio scaffold are all locked in here.

**Why lock these first?** Schema drift between phases is one of the most common causes of bugs in ML pipelines. By defining the data models and database schema in Phase 0, every later module is guaranteed to produce and consume the same data shapes.

---

## Files Created

| File | Purpose |
|------|---------|
| `requirements.txt` | All project dependencies with rationale comments |
| `.env.example` | Environment variable template (no secrets) |
| `config.py` | Pydantic-settings centralized config singleton |
| `voicevault/__init__.py` | Package init with `__version__`, `__author__` |
| `voicevault/models.py` | All Pydantic data contracts (8 models) |
| `voicevault/asr/__init__.py` | ASR sub-package declaration |
| `voicevault/ingestion/__init__.py` | Ingestion sub-package declaration |
| `voicevault/retrieval/__init__.py` | Retrieval sub-package declaration |
| `voicevault/generation/__init__.py` | Generation sub-package declaration |
| `voicevault/kb/__init__.py` | KB management sub-package declaration |
| `voicevault/tts/__init__.py` | TTS sub-package declaration |
| `voicevault/storage/__init__.py` | Storage sub-package declaration |
| `voicevault/storage/sqlite_store.py` | Full SQLite schema + all CRUD operations |
| `ui/__init__.py` | UI package declaration |
| `ui/tabs/__init__.py` | Tabs sub-package declaration |
| `ui/tabs/ask_tab.py` | Ask tab placeholder (Phase 5 activates it) |
| `ui/tabs/kb_tab.py` | KB Manager tab placeholder |
| `ui/tabs/analytics_tab.py` | Analytics tab placeholder |
| `ui/tabs/settings_tab.py` | Settings tab placeholder |
| `ui/components/__init__.py` | Components sub-package declaration |
| `ui/components/citation_panel.py` | Citation formatter + placeholder |
| `ui/components/audio_controls.py` | Web Speech API JS bridge + placeholder |
| `app.py` | Gradio Blocks entry point (4-tab scaffold) |
| `tests/__init__.py` | Test package declaration |
| `tests/conftest.py` | Shared pytest fixtures |
| `tests/test_phase0.py` | 58 smoke tests covering all Phase 0 deliverables |
| `PLAN.md` | Master E2E implementation plan |
| `DOCS/phase0_foundation.md` | This document |

---

## Architecture Decisions

### 1. Pydantic-Settings for Config (`config.py`)
**What:** A single `VoiceVaultConfig` class inheriting from `pydantic_settings.BaseSettings`. One singleton `cfg` object imported everywhere.

**Why:** Raw `os.environ` calls scattered across modules create maintenance hell. With pydantic-settings:
- Every env var has a typed field with a documented default
- Missing required vars raise a clear `ValidationError` at startup, not a `KeyError` buried in a hot path
- `model_config = SettingsConfigDict(env_file=".env")` means local dev just needs a `.env` file — no export commands
- The `ensure_directories()` method runs once at startup to create `data/`, `data/uploads/`, `models/` — never fails midway through a request

**Key design choice — path helpers as properties/methods, not raw strings:**
```python
# Bad: scattered across modules
path = "data/" + kb_name + "/chroma"

# Good: single definition in config
path = cfg.kb_chroma_dir(kb_name)
```
If the directory layout ever changes, only `config.py` needs to be updated.

**Security fields locked early:**
- `bcrypt_rounds: int = 12` — minimum safe work factor enforced at config level
- `share_link_expiry_days: int = 7` — default expiry for HMAC share tokens
- `allowed_extensions: frozenset` — immutable security whitelist at config level

---

### 2. Pydantic Data Models (`voicevault/models.py`)
**8 models defined:**

| Model | Role | Key Fields |
|-------|------|-----------|
| `DocumentChunk` | A single indexed text chunk | `chunk_id` (UUID), `text_hash` (SHA-256), `page_number`, `section` |
| `IngestionReport` | Result of indexing one document | `status` (success/error/skipped), `chunk_count`, `duration_ms` |
| `RetrievalResult` | A retrieved chunk with scores | `rrf_score`, `rerank_score` |
| `Citation` | One source reference in an answer | `source_file`, `page_number`, `excerpt`, `relevance_score` |
| `QuerySession` | Full query → answer audit record | All latencies, `groq_tokens_used`, `citations` list |
| `KnowledgeBase` | A named document collection | `kb_name` (slug), `password_hash`, `is_protected` property |
| `Document` | A source document in a KB | `file_hash` (SHA-256 for dedup), `is_private` |
| `TranscriptResult` | Whisper ASR output | `transcript`, `model_used`, `confidence`, `query_type` |

**Why lock models in Phase 0?**
Every module from Phase 1 onwards produces or consumes these types. If `DocumentChunk` were defined in `ingestion/` and `RetrievalResult` in `retrieval/`, circular imports would be inevitable. Centralizing in `models.py` breaks all circular dependencies.

**UUID auto-generation:**
```python
chunk_id: str = Field(default_factory=lambda: str(uuid.uuid4()))
```
Every entity gets a unique ID without any external ID generator. Safe for SQLite + ChromaDB + in-memory use.

---

### 3. SQLite Metadata Store (`voicevault/storage/sqlite_store.py`)

**Schema — 4 tables:**

```sql
knowledge_bases  -- KB registry (name, password hash, owner, counts)
documents        -- Per-KB document registry (file hash for dedup, page/chunk count)
chunks           -- Chunk-level metadata (text hash, page, section, language)
query_log        -- Append-only audit trail (anonymized query hash, all latencies)
```

**Critical security decision — query log anonymization:**
The `query_log` table stores `voice_query_hash` (SHA-256 of the query text), **not** the raw query text. This is enforced in the schema (`voice_query_hash TEXT` column, no `voice_query` column) and verified in `test_query_log_schema`. Raw voice queries could contain PII — they are never persisted.

**WAL mode enabled on every connection:**
```python
conn.execute("PRAGMA journal_mode=WAL;")
```
WAL (Write-Ahead Logging) allows concurrent readers while a writer is active — essential for the Analytics tab reading query stats while the main thread is writing a new query log entry.

**Foreign keys with CASCADE:**
```sql
kb_name TEXT REFERENCES knowledge_bases(kb_name) ON DELETE CASCADE
```
Deleting a KB automatically deletes all its documents and chunks. No orphaned rows possible.

**Parameterized queries everywhere — example:**
```python
# CORRECT: parameterized
conn.execute("SELECT * FROM knowledge_bases WHERE kb_name = ?", (kb_name,))

# NEVER: f-string SQL (SQL injection vulnerability)
# conn.execute(f"SELECT * FROM knowledge_bases WHERE kb_name = '{kb_name}'")
```
This pattern is enforced throughout the module. The test suite verifies the schema is correct but also that raw queries are never used (code review confirms all `?` placeholders).

**Idempotent initialization:**
`initialize_database()` uses `CREATE TABLE IF NOT EXISTS` — safe to call on every app startup. The application calls it in `_startup()` before accepting any requests.

---

### 4. Gradio App Scaffold (`app.py`)

**4-tab Blocks layout:**
```
gr.Blocks
  └── gr.Tabs
        ├── Tab 1: 🎙️ Ask VoiceVault   ← build_ask_tab()
        ├── Tab 2: 📂 Knowledge Bases   ← build_kb_tab()
        ├── Tab 3: 📊 Analytics          ← build_analytics_tab()
        └── Tab 4: ⚙️ Settings           ← build_settings_tab()
```

Each tab is a separate function in its own module (`ui/tabs/`). This enables:
- Phase-by-phase activation: each tab becomes functional as its phase completes
- Independent testing of each tab builder
- Clear separation — the tab builder returns nothing, just renders into the active Blocks context

**Startup sequence:**
```python
_startup()      # ensures directories, logs config summary (no secrets)
app = build_app()  # constructs Gradio Blocks
app.launch(...)    # binds to host:port
```

**Gradio version compatibility:**
Discovered during testing that Gradio 6.x moved `theme` and `css` from `gr.Blocks(...)` to `launch(...)`. The test suite caught this immediately (`test_gradio_app_builds`), and the fix was isolated to `app.py`. This is an example of why Phase 0 tests exist — catching API drift before it causes runtime failures.

---

### 5. Web Speech API Bridge (`ui/components/audio_controls.py`)

The JavaScript that drives browser TTS is declared as a module constant (`WEB_SPEECH_JS`) in Phase 0. It will be injected via `gr.HTML` in Phase 5.

**Why declare it now?**
The JS bridge is a security-sensitive piece (it executes in the browser). By declaring it as a constant rather than building it dynamically, it is:
- Auditable as a static artifact (code review can inspect it)
- Testable (`test_tts_html_contains_js` verifies `speechSynthesis` and `_vv_tts` are present)
- Not constructable from user input (no injection surface)

---

## Test Results

```
58 passed, 0 failed — 18.91s

TestConfig          (10 tests) — config loading, types, defaults, path helpers, security
TestModels           (9 tests) — all 8 Pydantic models instantiate and validate correctly
TestSQLiteSchema     (6 tests) — tables created, idempotent, schema columns verified
TestSQLiteCRUD      (11 tests) — full CRUD round-trips for all tables
TestPackageImports  (14 tests) — every __init__.py and public function importable
TestUIComponents     (8 tests) — citation formatter, TTS HTML, Gradio build
```

**Warnings noted (not failures):**
- `datetime.utcnow()` deprecation — Pydantic v2 internally calls this on default factories. Not our code. Will resolve when Pydantic updates its internals. Tracked for future upgrade.

---

## Security Audit — Phase 0

| Check | Status | Notes |
|-------|--------|-------|
| No API keys in code | ✅ Pass | `.env.example` has placeholders only |
| No hardcoded secrets | ✅ Pass | All sensitive values via env vars |
| Parameterized SQL | ✅ Pass | All queries use `?` placeholders |
| Query log anonymization | ✅ Pass | `voice_query_hash` only, no raw text |
| bcrypt rounds ≥ 12 | ✅ Pass | Enforced by config default + test |
| Extension whitelist defined | ✅ Pass | `frozenset` in config — immutable |
| Data dir not git-tracked | ✅ Pass | `.gitignore` covers `data/` |
| `.env` not committed | ✅ Pass | `.gitignore` covers `.env` |

---

## Progress Tracker Update

| Phase | Status | Tests | Docs |
|-------|--------|-------|------|
| **Phase 0 — Foundation** | ✅ Done | ✅ 58/58 | ✅ Done |
| Phase 1 — Ingestion | ⬜ Next | ⬜ | ⬜ |
| Phase 2 — Retrieval | ⬜ | ⬜ | ⬜ |
| Phase 3 — ASR | ⬜ | ⬜ | ⬜ |
| Phase 4 — Generation | ⬜ | ⬜ | ⬜ |
| Phase 5 — UI & Access | ⬜ | ⬜ | ⬜ |