docs/architecture.md

# AI Time Machine Architecture

Date: 2026-06-06

## Purpose

This document defines the first project architecture and skeleton direction for the AI Time Machine Gradio app. The goal is a modular, testable, model-backed application that can support the MVP voice loop now and later replace individual components such as STT, TTS, LLM, or the whole speech conversation stack without rewiring the app.

The architecture should make it practical for two people to work independently on different areas, for example cockpit/UI and speech/conversation, while sharing stable contracts and fixture data.

## Architectural Decision

Use a ports-and-adapters architecture with distributed deployment: Gradio UI on Hugging Face Spaces, Together AI API for LLM inference, and Modal for audio models (Nemotron STT, Qwen3-TTS).

The architecture transitioned from a single-process design to this distributed approach. The ports-and-adapters pattern made the transition seamless — adapters simply call remote endpoints instead of local models, while the UI and orchestration code remain unchanged. The single-process mode still works for local development via the `dev` and `fixture` adapter profiles.

High-level shape:

```text
Gradio UI (HF Spaces)
  -> UI event handlers
  -> Application use cases
  -> Domain contracts
  -> Ports
  -> Adapters:
       LLM: Together AI API (Qwen3-8B)
       STT: Modal endpoint (Nemotron)
       TTS: Modal endpoint (Qwen3-TTS) or local Kokoro
       Storage, traces: local
```

## Confirmed Implementation Decisions

- Persistence starts with `JsonlEncounterStore`.
- The old `src/app.py` entrypoint is replaced by root `app.py` for Hugging Face Space deployment.
- The old `src/landing_page.html` prototype is not reused.
- Fixture adapters are the default local skeleton profile, but they are normal adapter implementations rather than app-wide mock branches.
- The LLM is Qwen3-8B via Together AI API. Qwen3-4B remains available as a lower-cost alternative.
- Modal hosts the audio models (Nemotron STT, Qwen3-TTS). Together AI handles LLM inference with native JSON mode support.

## Goals

- Keep UI, orchestration, model calls, audio, persistence, tests, and evals loosely coupled.
- Make STT + LLM + TTS replaceable with a future speech-to-speech adapter.
- Shape the system around streaming from day one.
- Use Pydantic contracts for generated JSON and cross-component data.
- Treat model JSON outputs as hard contracts with validation and repair.
- Include fixture encounters so UI and conversation work can proceed independently.
- Support real model integration in the MVP.
- Avoid technical debt from mock mode by making fixture adapters normal test/dev adapters behind the same ports.
- Keep deployment optimized for Hugging Face Spaces.
- Track model parameter budgets explicitly for hackathon compliance.

## Non-Goals

- No separate backend service for the initial skeleton.
- No long-lived frontend source of truth. The frontend can animate and render state, but Python owns encounter state.
- No hard dependency on any single STT, TTS, or LLM implementation.
- No stretch avatar, generated image, llama.cpp, or full speech-to-speech runtime on the MVP critical path.

## Core Principles

1. Domain contracts are stable.

   Components exchange typed objects such as `Destination`, `Persona`, `ConversationTurn`, `VoiceProfile`, and `Souvenir`, not ad hoc dictionaries.

2. Application use cases own orchestration.

   Gradio handlers should be thin. They should translate user actions into use-case calls and render returned events.

3. Adapters are replaceable.

   Real models, fixture data, local inference, Modal endpoints, and future speech-to-speech engines implement the same ports.

4. Streaming is an event stream.

   Long-running operations emit typed `TemporalEvent` records: launch state, partial transcript, final transcript, assistant text, audio status, visual cue, error, or souvenir.

5. Mocks are not a forked app path.

   Fixture adapters should be used for local development, tests, and UI isolation. They should not introduce `if mock_mode` branches inside domain or use-case code.

6. Failures are typed.

   Contract validation, model failures, audio failures, and storage failures should map to recoverable application errors that can be shown in the cockpit UI.

## Runtime Architecture

### Single Process, Internal Boundaries

The MVP should run as one Gradio Space process:

```text
root app.py
  imports time_machine.ui.gradio_app:create_app
  creates dependency container
  launches Gradio Blocks
```

Inside the process, code should be organized as if services were independent:

- UI calls application services.
- Application services depend on ports.
- Ports are implemented by adapters.
- Adapters may call local models, Hugging Face APIs, Modal endpoints, or fixtures.

This gives us an easy future migration path:

```text
Local Qwen TTS adapter
  can be replaced by
ModalQwenTTSAdapter
  without changing
Conversation use case or Gradio UI
```

### Session Model

The app should support multiple simultaneous HF Space users as long as it does not materially expand scope.

Rules:

- No global mutable active encounter.
- Each user interaction carries or resolves a `session_id`.
- Gradio `State` stores per-session state for the active encounter.
- Persistence stores encounters by `session_id` and `encounter_id`.
- Model adapters may be shared singletons if they are thread-safe or protected.

For the MVP, in-memory session state is acceptable for active conversations, but saved encounters/passport stamps should go through a storage port.

## Major Layers

### 1. UI Layer

Responsibilities:

- Build the Gradio Blocks app.
- Render cockpit, portal, controls, transcript, audio controls, and souvenir view.
- Load CSS/JS assets.
- Map UI actions to application use cases.
- Render typed streaming events.

The UI layer may include Python, HTML, CSS, and JavaScript. It is not just static assets. Its boundary is behavioral: it owns presentation and interaction, but not conversation logic or model behavior.

Expected files:

```text
src/time_machine/ui/
  gradio_app.py
  handlers.py
  view_models.py
  assets/
    cockpit.html
    cockpit.css
    cockpit.js
```

UI input examples:

- `launch_requested`
- `coordinate_prompt_changed`
- `audio_chunk_received`
- `text_turn_submitted`
- `souvenir_requested`
- `encounter_save_requested`

UI output examples:

- `machine_state_changed`
- `destination_revealed`
- `persona_locked`
- `transcript_partial`
- `transcript_final`
- `character_text_delta`
- `character_audio_ready`
- `souvenir_ready`
- `recoverable_error`

### 2. Application Layer

Responsibilities:

- Coordinate launch, destination generation, persona generation, voice loop, conversation, TTS, souvenir generation, and persistence.
- Convert user actions into domain state changes and event streams.
- Keep orchestration independent of Gradio.

Expected files:

```text
src/time_machine/application/
  container.py
  encounter_service.py
  speech_orchestrator.py
  souvenir_service.py
  session_state.py
```

Core use cases:

- `start_encounter(mode, coordinate_prompt, session_id) -> Iterator[TemporalEvent]`
- `handle_audio_stream(encounter_id, audio_chunks) -> Iterator[TemporalEvent]`
- `handle_text_turn(encounter_id, text) -> Iterator[TemporalEvent]`
- `generate_souvenir(encounter_id) -> Iterator[TemporalEvent]`
- `save_encounter(encounter_id) -> SavedEncounter`

### 3. Domain Layer

Responsibilities:

- Define core business objects and invariants.
- Define event types.
- Define errors.
- Contain no Gradio, model runtime, file system, or network dependencies.

Expected files:

```text
src/time_machine/domain/
  models.py
  events.py
  errors.py
```

Primary Pydantic models:

- `Destination`
- `VisualMotif`
- `Persona`
- `VoiceProfile`
- `ConversationTurn`
- `ConversationHistory`
- `Souvenir`
- `EncounterSession`
- `ModelSpec`
- `ModelBudget`

### 4. Ports Layer

Responsibilities:

- Define interfaces that application services depend on.
- Keep external systems replaceable.

Expected files:

```text
src/time_machine/ports/
  world.py
  persona.py
  conversation.py
  speech.py
  souvenir.py
  storage.py
  model_registry.py
  trace.py
```

Core ports:

```text
DestinationGenerator
  generate_destination(mode, coordinate_prompt, seed) -> Destination

PersonaGenerator
  generate_persona(destination, seed) -> Persona

ConversationEngine
  respond(destination, persona, history, latest_user_text) -> ConversationResponse
  stream_response(...) -> Iterator[ConversationEvent]

STTAdapter
  transcribe(audio) -> Transcript
  stream_transcript(audio_chunks) -> Iterator[TranscriptEvent]

TTSAdapter
  prepare_voice(persona) -> VoiceProfile
  synthesize(text, voice_profile, prosody_hint) -> AudioResult
  stream_audio(text, voice_profile, prosody_hint) -> Iterator[AudioEvent]

SpeechConversationAdapter
  stream_conversation(persona, destination, audio_chunks, history) -> Iterator[TemporalEvent]

SouvenirGenerator
  generate_souvenir(destination, persona, history) -> Souvenir

EncounterStore
  save_encounter(encounter) -> SavedEncounter
  load_encounter(encounter_id) -> EncounterSession
  list_saved(session_id) -> list[SavedEncounter]

TraceSink
  record_event(event) -> None
  record_model_io(component, prompt, raw_output, parsed_output, metadata) -> None
```

The important design point is `SpeechConversationAdapter`. The initial implementation can compose `STTAdapter + ConversationEngine + TTSAdapter`, but a future speech-to-speech model can implement the same high-level behavior.

### 5. Adapters Layer

Responsibilities:

- Implement ports using real models, fixture data, local files, or remote endpoints.
- Own dependency-specific code and model runtime quirks.

Expected files:

```text
src/time_machine/adapters/
  fixtures/
    fixture_world.py
    fixture_persona.py
    fixture_conversation.py
    fixture_speech.py
  llm/
    qwen_structured.py
    cloud_completion.py
    prompt_templates.py
    json_repair.py
  stt/
    nemotron.py
    distil_whisper.py
    whisper_stt.py
  tts/
    qwen_tts.py
    magpie.py
    kokoro.py
  speech_to_speech/
    composed.py
    personaplex.py
  storage/
    jsonl_store.py
    sqlite_store.py
  trace/
    jsonl_trace.py
```

- `llm/cloud_completion.py` — Cloud LLM completion function for Together/OpenRouter.
- `stt/whisper_stt.py` — Whisper STT adapter for development.

Fixture adapters are first-class adapter implementations for tests and development. They should not change the app architecture.

## Streaming Design

The app should stream typed events instead of returning one giant response.

Example event types:

```text
MachineStateEvent
DestinationEvent
PersonaEvent
TranscriptPartialEvent
TranscriptFinalEvent
ConversationTextDeltaEvent
ConversationTextFinalEvent
AudioChunkEvent
AudioReadyEvent
VisualCueEvent
SouvenirEvent
RecoverableErrorEvent
FatalErrorEvent
TraceEvent
```

For MVP turn-based audio, the same event model still applies:

1. UI receives microphone clip.
2. App emits `TranscriptPartialEvent` or `TranscriptFinalEvent`.
3. Conversation engine emits text deltas or final text.
4. TTS emits audio-ready or audio chunks.
5. UI renders each event.

For future true streaming:

1. UI sends audio chunks.
2. STT emits partial transcript events.
3. Conversation engine may wait for final transcript or respond incrementally.
4. TTS streams audio chunks.
5. UI renders the same event types.

For future speech-to-speech:

1. UI sends audio chunks.
2. `SpeechConversationAdapter` emits transcript, character audio, and state events.
3. UI and application orchestration remain mostly unchanged.

## JSON Contract Strategy

Generated destination, persona, conversation metadata, voice profile, and souvenir outputs should be treated as hard contracts.

Process:

1. Prompt includes the expected JSON schema or field contract.
2. Adapter parses model output.
3. Pydantic validates the object.
4. If validation fails, adapter performs one bounded repair attempt.
5. If repair fails, adapter raises a typed `ContractValidationError`.
6. Application layer converts this to a recoverable cockpit error or fixture fallback only where explicitly configured.

The domain should never consume unvalidated model dictionaries.

## Model Registry And Parameter Budget

Track all model choices in a small config file, likely:

```text
config/models.yaml
```

Suggested fields:

```yaml
models:
  - role: llm
    provider: huggingface
    model_id: Qwen/...
    parameters_billion: 8.0
    enabled: true
    runtime: transformers
    license: unknown
    notes: primary instruction model
  - role: stt
    provider: huggingface
    model_id: nvidia/nemotron-3.5-asr-streaming-0.6b
    parameters_billion: 0.6
    enabled: true
    runtime: nemo
    notes: preferred streaming ASR
```

Add an eval or test that sums enabled model parameters and fails above 32B.

This should be explicit because the hackathon constraint is central to judging.

## Persistence

Use a storage port from the start.

MVP storage should be simple:

- `JsonlEncounterStore` for easy inspection and low setup.
- Optional `SQLiteEncounterStore` if querying saved passport stamps becomes useful.

Saved data:

- Encounter metadata.
- Destination.
- Persona.
- Conversation transcript.
- Souvenir.
- Model registry snapshot.
- Trace file references.

Suggested local paths:

```text
data/
  encounters/
  traces/
  souvenirs/
```

Do not let persistence leak into UI or model adapters. UI asks the application layer to save. Application layer calls `EncounterStore`.

## Testing Strategy

Use three layers of tests.

### Unit Tests

Location:

```text
tests/
```

Coverage:

- Domain model validation.
- JSON contract parsing.
- Event stream ordering.
- Encounter state transitions.
- Storage adapter behavior.
- Model budget calculation.

These should run quickly and use fixture adapters.

### Contract Tests

Coverage:

- Every adapter implementation satisfies its port.
- Real model adapters can parse and validate expected outputs.
- TTS adapters return playable audio metadata.
- STT adapters return transcripts in the expected shape.

Real model contract tests may be marked slow or integration-only.

### UI Smoke Tests

Coverage:

- Gradio app imports.
- App can be created without launching.
- Fixture-backed launch flow reaches destination/persona state.
- Text-turn flow returns transcript and response events.

Playwright can be added once the cockpit UI exists.

## Eval Strategy

Keep evals separate from tests.

Location:

```text
evals/
```

Initial evals:

- Destination JSON validity.
- Persona JSON validity.
- Ordinary-person constraint.
- No famous historical figures.
- Response length suitable for TTS.
- In-character continuity.
- Persona worldview consistency.
- Souvenir completeness.
- Safety around painful historical contexts.
- Model parameter budget compliance.
- Latency snapshots for STT, LLM, and TTS.

Eval fixtures:

```text
evals/fixtures/
  coordinate_prompts.jsonl
  known_bad_personas.jsonl
  conversation_turns.jsonl
```

Eval outputs:

```text
evals/runs/
  YYYYMMDD-HHMMSS/
    results.json
    failures.jsonl
    traces/
```

Evals should call the same ports as the application. They should not duplicate prompt logic.

## Fixture Encounters

Include sample fixture encounters for development.

Location:

```text
fixtures/encounters/
  edo_1712.json
  orbital_repair_bay_2194.json
  medieval_port_1180.json
```

Each fixture should include:

- Destination.
- Visual motifs.
- Persona.
- Voice profile.
- Seed conversation turns.
- Souvenir.

This lets UI development, storage work, and transcript rendering continue before real models are stable.

## Configuration

Prefer `pyproject.toml` as the main project/dependency definition.

For Hugging Face Space compatibility, add a minimal deployment file only if required by the selected Space builder. If both are needed, keep `pyproject.toml` as source of truth and make any compatibility file small and mechanical.

Suggested config files:

```text
pyproject.toml
config/
  app.yaml
  models.yaml
  prompts.yaml
```

Runtime settings:

- `TIME_MACHINE_ENV`: `local`, `space`, `test`
- `TIME_MACHINE_ADAPTER_PROFILE`: `fixture`, `dev`, `local_models`, `modal`
- `TIME_MACHINE_DATA_DIR`
- `TIME_MACHINE_TRACE_DIR`
- `TIME_MACHINE_LLM_API_KEY`: Together AI / OpenRouter API key
- `TIME_MACHINE_LLM_BASE_URL`: Cloud LLM endpoint (default: Together AI)
- `TIME_MACHINE_LLM_MODEL`: Cloud LLM model ID
- `TIME_MACHINE_WHISPER_MODEL`: Whisper model size for dev STT
- model-specific cache/runtime settings

The adapter profile should only affect dependency composition in `container.py`. It should not create branches throughout the app.

## Proposed Skeleton

Target structure:

```text
app.py
pyproject.toml
config/
  app.yaml
  models.yaml
docs/
  architecture.md
  hackathon_details.md
  ai_time_machine_idea.md
  tech_stack_decision.md
fixtures/
  encounters/
    edo_1712.json
    orbital_repair_bay_2194.json
src/
  time_machine/
    __init__.py
    application/
      __init__.py
      container.py
      encounter_service.py
      session_state.py
      speech_orchestrator.py
      souvenir_service.py
    domain/
      __init__.py
      errors.py
      events.py
      models.py
    ports/
      __init__.py
      conversation.py
      model_registry.py
      persona.py
      souvenir.py
      speech.py
      storage.py
      trace.py
      world.py
    adapters/
      __init__.py
      fixtures/
      llm/
      speech_to_speech/
      stt/
      storage/
      trace/
      tts/
    prompts/
      destination.md
      persona.md
      conversation.md
      souvenir.md
    ui/
      __init__.py
      gradio_app.py
      handlers.py
      view_models.py
      assets/
        cockpit.html
        cockpit.css
        cockpit.js
tests/
  unit/
  contract/
  smoke/
evals/
  fixtures/
  runners/
  validators/
data/
  .gitkeep
```

Existing `src/app.py` and `src/landing_page.html` can be removed or migrated when the skeleton is created. The preferred future entrypoint is root `app.py` because it is simple for Space deployment.

## Implementation Plan: Walk / Run / Sprint

### Walk: Text Conversation With Cloud LLM

- Together AI API adapter for Qwen3-8B (destination, persona, conversation, souvenir).
- Kokoro local TTS for character voice output.
- Text input only — no microphone yet.
- Fixture adapters remain available for offline development.
- Validates end-to-end logic: launch → destination → persona → conversation → souvenir.

Result: the core conversation loop works with real LLM output and audio replies.

### Run: Full Voice Loop With Modal Audio

- Add Modal endpoint for Nemotron STT.
- Add Modal endpoint for Qwen3-TTS.
- Push-to-talk clip audio input from microphone.
- Full voice loop: user speaks → STT → LLM → TTS → character speaks.
- Persistence and traces.

Result: the demo works as a voice-first experience.

### Sprint: Streaming Audio And Text

- Streaming STT partial transcripts.
- Streaming LLM text deltas.
- Streaming TTS audio chunks.
- Only attempted if Run works cleanly.

Result: lower perceived latency and a more polished experience.

## Resolved Decisions

Decisions that were deferred during skeleton design, now resolved:

1. **SQLite vs JSONL**: Not needed for MVP. JSONL is sufficient for encounter storage.
2. **Qwen3 4B vs 8B**: Decided — Qwen3-8B via Together AI API. Quality is noticeably better. 4B remains available as a lower-cost alternative.
3. **Modal scope**: Modal hosts STT (Nemotron) and TTS (Qwen3-TTS). LLM inference is via Together AI API.
4. **OpenRouter vs Together AI**: Together AI chosen as primary LLM provider (native JSON mode, competitive pricing). OpenRouter is an acceptable fallback.
5. **Voice loop strategy**: Text-first input for Walk phase. Push-to-talk clip audio for Run phase. Streaming only in Sprint phase if time allows.