PLAN.md

Codette Multi-Adapter Inference + Chat System — Implementation Plan

Overview

Build three things inside codette-training-lab:

1. HF Upload Scripts + Model Cards — publish each trained adapter to HuggingFace
2. Multi-Adapter Inference Engine — loads Llama 3.1 8B + dynamically switches between 8 LoRA adapters
3. Gradio Real-Time Chat App — interactive UI to test any adapter with streaming responses, deployable to HF Spaces

---

Architecture

codette-training-lab/
├── inference/                    ← NEW
│   ├── __init__.py
│   ├── model_loader.py          ← Core: loads base model + all adapters via PEFT
│   ├── multi_adapter_engine.py  ← Orchestrates multi-perspective generation
│   └── chat_app.py              ← Gradio UI with streaming chat
├── scripts/
│   ├── upload_adapters.py       ← NEW: push adapters to HF Hub
│   └── model_card_template.md   ← NEW: model card for each adapter
└── app.py                       ← NEW: HF Spaces entry point (launches chat_app)

---

Part 1: HF Upload Scripts + Model Cards (2 files)

scripts/upload_adapters.py

• Scans adapters/ directory for trained adapter folders
• For each adapter: creates an HF repo Raiff1982/codette-{adapter_name}, uploads safetensors + adapter_config.json + tokenizer
• Generates a model card from template with correct metadata (base_model, datasets, pipeline_tag, etc.)
• Supports --adapter newton to upload one or --all to upload all 8

scripts/model_card_template.md

• Standard HF model card with YAML frontmatter
• Fields: base_model, datasets, tags, pipeline_tag, license
• Sections: description, intended use, training details, how to use

---

Part 2: Multi-Adapter Inference Engine (2 files)

inference/model_loader.py — CodetteModelLoader

• Loads meta-llama/Llama-3.1-8B-Instruct in 4-bit QLoRA (same config as training)
• Uses PEFT's PeftModel.from_pretrained() to load the first adapter
• Uses model.load_adapter("path", adapter_name="name") for each additional adapter
• Exposes set_active_adapter(name) to switch between loaded adapters at runtime
• Manages tokenizer (Llama 3.1 chat template with apply_chat_template)
• GPU memory footprint: ~5GB base + ~20MB per adapter = ~5.2GB total (fits A10G/T4/consumer GPUs)

inference/multi_adapter_engine.py — CodetteEngine

• Takes a CodetteModelLoader instance
• Single-perspective mode: user picks one adapter, generates with it
• Multi-perspective mode: runs the query through N selected adapters, collects responses, synthesizes
• Synthesis: combines multiple adapter responses into one unified answer (using the multi_perspective adapter or a template)
• Streaming support via TextIteratorStreamer for real-time token output
• Generation params: temperature, top_p, max_tokens, repetition_penalty — all configurable per adapter from adapter_registry.yaml

---

Part 3: Gradio Chat Interface (2 files)

inference/chat_app.py — create_chat_app()

• Chat Tab: streaming chatbot with adapter selector dropdown
  • Dropdown: "Newton", "DaVinci", "Empathy", "Philosophy", "Quantum", "RC-XI", "Multi-Perspective", "Systems", "All (synthesized)"
  • Slider controls: temperature, max tokens, top_p
  • Streaming output token-by-token
  • Chat history with system/user/assistant roles
• Compare Tab: side-by-side adapter comparison
  • Select 2-4 adapters, send same prompt, see responses side by side
  • Quality scores from ReasoningMetrics displayed per response
• Status Tab: model info, loaded adapters, GPU memory, adapter configs
• Theme: gr.themes.Soft() matching existing Codette aesthetic

app.py (project root) — HF Spaces entry point

• Minimal: imports and launches create_chat_app()
• Loads adapters from HF Hub (for Spaces) or local adapters/ directory
• Configurable via env vars: CODETTE_ADAPTER_SOURCE=hub|local, HF_TOKEN, ADAPTER_NAMES

---

Key Design Decisions

1. PEFT multi-adapter — PEFT natively supports loading multiple LoRA adapters on one base model and switching with set_adapter(). No need to load 8 separate models.

2. Streaming — TextIteratorStreamer from transformers, threaded generation, yielded to Gradio chatbot for real-time display.

3. Chat template — Llama 3.1 uses <|begin_of_text|><|start_header_id|>system<|end_header_id|>... format. We use tokenizer.apply_chat_template() which handles this automatically.

4. System prompts from registry — Each adapter's system prompt comes from adapter_registry.yaml, injected as the system message in chat.

5. HF Spaces compatible — The app.py + requirements.txt are structured so deploying to a HF Space with GPU runtime works out of the box.

---

File Count: 7 new files

File	Purpose	~Lines
inference/__init__.py	Package exports	10
inference/model_loader.py	Load base + adapters	200
inference/multi_adapter_engine.py	Generation orchestration	250
inference/chat_app.py	Gradio UI	350
app.py	HF Spaces entry point	50
scripts/upload_adapters.py	Push to HF Hub	180
scripts/model_card_template.md	Model card template	80

Total: ~1,120 lines of new code

---

Execution Order

1. Upload scripts + model cards (so adapters are on HF when chat loads)
2. Model loader (core inference)
3. Multi-adapter engine (orchestration)
4. Chat app + entry point (UI)
5. Test locally, then deploy to HF Space