docs/proactive_adaptive_system.md

# Adaptive Proactive Messaging System — Design Doc

_Last updated: 2026-04-21_

---

## Overview

The proactive pipeline currently sends one fixed-schedule daily message per user
with a cycling topic pattern. This doc describes the full adaptive system discussed
to replace it: per-user optimal send time, topic drill-down based on engagement,
character weighting, and a feedback loop that adjusts both time and topic automatically.

---

## 1. Trigger architecture change

### Current
```
GitHub Action → daily cron 07:00 UTC → trigger_proactive Space → /proactive/run (all users)
```

### New
```
GitHub Action → hourly cron '0 * * * *' → trigger_proactive Space → /proactive/run
    → reads proactive_schedule table
    → sends only to users whose send_hour_utc == current_hour
```

The GitHub Action fires every hour. Each hour it queries `proactive_schedule`
for users scheduled at that UTC hour and runs the pipeline only for those users.

**Why hourly trigger instead of per-user cron:**
- Simpler infrastructure — one cron, one trigger Space
- The per-user variation is handled in the application layer, not the scheduler
- Easy to add/remove users without touching the cron

---

## 2. Onboarding — age capture

At first login, after collecting name and country, ask the user's age.

- Store in `user_profiles.age` (new field, integer)
- If user skips: mark as `null`, fall back to default slot
- Age is used only for time prediction — not surfaced in conversation

---

## 3. Default send time prediction

Age + living country define a best-guess local hour for the first proactive message.
If either is missing, use the global fallback slot (19:00 local).

### Age × lifestyle mapping

| Age range | Assumed lifestyle | Preferred slots (local) |
|-----------|------------------|------------------------|
| 13–22 | Student | 16:00–19:00, 21:00–23:00 |
| 23–35 | Young professional | 07:30–08:30, 12:30, 20:00–22:00 |
| 36–55 | Working adult | 07:00–08:00, 12:00–13:00, 20:00–21:00 |
| 56–70 | Pre/post retirement | 08:00–10:00, 15:00–16:00 |
| 70+ | Retired | 09:00–10:00, 14:00–15:00 |

### Country → timezone mapping
Use `pytz` or a country→UTC offset table to convert local slot → UTC hour stored
in `proactive_schedule.send_hour_utc`.

### Fallback
If age or country missing: default to 19:00 local (or 19:00 UTC if timezone unknown).

---

## 4. proactive_schedule table (new)

```sql
create table proactive_schedule (
    user_id          uuid primary key references auth.users(id),
    send_hour_utc    smallint not null default 19,  -- 0–23
    timezone         text,                           -- e.g. 'Australia/Sydney'
    local_hour       smallint,                       -- cached local equivalent
    no_response_streak  smallint default 0,          -- consecutive misses
    last_sent_at     timestamptz,
    last_responded_at timestamptz,
    created_at       timestamptz default now(),
    updated_at       timestamptz default now()
);
```

---

## 5. Topic pool — full list

The proactive pipeline cycles through these types (replacing the current 3-type pattern):

| Type | Source | Description |
|------|--------|-------------|
| `news` | Tavily search | News on user's current interest topic (see drill-down §6) |
| `personal_event` | `life_events` table | Follow up on a specific recorded life event |
| `personal_dialogic` | `past_dialogues` / FAISS | Reopen an unresolved topic from chat history |
| `story` | `story_chapters_*.json` | Follow up on the current chapter of the user's active story |
| `philosophy_thread` | `philosophy_threads_all.json` | Reopen an unresolved philosophical thread |

**Character selection (§7)** determines which character sends each message.

---

## 6. News topic drill-down

The system starts from the user's broad interests and narrows based on engagement.

### Data model — topic_drill_down (new column in proactive_state or new table)

```json
{
  "current_path": ["sport", "soccer", "Roma"],
  "engagement_history": {
    "sport":  { "sent": 4, "responded": 3 },
    "soccer": { "sent": 2, "responded": 2 },
    "Roma":   { "sent": 1, "responded": 1 }
  }
}
```

### Drill-down logic

```
Start: broad interest from user_profile.interests  (e.g. "sport")
    │
    ▼
User responds / engages with the content
    │
    ▼
Extract the most specific topic mentioned in the response
(LLM call: "What specific sub-topic did the user engage with? e.g. sport → soccer")
    │
    ▼
Save sub-topic as next level in path
    │
    ▼
Next news query uses the more specific topic ("soccer" instead of "sport")
    │
    ▼
Repeat: "soccer" → "Roma" → "Roma transfer news"
```

### Drill-up logic (if engagement drops)
If 2 consecutive messages at a specific level get no response → step back up one level.

```
"Roma" × 2 no response → fall back to "soccer"
"soccer" × 2 no response → fall back to "sport"
```

### Implementation note
The drill-down path is stored per-user in `proactive_state.topic_drill_down` (JSONB column).
The Tavily query uses `current_path[-1]` (the most specific active topic).

---

## 7. Character selection

Track which character the user interacts with most and weight proactive messages
toward that character.

### Source
`chat_history_short` already stores `character_id` on each assistant message.
Count per character over the last N sessions.

### Logic
```python
# count assistant messages per character in short-term history
counts = Counter(
    m["character_id"]
    for m in history
    if m.get("role") == "assistant" and m.get("character_id")
)
preferred_character = counts.most_common(1)[0][0] if counts else "socrates"
```

Store result in `proactive_state.character_id` (already exists).
Recalculate on each proactive run (cheap, no extra DB call needed).

---

## 8. Feedback loop — adaptive rules

### 8a. Time adaptation

| Signal | Action |
|--------|--------|
| User responds to proactive message | Record `last_responded_at`; if response time is consistently at a different hour, shift `send_hour_utc` toward that hour (±1h per session) |
| No response (after 2h window) | Increment `no_response_streak`; shift `send_hour_utc` +1 |
| `no_response_streak` reaches 24 | Full reset: try a completely different slot (±6h from current) |
| User responds after a miss | Reset `no_response_streak` to 0 |

**Response detection:**
A response is counted when a user message is saved to `chat_history_short`
within 2 hours of a proactive message's `sent_at` timestamp in `proactive_messages`.

### 8b. Topic adaptation — news drill-down
Covered in §6. In summary:
- Engagement → drill down (more specific topic)
- 2× no engagement → drill up (broader topic)
- Engagement score stored in `topic_drill_down.engagement_history`

### 8c. Topic type rotation
If a certain message type (`news`, `philosophy_thread`, etc.) consistently gets
no response over 3 consecutive sends, deprioritise it by reducing its weight
in the TYPE_PATTERN cycling.

---

## 9. How proactive/run changes

```python
def run_proactive_for_scheduled_users(current_hour_utc: int):
    """Called every hour. Only runs for users scheduled at this UTC hour."""
    resp = (
        supabase.table("proactive_schedule")
        .select("user_id")
        .eq("send_hour_utc", current_hour_utc)
        .execute()
    )
    for row in (resp.data or []):
        run_proactive_pipeline(row["user_id"], ...)
```

The `/proactive/run` endpoint receives the current UTC hour from the trigger
and passes it to `run_proactive_for_scheduled_users`.

---

## 10. Implementation order (suggested)

| Step | What | Effort |
|------|------|--------|
| 1 | Add `age` field to `user_profiles`, ask at onboarding | Small |
| 2 | Create `proactive_schedule` table in Supabase | Small |
| 3 | Populate schedule on signup using age + country heuristic | Small |
| 4 | Change cron to hourly, update `/proactive/run` to filter by hour | Small |
| 5 | Add `story` and `philosophy_thread` to topic pool | Medium |
| 6 | Add character weighting from chat history | Small |
| 7 | Add `topic_drill_down` JSONB to `proactive_state`, wire drill-down logic | Medium |
| 8 | Add response detection + time shift feedback loop | Medium |
| 9 | Add topic type deprioritisation on 3× miss | Small |

Steps 1–4 can ship independently and already improve the system significantly.
Steps 5–9 can follow incrementally.