initial_docs/docs/RL_AND_FAST_POLICY.md

# RL and Fast Policy

## Purpose

The fast mind must be good enough before RL.

The playable MVP should use a deterministic or heuristic fast mind first. A learned policy can later imitate or improve it.

Do not make the project depend on RL succeeding.

## Fast mind responsibilities

Fast mind owns:

- primitive action choice,
- movement,
- consume/inspect/rest execution,
- immediate resource pressure,
- immediate fear/danger reaction,
- local target scoring,
- action legality handoff.

Fast mind does not own:

- narrative thoughts,
- long-term interpretation,
- beliefs as text,
- object compilation,
- user speech interpretation beyond structured social channels.

## Primitive action space

Start with:

```text
move_north
move_south
move_east
move_west
wait
inspect
consume
rest
vocalize
```

Movement is one cell per action unless changed later.

## Higher-level directive resolution

Slow cognition may output high-level directives such as:

```json
{
  "mode": "seek",
  "target": {
    "type": "object",
    "id": "obj_17"
  },
  "strength": 0.35,
  "urgency": 0.25,
  "duration_ticks": 20
}
```

The fast mind cannot execute “seek” directly.

Use a **directive resolver** to convert high-level directives into fast-policy inputs.

The resolver should produce:

- target vector,
- path-distance gradient,
- target-reachable flag,
- urgency scalar,
- directive strength,
- action priors,
- trait-seek/avoid/inspect modifiers.

Example:

```json
{
  "directive_features": {
    "mode_seek": 1,
    "mode_avoid": 0,
    "target_dx": 2,
    "target_dy": -1,
    "path_distance": 5,
    "target_reachable": true,
    "strength": 0.35,
    "urgency": 0.25,
    "time_remaining": 18
  },
  "action_priors": {
    "move_east": 0.2,
    "move_north": 0.1
  }
}
```

The fast policy then chooses primitive actions.

A safety/resource arbiter validates the chosen action before applying it.

## Safety/resource arbiter

The arbiter can override or reject actions when:

- the move is illegal,
- the target is unreachable,
- fear is extreme,
- health is at risk,
- the action repeats too long,
- the action conflicts with critical resource survival,
- the user directive would lead into severe danger.

This creates conditional obedience:

```text
slow cognition proposes;
fast policy chooses;
arbiter validates;
world resolves.
```

## Heuristic fast policy

The heuristic policy should be clear and inspectable.

It should score possible actions from:

- resource pressure,
- local object appraisals,
- memory valence,
- user/social signals,
- slow priority modifiers,
- directive features,
- terrain/path legality.

Minimum behaviors:

- seek food when hungry,
- seek water if thirst is enabled,
- rest when tired,
- inspect unfamiliar safe-ish objects,
- avoid high threat,
- move toward user call only when trust/safety allow,
- abandon unreachable or repeatedly failed targets.

## Learned policy interface

A learned policy should use the same interface as the heuristic policy.

Observation should include:

- local grid channels,
- resources,
- user/social channels,
- memory/appraisal channels,
- directive features,
- priority modifiers,
- recent action loop/stuck indicators.

Output should be a primitive action.

The learned policy must not receive raw text.

## Should imitated users appear in training?

Yes.

If the user exists in the real simulation, the policy must experience user-like entities during training/evaluation.

Training should include simulated users that:

- stand still,
- move randomly,
- call the creature,
- stand near food,
- stand near danger,
- helpfully call from safe locations,
- misleadingly call from unsafe locations,
- place objects if supported by training environment.

This prevents distribution shift. The policy should not encounter social channels for the first time only in the real app.

## Training strategy

Preferred order:

1. Build heuristic policy.
2. Record heuristic traces.
3. Train small imitation policy.
4. Evaluate on randomized scenarios.
5. Optionally add RL fine-tuning.
6. Ship heuristic or hybrid if learned policy is weak.

## Training environments

Training environments should be rich enough to exercise the interface.

Include scenarios:

- food seeking,
- food vs heat conflict,
- shelter/rest,
- unknown object inspection,
- user safe call,
- user unsafe call,
- ambiguous food with delayed harm,
- repeated unreachable target,
- memory-influenced avoidance.

Do not train only on final demo scenes.

## Reward design

Do not reward “interestingness” directly.

Reward event patterns that produce watchable, coherent behavior:

- staying alive,
- reducing hunger/thirst when high,
- resting when depleted,
- inspecting unknown objects once,
- consuming useful objects,
- avoiding known harm,
- responding to safe user calls,
- ignoring unsafe user calls,
- following safe slow directives,
- ignoring unsafe slow directives,
- avoiding repeated loops.

Use diminishing returns for repeated actions.

Penalize:

- wall bump loops,
- oscillation,
- repeated failed target pursuit,
- consuming known harmful objects,
- ignoring critical resources,
- following directive into severe danger,
- doing nothing without rest/fear reason.

## Evaluation

Evaluate behavior on seeded maps.

Metrics should include:

- survival duration,
- hunger/thirst satisfaction,
- useful consumption count,
- inspection/discovery count,
- avoidance/fear conflict count,
- safe user-call response rate,
- unsafe user-call refusal rate,
- stuck-loop count,
- repeated action count,
- directive-following when safe,
- directive-overriding when unsafe.

The UI should expose policy mode:

```text
heuristic
learned
hybrid
```

If learned policy is weak, keep heuristic default.