initial_docs/docs/RL_AND_FAST_POLICY.md

RL and Fast Policy

Purpose

The fast mind must be good enough before RL.

The playable MVP should use a deterministic or heuristic fast mind first. A learned policy can later imitate or improve it.

Do not make the project depend on RL succeeding.

Fast mind responsibilities

Fast mind owns:

• primitive action choice,
• movement,
• consume/inspect/rest execution,
• immediate resource pressure,
• immediate fear/danger reaction,
• local target scoring,
• action legality handoff.

Fast mind does not own:

• narrative thoughts,
• long-term interpretation,
• beliefs as text,
• object compilation,
• user speech interpretation beyond structured social channels.

Primitive action space

Start with:

move_north
move_south
move_east
move_west
wait
inspect
consume
rest
vocalize

Movement is one cell per action unless changed later.

Higher-level directive resolution

Slow cognition may output high-level directives such as:

{
  "mode": "seek",
  "target": {
    "type": "object",
    "id": "obj_17"
  },
  "strength": 0.35,
  "urgency": 0.25,
  "duration_ticks": 20
}

The fast mind cannot execute “seek” directly.

Use a directive resolver to convert high-level directives into fast-policy inputs.

The resolver should produce:

• target vector,
• path-distance gradient,
• target-reachable flag,
• urgency scalar,
• directive strength,
• action priors,
• trait-seek/avoid/inspect modifiers.

Example:

{
  "directive_features": {
    "mode_seek": 1,
    "mode_avoid": 0,
    "target_dx": 2,
    "target_dy": -1,
    "path_distance": 5,
    "target_reachable": true,
    "strength": 0.35,
    "urgency": 0.25,
    "time_remaining": 18
  },
  "action_priors": {
    "move_east": 0.2,
    "move_north": 0.1
  }
}

The fast policy then chooses primitive actions.

A safety/resource arbiter validates the chosen action before applying it.

Safety/resource arbiter

The arbiter can override or reject actions when:

• the move is illegal,
• the target is unreachable,
• fear is extreme,
• health is at risk,
• the action repeats too long,
• the action conflicts with critical resource survival,
• the user directive would lead into severe danger.

This creates conditional obedience:

slow cognition proposes;
fast policy chooses;
arbiter validates;
world resolves.

Heuristic fast policy

The heuristic policy should be clear and inspectable.

It should score possible actions from:

• resource pressure,
• local object appraisals,
• memory valence,
• user/social signals,
• slow priority modifiers,
• directive features,
• terrain/path legality.

Minimum behaviors:

• seek food when hungry,
• seek water if thirst is enabled,
• rest when tired,
• inspect unfamiliar safe-ish objects,
• avoid high threat,
• move toward user call only when trust/safety allow,
• abandon unreachable or repeatedly failed targets.

Learned policy interface

A learned policy should use the same interface as the heuristic policy.

Observation should include:

• local grid channels,
• resources,
• user/social channels,
• memory/appraisal channels,
• directive features,
• priority modifiers,
• recent action loop/stuck indicators.

Output should be a primitive action.

The learned policy must not receive raw text.

Should imitated users appear in training?

Yes.

If the user exists in the real simulation, the policy must experience user-like entities during training/evaluation.

Training should include simulated users that:

• stand still,
• move randomly,
• call the creature,
• stand near food,
• stand near danger,
• helpfully call from safe locations,
• misleadingly call from unsafe locations,
• place objects if supported by training environment.

This prevents distribution shift. The policy should not encounter social channels for the first time only in the real app.

Training strategy

Preferred order:

1. Build heuristic policy.
2. Record heuristic traces.
3. Train small imitation policy.
4. Evaluate on randomized scenarios.
5. Optionally add RL fine-tuning.
6. Ship heuristic or hybrid if learned policy is weak.

Training environments

Training environments should be rich enough to exercise the interface.

Include scenarios:

• food seeking,
• food vs heat conflict,
• shelter/rest,
• unknown object inspection,
• user safe call,
• user unsafe call,
• ambiguous food with delayed harm,
• repeated unreachable target,
• memory-influenced avoidance.

Do not train only on final demo scenes.

Reward design

Do not reward “interestingness” directly.

Reward event patterns that produce watchable, coherent behavior:

• staying alive,
• reducing hunger/thirst when high,
• resting when depleted,
• inspecting unknown objects once,
• consuming useful objects,
• avoiding known harm,
• responding to safe user calls,
• ignoring unsafe user calls,
• following safe slow directives,
• ignoring unsafe slow directives,
• avoiding repeated loops.

Use diminishing returns for repeated actions.

Penalize:

• wall bump loops,
• oscillation,
• repeated failed target pursuit,
• consuming known harmful objects,
• ignoring critical resources,
• following directive into severe danger,
• doing nothing without rest/fear reason.

Evaluation

Evaluate behavior on seeded maps.

Metrics should include:

• survival duration,
• hunger/thirst satisfaction,
• useful consumption count,
• inspection/discovery count,
• avoidance/fear conflict count,
• safe user-call response rate,
• unsafe user-call refusal rate,
• stuck-loop count,
• repeated action count,
• directive-following when safe,
• directive-overriding when unsafe.

The UI should expose policy mode:

heuristic
learned
hybrid

If learned policy is weak, keep heuristic default.