initial_docs/docs/EVAL_AND_TESTING.md

# Evaluation and Testing

## Testing philosophy

The frontend is part of the product. The simulation is only successful if it can be manually played and browser-tested.

Tests should verify real state transitions, not only UI existence.

Every stage must include:

- unit tests for core logic,
- schema/validation tests where relevant,
- frontend tests,
- browser-level interaction tests,
- manual verification instructions,
- regression checks for previous demos.

## Manual evaluation target

A five-minute run should produce visible behavior without requiring user interaction.

Target events:

- consumption or attempted consumption,
- inspection/discovery,
- avoidance/fear/conflict,
- rest/shelter/comfort-seeking if resources demand it,
- memory or belief update,
- response to environment change,
- no long stuck loops.

## Frontend test requirements

Frontend tests should interact with the app like a user.

They should verify:

- grid renders,
- play/pause/reset works,
- tick count changes,
- resources change over time,
- objects can be placed,
- user can move/speak,
- creature moves/actions update state,
- event log receives real events,
- Mind Lens displays real state,
- notebook/export works,
- model provenance/compliance is visible.

Canvas/SVG visuals must be backed by inspectable state.

If a visual changes but state does not, the feature is not complete.

## Stage-level frontend interactions

Each stage should include at least one browser-level test or scripted manual test for the main user-facing behavior added in that stage.

Examples:

- Stage 1: start grid, tick, pause, reset.
- Stage 2: generated map has reachable objects and creature acts in simple preset world.
- Stage 3: user creates object from text, places it, sees schema/provenance.
- Stage 4: slow cognition updates Mind Lens and changes behavior modifiers.
- Stage 5: demo mode runs and exports summary.

## Model tests

LLM tests should verify:

- request uses approved model,
- request uses approved provider target,
- fallbacks are disabled,
- strict JSON schema is present,
- invalid output is rejected,
- valid output is accepted,
- frontend never sees API key,
- final-submission mode blocks prototype-only model.

## Map generator tests

Map tests should verify:

- creature spawn reachable,
- user spawn reachable,
- at least one useful resource reachable,
- at least one shelter reachable,
- at least one risky/ambiguous object exists,
- hazards do not block all paths,
- map seed is deterministic,
- generated maps vary across seeds.

## Memory/event tests

Memory tests should verify:

- events are emitted by real state transitions,
- recent events are derived from event log,
- object memory valence updates from outcomes,
- rolling memory text stays within limit,
- beliefs update from event evidence,
- slow cognition cannot invent direct state changes.

## Fast/slow interaction tests

Tests should verify:

- slow directive changes priority modifiers or target bias,
- fast policy may follow safe directive,
- fast policy may reject unsafe directive,
- user “come here” is social signal, not forced movement,
- high fear can override social approach,
- critical hunger can override weak curiosity.

## Acceptance standard

A stage is not done unless:

- app runs locally,
- new behavior can be manually verified,
- tests pass,
- frontend interaction is tested,
- debug/Mind Lens state is inspectable,
- docs are updated.