docs/oasis_integration.md

# Oasis Action World Model Integration

ADAM treats Oasis as a first-class trainer named `Oasis Action World Model`.
It uses the existing `Oasis-Game-Trainer/roblox_action_flow_app.py` worker and
keeps training and playable inference in separate processes.

Choose **temporal_latent** for new recordings. It learns a timed transition
from several recent frames and the inputs that happened between them. That lets
it distinguish, for example, a stationary car from a car still coasting after
the accelerator was released, or a player standing still from one falling with
no keys held. The player has a **Temporal: advance during no-input** switch:
turn it on to see learned idle dynamics, or turn it off to advance only when
you provide an input.

Choose **temporal_pixel_flow** when you want the same timed frame and input
history while predicting full RGB frames directly. It does not use a VAE, so it
avoids the latent codec's spatial compression and uses more GPU memory. It is a
separate checkpoint format; it cannot continue a Temporal Latent or Pixel Flow
checkpoint. Start with batch size 1 if full RGB training runs out of memory.

## Dataset Layout

Use one or more recorder folders. Multiple folders can be supplied with
semicolon separators.

```text
My_Oasis_Dataset/
  dataset_info.json
  actions.jsonl
  frames/
    frame_00000000.png
    frame_00000001.png
```

Each `actions.jsonl` row must identify the frame and its labels:

```json
{"session_id":"run1","frame_index":1,"filename":"frame_00000001.png","w":1,"a":0,"s":0,"d":0,"jump":0,"mouse_dx":0.0,"mouse_dy":0.0,"zoom":0.0}
```

Rows describe the action for the transition into that frame. ADAM validates
missing frames, invalid labels, duplicate or gapped frame indexes, broken
images, inconsistent resolutions, empty sequences, and missing action columns.
It does not guess missing labels.

For temporal models, use a current v5 recorder dataset. It contains timestamps,
input intervals, episode IDs, and discontinuity markers so ADAM never trains
across a reset, teleport, pause screen, or missing frame. Keep no-input frames:
they are the evidence needed to learn coasting, gravity, and other motion that
continues after an action ends.

## Minecraft Beta-Style Recording Notes

Record your own gameplay. Start with `256x144` at 8-12 FPS for a first pipeline
test, then increase data before long runs. Keep every clip as its own session or
recording folder so training never learns fake transitions between worlds.

Capture a mix of walking forward, strafing, turning, jumping, climbing, terrain
changes, interiors, caves, water, sky, and short idle moments. Avoid long
standing-still stretches; idle is useful, but a dataset dominated by idle frames
will make the model ignore controls.

A good smoke dataset is 300-1,000 labelled frames. A more meaningful first
Minecraft experiment is 5,000-20,000 labelled frames with clean W/A/S/D/Space
coverage and synchronized frame/action timestamps.

## RTX 3060 12 GB Starting Point

Use `temporal_latent`, `256x144`, batch size `2`, mixed precision `no`, history
frames `4`, future frames `3`, VAE warm-up `5`, gradient accumulation `1`,
loader workers `2`, save every `5` epochs, preview every `5` epochs, and 5-10
epochs for a pipeline test. Try `fp16` only after full precision is stable.