docs/runtime.md

# Runtime And Model Support

## Execution Model

Pipeline:

1. Generate visible reasoning trace.
2. Normalize and split trace into sentences.
3. Map sentence spans to token spans.
4. Run forward + backward pass.
5. Build sentence influence matrix from gradient x attention.
6. Summarize top edges and importance scores.

## Device And Dtype Policy

Default policy:

- CUDA:
  - `bfloat16` if supported
  - else `float16`
- MPS:
  - `float16`
- CPU:
  - `float32`

Override with:

- `DTYPE_PREFERENCE`
- request `dtype_preference`

## Model Requirements

Model must support all of:

- causal LM generation
- `output_attentions=True`
- eager attention
- supported decoder layer layout
- supported attention module attribute

Supported layer paths:

- `model.layers`
- `model.model.layers`
- `transformer.h`
- `gpt_neox.layers`

Supported attention attrs:

- `self_attn`
- `attn`
- `attention`

## Why Trace Limits Exist

Attribution path uses full backward pass over attention tensors. Cost grows with:

- sequence length
- layer count
- head count
- sentence count

Public defaults stay small to protect uptime.

## Good First Runtime Settings

For public demo:

- `max_new_tokens=128`
- `max_trace_tokens=256`
- `max_sentences=16`
- `validate_top_k=0`

For deeper analysis on bigger GPU:

- raise trace tokens slowly
- watch latency and memory first