docs/method.md

# Method

[← Meridian](../README.md) · [Inference](inference.md) · [Studio](studio.md)

Meridian synthesizes a new observation of an existing event. It separates **which source moment is
shown** from **which camera observes it**, then uses geometry to make that choice visible to a video
model. The geometry supplies a spatial constraint; the model supplies the appearance of the completed
shot, including regions the source camera did not see.

This is geometry-guided video re-camera, not a persistent 4D reconstruction or an action-conditioned
simulator. A new view is a generated interpretation of the recorded event, not evidence of what an
unobserved camera would actually have captured.

[![Meridian method overview with actual source, geometric-reference and generated flower frames](assets/research/meridian_method.png)](assets/research/meridian_method.svg)

**Overview.** Source time selects both appearance and geometry; the authored camera makes a
projected reference. Both references condition the video model. Real example: *Spring* (2019),
© Blender Foundation, CC BY 4.0; input retimed, view projected and generated.
[Full figure, attribution and provenance](assets/research/README.md).

## 1. Choose a source timeline

For each output frame `t`, a source-frame map `s(t)` selects the image and geometry to use:

| Timeline | Source-frame selection |
|---|---|
| Preserve the input's pace | Advance one source frame per output frame. |
| Hold a moment | Repeat one source frame while the target camera can keep moving. |
| Slow motion or accelerated action | Retime the input to a constant 24 fps **before** reconstruction, then advance through that export normally. |

Both video references follow the same selected timeline. Meridian is not asked to invent a different
action speed from an unchanged reference. See [Timing](inference.md#timing) for frame-index semantics,
freeze windows, and FFmpeg recipes.

The CLI constructs `s(t)` from `--start`, `--frames`, and optionally `--freeze`. The studio constructs it
from keyframes: source indices interpolate linearly and are rounded to integers. Studio source keys
must be non-decreasing; easing affects the camera path, not the source-frame mapping.

## 2. Reconstruct the source span

The input is resized and letterboxed into a 1280 × 1280 square, then downsampled to 512 × 512 for
VGGT-Omega. **One model call processes the selected source span jointly**, returning per-frame depth,
confidence, camera extrinsics, and intrinsics. Per-frame outputs do not mean independent single-frame
inference. Changing the reconstruction span can change estimates for frames shared by both spans.

Before unprojection, the implementation removes:

- Non-finite depth or confidence, and confidence values at or below `1e-5`.
- Depth discontinuities whose 3 × 3 local range exceeds 30% of the depth magnitude.
- The lowest-confidence 2% of the remaining candidates in each frame.

Depth and validity are upsampled to the letterboxed input resolution. A pixel is retained only when
the interpolated validity exceeds `0.999`, limiting points introduced across rejected boundaries.
Source RGB supplies the point colors. There is no fused mesh, persistent scene optimization, or
cross-frame point-cloud accumulation in this stage.

### Coordinates and scale

Geometry has a reconstruction-relative scale, not calibrated meters. Camera translations use `zm`,
a median scene depth. Choosing a distant background as the depth reference makes the same numerical
move much larger than choosing the subject.

- **CLI:** a camera offset is applied in each selected source camera's local coordinates:
  `C_target(t) = C_source(s(t)) @ delta(t)`. By default, `zm` comes from valid depths in the first
  selected frame; for `--freeze`, it comes from the held frame. `--pivot fx,fy` restricts the depth
  measurement to a neighborhood of a pixel in the **cropped image**. `--pivot-lock` also places the
  orbit center at the corresponding 3D point.
- **Studio:** all keys share the coordinate frame of the source camera at `start`: **x right,
  y down, z forward**. Positions and look-at points are expressed in units of `zm`. The API measures
  `zm` around a chosen pixel at `pivot_frame`, falling back to valid picture depths when too few
  local points remain. The current page uses the picture center at `start` as this scale reference;
  a key's **aims at** control changes its look-at point, not the scale reference.

The CLI's source-relative trajectory and the studio's shared-frame trajectory are different ways of
authoring a camera. Similar-looking controls need not produce identical paths on a moving-camera clip.

## 3. Render a geometric reference

At each output time, the selected source frame's colored point cloud is projected through the target
camera and its lens. A z-buffer resolves visibility; each point splats onto a 3 × 3 pixel neighborhood.
Uncovered pixels are filled with RGB `(128, 128, 128)`.

The current implementation rasterizes at the **output canvas**, then downsamples the result to the
**480-class conditioning canvas**. There is no geometric inpainting before generation. Coverage is
computed for diagnostics, but **no coverage mask is fed to the transformer**.

The studio's *what the model sees* preview and the saved `render.mp4` show this downsampled reference.
They are encoded video previews, not lossless copies of the in-memory conditioning pixels. The
magenta-hole view is a diagnostic visualization only; the model receives the grey-hole version.

## 4. Condition the video transformer

MiniMax-H3's VAE encodes two references:

1. **`<Video 1>` — source:** the selected source images, at the 480 class.
2. **`<Video 2>` — geometry:** the rendered target view, at the same conditioning class.

The target is generated at the 768 class. Here “class” means an aspect-ratio bucket, not a fixed
width or height. For a 16:9 input, the reference canvas is 832 × 480 and the output is 1344 × 768;
square inputs use 640 × 640 and 1024 × 1024 respectively. The nearest bucket is chosen by log aspect
ratio, with a centered crop inside the letterbox.

```text
source span ──► joint VGGT-Omega reconstruction ──► per-frame geometry
      │                                                   │
      │                     source-frame map + camera path│
      │                                                   ▼
      │                                       z-buffered point splat
      │                                                   │
      ▼                                                   ▼
source reference, 480 class                    view reference, 480 class
      └────────────────────────┬──────────────────────────┘
                               ▼
              VAE → packed reference tokens + fixed text
                               ▼
        stock H3 + re-camera and turbo adapters → VAE decode
                               ▼
                     new shot, 768 class, 24 fps
```

**The references are concatenated as tokens, not added as channels.** `recam/h3.py` uses Diffusers'
`MiniMaxH3Ref2VAPrepareLayoutStep.build_ref2va_packed_sequence` to create the reference layout,
position IDs, and modality tags. The transformer architecture is unchanged.

Reference video rows receive the upstream conditioning-noise convention
`0.999 × latent + 0.001 × noise` and stay fixed during denoising. Target video rows begin as random
noise. The source is also VAE-encoded at target resolution to establish the target latent shape;
its values are **not** used to initialize the target rows.

### Fixed text and the audio branch

[`assets/prompt.txt`](../assets/prompt.txt) describes the two-reference editing task: retain the source
event and complete the geometry reference's grey holes. Its embeddings are precomputed for each
supported output length, so inference does not load Qwen3-VL. Editing the text file alone does not
change inference; the shipped embeddings are what the model reads. An embedding-generation script
is not included in this release.

The packed layout retains H3's audio branch. Cached silence latents supply its shape and length;
the current code initializes audio rows with noise, denoises them, and discards the result. Meridian
does not generate or preserve a soundtrack through that branch. The CLI's optional audio file is
instead made by muxing the source soundtrack after video generation.

## 5. Sample the new shot

| Mode | CLI settings | Transformer evaluations |
|---|---|---|
| Fast adapter, default | `--steps 4 --flow-shift 3` with both adapters loaded | 3 |
| Teacher | `--lora teacher_lora --steps 50 --flow-shift 12` | 49 |

The H3 scheduler counts the terminal zero-noise point in `--steps`; that endpoint does not require
another model evaluation. The turbo adapter must be loaded alongside the re-camera teacher, which is
what it was distilled against; neither works on the unmodified transformer alone, and neither should
be merged into it. Forward counts do not equal end-to-end speedups: geometry,
VAE work, and file writing still take time.

## Training overview

Training provenance, augmentations, and distillation design have moved to [Training and distillation](training.md).

## Implementation map

| Source | What to read |
|---|---|
| [`recam/geometry.py`](../recam/geometry.py) | `reconstruct`, `warp`, and `render_hw`: geometry filtering, projection, and visibility. |
| [`recam/path.py`](../recam/path.py) | `plan_path` and `hermite`: keyframe interpolation, time mapping, and zero-roll look-at cameras. |
| [`recam/h3.py`](../recam/h3.py) | `bucket`, `pack`, and `denoise`: canvases, reference conditioning, and the scheduler. |
| [`inference/sample.py`](../inference/sample.py) | CLI time windows, parametric camera moves, diagnostics, and output files. |
| [`service/app.py`](../service/app.py) | `prepare`, `geo`, and `do_render`: cached reconstruction and resident inference. |

For weight provenance and modification notices, see [`MODIFICATIONS.md`](../MODIFICATIONS.md).