README.md

---
license: other
library_name: pytorch
pipeline_tag: text-to-video
tags:
- text-to-video
- video-generation
- world-model
- diffusion-transformer
- flow-matching
- sovereign
- mother-exo
---

> ⚠️ **EARLY PREVIEW / PLACEHOLDER.** This is an early DiT checkpoint (step 13154) — outputs are **low quality**. It exists to test MOTHER Inference and hold this repo; the full-quality release follows shortly. Do not benchmark this build.

# MOTHER T2V — Sovereign Text-to-Video World Model

**MOTHER T2V** is a from-scratch, sovereign latent text-to-video generator: MOTHER's own 3D video
VAE and flow-matching Diffusion Transformer, conditioned on the frozen MOTHER CORE-7B text backbone.
No external generator weights (Sora / Wan / LTX / SVD) are used — the entire generative stack is
MOTHER-trained. It is the generative half of the MOTHER EXO world model: where the `worldmodel_latent`
head *predicts* future latents (dynamics), MOTHER T2V *synthesises* video from text.

> **Status of this card.** It describes the **release target** of MOTHER T2V. Architecture and training
> configuration are final and factual. Benchmark results are populated **only** from the official
> VBench package at release — no score is ever hand-entered or estimated. Cells marked *"pending"* are
> filled in when measured.

---

## Model Overview

| | |
|---|---|
| **Task** | Text → short video (latent diffusion, flow-matching) |
| **Total parameters** | ~0.85 B (≈118 M VAE + ≈734 M DiT) |
| **Resolution / length** | 256×256, 16 frames (temporally compressed to 4 latent frames) |
| **Conditioning** | Frozen MOTHER CORE-7B text embeddings (3072-d), classifier-free guidance |
| **Sovereignty** | 100% MOTHER-trained; external models are data-only teachers, never embedded |
| **Posture** | Observe-and-advise · human-in-the-loop · Strike = 0 |
| **Files** | `mother_t2v_vae256.pt` (VAE) · `mother_t2v_dit.pt` (DiT) |

---

## Architecture

### 1. 3D Video VAE — `VideoVAE` (≈118 M)
- Convolutional KL-regularised 3D VAE; **8× spatial / 4× temporal** compression to a **16-channel**
  spatiotemporal latent, so diffusion runs in a tractable space.
- GroupNorm + SiLU residual `Conv3d`; gradient-checkpointing friendly; resolution-agnostic.
- **Reconstruction training:** L1 + light edge/gradient term + small KL. An optional perceptual (VGG,
  LPIPS-style) + PatchGAN adversarial fine-tune ("VAE-GAN") sharpens high-frequency detail for the
  release VAE, resumed from the L1 VAE (the L1 VAE is preserved, never overwritten).

### 2. Latent Diffusion Transformer — `LatentDiT` (≈734 M)
- Diffusion transformer operating **in the frozen VAE latent space** — patchified `Conv3d`, 24 blocks,
  hidden 1024, 16 heads, patch 2.
- **Objective:** flow-matching velocity prediction (`t=0` data → `t=1` noise), MSE on the velocity.
- **Guidance:** classifier-free guidance via text-dropout during training.
- **Conditioning:** cross-attention to frozen MOTHER CORE-7B text hidden states (3072-d).
- **adaLN-zero** time conditioning; **axial-sinusoidal 3D positions** → resolution-agnostic (trainable
  at 256p, samplable higher).
- `latent_scale` is calibrated to the frozen VAE's measured latent variance before DiT training, so the
  DiT trains on ~unit-variance latents.

### 3. Text encoder — frozen MOTHER CORE-7B
- The sovereign CORE-7B (48 layers, 3072-d) provides text conditioning. Frozen; not fine-tuned by T2V.

---

## Training

- **Data:** real captioned video clips at 256 px, 16 frames (~13 k clips). Real data only for the
  generative fit; external models are teachers for captions, not embedded weights.
- **Stages:** (1) train the 3D VAE → freeze; (2) optional VAE-GAN sharpening → freeze; (3) train the
  latent DiT (flow-matching, CFG) on the frozen VAE latents.
- **Checkpointing:** save-best on the loss EMA, atomic writes, resume-safe.
- **Hardware:** single NVIDIA GB10 (Blackwell), bf16 where it helps (transformer), fp32 for the
  3D-conv VAE.

## Inference

```bash
python training/mother_t2v/latent/sample_latent.py \
  --prompt "a drone shot over a snowy mountain range at sunset" --steps 50 --guidance 5
# → montage PNG (+ mp4 if imageio present)
```
Sampling: CORE text → latent DiT (Euler ODE, `t: 1→0`, classifier-free guidance) → frozen VAE decode.

---

## Evaluation — VBench (official text-to-video benchmark)

Clips are generated per official VBench prompt with `training/leaderboards/vbench_eval.py`, then scored
by the official `vbench` package. **Numbers below are measured, never fabricated.**

| Dimension | Score |
|---|---|
| Subject consistency | pending |
| Background consistency | pending |
| Motion smoothness | pending |
| Aesthetic quality | pending |
| Imaging quality | pending |
| Object class | pending |
| Spatial relationship | pending |
| Temporal flickering | pending |
| **Overall** | **pending** |

*Scale context (reported, not hidden):* MOTHER T2V is a ~0.85 B generator trained from scratch on
~13 k clips — orders of magnitude smaller than the frontier systems that top VBench. Expect a
**credible, sovereign-eligible score, not a frontier one**. The value is full-stack ownership and a
transparently-measured entry, not beating Sora/Wan on raw fidelity.

---

## Intended Use & Scope

- **Intended:** text-conditioned short-video synthesis for world-modelling, scenario/ISR preview,
  simulation, and research within the MOTHER EXO stack.
- **Out of scope / prohibited:** any autonomous-action, targeting, fire-control, or kill-chain use.
  MOTHER T2V is observation and decision-support only, under meaningful human control (Strike = 0).

## Limitations

- ~0.85 B params, ~13 k training clips, 256 px, 16-frame clips → limited scene complexity, motion
  range, and fine detail vs. frontier T2V. Expect artifacts on out-of-distribution prompts.
- Text alignment is bounded by the frozen CORE-7B text representation (MOTHER-domain-specialised).
- Not a photorealistic or long-form video model; short clips at moderate resolution.

## Safety & Sovereignty

- **Sovereign:** only MOTHER's own trained VAE + DiT run; CORE-7B is MOTHER's frozen backbone. No
  Sora/Wan/LTX/SVD weights are embedded; external models are data-only teachers, codecs are renderers.
- **Posture:** observe-and-advise, human-in-the-loop, Strike = 0. Consistent with the MOTHER EXO
  non-weapon posture.

## Provenance

Built and evaluated on-node; benchmark numbers recorded only as measured by the official VBench
package. Part of the **MOTHER EXO** sovereign world model (`MediaStreamAI/MOTHER_EXO`).