Delete docs

docs/first-step-comparison.md

@@ -1,76 +0,0 @@
-> Historical validation record: this comparison predates removal of the disabled auxiliary loss branches. Its source identifiers and field counts describe that earlier snapshot. Use scripts/check_readability.py and scripts/check_loss_parity.py for the cleaned release.
-
-# Readability refactor: first optimizer-step comparison
-
-The refactored 1.3B three-encoder code completed one full optimizer update on four H200 GPUs. Initial state, sampled inputs, forward features, loss and feature gradients match the earlier public-source run bitwise. The first recorded numerical difference is in `encoder_and_vae_backward`. Compare against the same-source repeated-run control below.
-
-## Source and run identity
-
-- Refactor base: `7985874ff49de326b8350a66516c50645a604f45`; the GPU comparison used a source snapshot captured before committing the refactor.
-- Tested source snapshot manifest SHA-256: `a806daa28aa2aa75c5adbe1e6bb930dbae822aee07cbfaf830980f6f8111b001`.
-- Previous public trace commit: `985ebec542b1c2bdb7af95a4a7404b3069cc15e7`. Its code tree is byte-identical to the pre-refactor 1.3B code tree: `60b92b4230405641d8d65a048d7b7cf842a653cf`.
-- Original-source control commit: `e8ec87e10c4825dfaaec993643724c5e8a2e28e9`.
-- Base seed **1862643**, with rank seeds **1862643–1862646**, matching the recovered historical effective seed. A legacy setting of `seed: 0` requests seed randomization and is not the effective seed.
-- Distribution / differentiation / reference batches: **256 / 128 / 128**. Four ranks, 64 forward samples and 32 backward samples per rank, microbatch size 1.
-- Encoder weights **10:1:1**, loss divisor **3**, fixed-reference / Monte Carlo reference weights **2:1**.
-- Same physical model/reference assets and four physical GPUs as the earlier runs. Independently queried software versions match.
-- All **173** readable input configuration fields resolve to the previous values in the same order. The new config loader and `--training-updates 1` alias were exercised. The configured cosine schedule remains steps 0–125, initial learning-rate setting 2.5e-6, floor 5e-7.
-- Observation hooks use the renamed methods and otherwise preserve the earlier observer. No production computation was changed for this run.
-
-## Scalar observations
-
-| Quantity | Previous public source | Readable source | Original A | Original B |
-|---|---:|---:|---:|---:|
-| Loss | 3.994718074798584 | 3.994718074798584 | 3.994718074798584 | 3.994718074798584 |
-| Gradient norm before clipping | 48.80884552001953 | 48.82303619384766 | 48.82280349731445 | 48.81880187988281 |
-
-## Full-vector differences
-
-| Quantity | Public → readable | Original A → readable | Original A → original B |
-|---|---:|---:|---:|
-| Latent gradient — relative L2 | 0.507891% | 0.508414% | 0.508167% |
-| Latent gradient — cosine similarity | 0.999987102368 | 0.99998707571 | 0.999987088324 |
-| Latent gradient — maximum absolute difference | 4.95910644531e-05 | 6.103515625e-05 | 6.103515625e-05 |
-| Generator gradient before clipping — relative L2 | 0.299963% | 0.301090% | 0.312999% |
-| Generator gradient before clipping — cosine similarity | 0.999995544683 | 0.999995467272 | 0.999995104525 |
-| Generator gradient before clipping — maximum absolute difference | 0.0146789550781 | 0.00958251953125 | 0.0114288330078 |
-| Generator gradient after clipping — relative L2 | 0.298507% | 0.301089% | 0.312905% |
-| Generator gradient after clipping — cosine similarity | 0.999995544684 | 0.999995467273 | 0.999995104526 |
-| Generator gradient after clipping — maximum absolute difference | 0.00274252891541 | 0.0019583106041 | 0.00238353013992 |
-| Actual Adam parameter update — relative L2 | 7.580502% | 7.667867% | 7.707839% |
-| Actual Adam parameter update — cosine similarity | 0.997126798637 | 0.997060191614 | 0.99702946052 |
-| Actual Adam parameter update — maximum absolute difference | 5.00679016113e-06 | 5.00679016113e-06 | 5.00679016113e-06 |
-
-Relative L2 means `||right - left||₂ / ||left||₂`, accumulated across all four ranks. Adam update differences compare `weights_after - initial_weights`; they are not normalized by the much larger model weight norm. Initial parameters are byte-identical across all variants.
-
-## Exact comparison with the previous public source
-
-| Recorded stage | Bitwise equality |
-|---|---|
-| effective_configuration | Exact |
-| initial_weights | Exact |
-| initial_rng | Exact |
-| forward_inputs_and_features | Exact |
-| mmd_result | Exact |
-| encoder_and_vae_backward | Different (645/2304 tensors) |
-| gradients_before_clip | Different (918/918 tensors) |
-| gradients_after_clip | Different (918/918 tensors) |
-| optimizer_state | Different (1836/2754 tensors) |
-| updated_weights | Different (916/3300 tensors) |
-| post_step_rng | Exact |
-
-All **424 tensor archives** (106 per rank) were compared, including all 256 forward samples and 128 selected backward samples. Equality checks compare raw bytes, including signed zeros. Configuration comparison excludes only output-directory and input-file-location fields. Timings and the final process-completion event are excluded from numerical comparison. All non-timing tensor values and structural fields remain compared.
-
-## Interpretation and scope
-
-The first recorded numerical difference is in `encoder_and_vae_backward`. The earlier same-source repeat has nonidentical encoder/VAE backward, generator gradients and Adam updates. The control demonstrates backward execution variability in this GPU configuration; it does not identify a specific nondeterministic kernel or establish a statistical tolerance from one control pair.
-
-The public-to-readable gradient relative L2 difference is 0.9583 times the measured same-source control difference; for the Adam update it is 0.9835 times. These ratios contextualize the observed discrepancies; they do not define a pass/fail tolerance or guarantee matching long-run trajectories.
-
-The readable-source run completed backward, clipping, Adam, native checkpoint saving, and all four ranks successfully (exit code 0). The previous public run also completed successfully. Original A completed the first update and every compared trace, then failed at checkpoint saving because its audit output directory was absent; original B completed normally.
-
-This comparison concerns reruns under the historical settings, not tensors captured during the original historical first step. It covers 1.3B only and does not evaluate 14B training, all training updates, generated-video quality, or VBench.
-
-## Software environment
-
-`{"cuda": "12.1", "cudnn": 90100, "flash_attn": "2.7.4.post1", "numpy": "1.26.4", "python": "3.10.12", "torch": "2.5.1+cu121", "torchvision": "0.20.1+cu121", "transformers": "4.53.3"}`

docs/paper-to-code.md

@@ -1,158 +0,0 @@
-# Reading the Anonymous Video Generation implementation
-
-Start with the training update, then follow the loss and replay calls. The paper's
-Sections 3.1–3.3 describe the objective and the separation between estimating a
-distribution and differentiating selected rollouts. Appendix C.4 and Algorithm 1
-describe the replay procedure. The notes below distinguish that mathematical
-presentation from the numerical choices retained in this release.
-
-Both releases contain one active distribution objective: the three-encoder MMD described below. The historical code/video_fd directory name is retained because its encoder adapters and replay VJPs are part of that active pipeline; it does not select an additional FD loss.
-
-## Reading order
-
-The two model directories have the same main entry points and both use DINOv3,
-V-JEPA 2 and VideoMAEv2. Start with 1.3B when reading the shared flow; the 14B
-release adds its evaluated step-80 schedule and memory-management details.
-
-| Read | Purpose | 1.3B | 14B |
-|---|---|---|---|
-| Configuration | Batch sizes, encoders, loss weights, initialization and optimizer | [three_encoder.yaml](../1.3B/configs/three_encoder.yaml) | [train.yaml](../14B/configs/train.yaml) |
-| Training entry | Load configuration and start distributed training | [train.py](../1.3B/code/train.py) | [train.py](../14B/code/train.py) |
-| `DistributionMatchingTrainer` | Assemble one complete update: collect rollouts, compute feature gradients, replay and update the generator | [trainer/distribution_matching.py](../1.3B/code/trainer/distribution_matching.py) | [trainer/distribution_matching.py](../14B/code/trainer/distribution_matching.py) |
-| `FeatureGradientReplay` | Gather the distribution batch and compute gradients with respect to selected features | [trainer/feature_gradient_replay.py](../1.3B/code/trainer/feature_gradient_replay.py) | [trainer/feature_gradient_replay.py](../14B/code/trainer/feature_gradient_replay.py) |
-| `MultiEncoderMMD` | Combine the independently scaled encoder losses | [losses/multi_encoder_mmd.py](../1.3B/code/losses/multi_encoder_mmd.py) | [losses/multi_encoder_mmd.py](../14B/code/losses/multi_encoder_mmd.py) |
-| `VideoNystromMMD` | Evaluate generated–generated repulsion and hybrid reference attraction | [losses/video_nystrom_mmd.py](../1.3B/code/losses/video_nystrom_mmd.py) | [losses/video_nystrom_mmd.py](../14B/code/losses/video_nystrom_mmd.py) |
-| `DistributionMatchingModel` | Connect rollout generation, frozen encoders, VAE decoding and generator replay | [model/distribution_matching.py](../1.3B/code/model/distribution_matching.py) | [model/distribution_matching.py](../14B/code/model/distribution_matching.py) |
-
-The previous module paths remain compatibility shims. Read the implementations
-linked above rather than the shims. Low-level network definitions in `wan/` are
-useful after the training flow is clear.
-
-In `DistributionMatchingTrainer`, start at `_run_distribution_matching_step` and
-follow `_collect_rollout_microbatch`. In `DistributionMatchingModel`, the corresponding
-stages are `collect_rollout_features`, `prepare_latent_gradients` and
-`replay_generator_gradients`.
-
-## Three different batch sizes
-
-| Paper symbol | Meaning | Released settings |
-|---|---|---|
-| $B$ | **Distribution batch:** generated videos that participate in the loss | 256 globally; 64 per GPU on 1.3B, 32 per GPU on 14B |
-| $b$ | **Differentiation batch:** generated videos replayed for generator backpropagation | 128 globally; 32 per GPU on 1.3B, 16 per GPU on 14B |
-| $B_{\mathrm{ref}}$ | **Reference minibatch:** stored reference-video features used by the Monte Carlo term | 128 globally |
-
-The readable configuration calls these `distribution_batch_size`,
-`distribution_samples_per_rank`, `differentiation_samples_per_rank` and
-`reference_minibatch_size`. `differentiation_gradient_scale` is $B/b$;
-`encoder_loss_divisor` is the final loss divisor. `utils/config.py` translates
-these names into the compatible runtime fields without changing their values.
-
-A microbatch is the smaller group processed at once; it is not $B$ or $b$.
-All 256 generated videos affect the loss, including the 128 whose generator
-computation is not replayed. Computing independent MMD losses on microbatches
-would omit their cross-microbatch interactions.
-
-Compatibility names can be misleading: `student_reference` means the full
-**generated** population, while `current_teacher` means the **reference
-minibatch**. Reference features are read from a frozen feature bank; this path
-does not run an online diffusion teacher.
-
-## One optimizer update
-
-```text
-Prompts + noise
-    → generate B videos without retained generator graphs
-    → extract frozen-encoder features and retain replay state
-    → gather the complete distribution batch across GPUs
-    → combine persistent Nyström statistics with a reference minibatch
-    → compute feature gradients for the b selected videos
-    → replay encoders: feature gradients → pixel gradients
-    → replay VAE: pixel gradients → latent gradients
-    → replay generator: latent gradients → parameter gradients
-    → accumulate contributions, clip gradients, update once
-```
-
-A **cotangent** is the incoming gradient used in a vector–Jacobian product. For
-example, a feature cotangent is $g_i^e=\partial L/\partial z_i^e$; replay passes it
-back through the corresponding encoder and generator. Frozen encoder and VAE
-parameters are not optimized, but derivatives with respect to their inputs are
-still required. Replay restores the original noise, random choices and segment
-inputs so that the gradient is paired with the intended computation.
-
-Two scaling factors have different purposes. The configured $B/b=2$ compensates
-for replaying only part of the distribution batch. Multiplication by the number
-of GPUs compensates for FSDP averaging parameter gradients across ranks.
-
-## Connecting the loss to the paper
-
-Section 3.1 compares generated and reference distributions in frozen encoder
-spaces. Section 3.2 combines two estimates of the reference kernel mean:
-
-- **Persistent Nyström mean:** landmark kernels multiplied by precomputed
-  coefficients, as in Equation (8) and Appendix C.2, Equations (18)–(21).
-- **Monte Carlo mean:** average kernel similarity to a sampled reference
-  minibatch, as in Equation (6).
-
-The released Nyström:Monte Carlo weights are 2:1, so the mixture in Equation (9)
-has $\alpha_e=1/3$. The tensor saved under the legacy name `alpha` is instead
-the **Nyström coefficient vector**, denoted $\boldsymbol\beta_e$ in Equation
-(19). These are different quantities.
-
-For each encoder, the released implementation evaluates
-
-$$
-M_e=\frac{1}{B^2}\sum_{i,j}k_e(z_i,z_j)
--\frac{2}{B}\sum_i\left[\frac{2}{3}m^e_{\mathrm{Nys}}(z_i)
-+\frac{1}{3}\widehat m^e_{\mathrm{MC}}(z_i)\right]+C_e,
-$$
-
-where $C_e$ is the retained reference–reference contribution. It then applies
-detached per-encoder loss scaling:
-
-$$
-L=\frac{1}{d}\sum_e w_e
-\frac{M_e}{\operatorname{stopgrad}(M_e)+\epsilon}.
-$$
-
-Here $d$ is the explicit aggregation divisor; it is not inferred from the sum
-of the weights. Both 1.3B and the human/VLM-evaluated 14B use
-DINOv3:V-JEPA 2:VideoMAEv2 = 10:1:1. Both configurations use $d=3$ and
-$\epsilon=10^{-4}$.
-
-## Details to preserve when reviewing or refactoring
-
-- **Self-pairs and constants.** Paper Equations (11)/(13) use the off-diagonal
-  denominator $B(B-1)$ and omit reference–reference constants. This release
-  retains self-pairs with denominator $B^2$ and retains the constants. Although
-  a reference constant has no direct generator derivative, it affects the
-  detached normalization denominator above. Changing either convention changes
-  the released numerical objective.
-- **Replay selection.** Algorithm 1 uniformly samples a subset. The released
-  implementation selects a fixed suffix of each rank's generated batch and
-  creates feature-gradient leaves for those selected samples. Their gradients
-  still include interactions with the full distribution batch. The conditional
-  unbiasedness statement in Equation (27) assumes uniform subset sampling;
-  it should not be claimed for this fixed-suffix implementation.
-- **VAE temporal derivatives.** The active model paths call `replay_decoder_vjp`,
-  which carries temporal-cache values but detaches their gradients between latent
-  frames. This preserves decoded values while truncating cross-frame cache
-  derivatives. The 14B tree also provides `replay_decoder_vjp_exact` and
-  `wan_recurrent_decoder_vjp.py`, which propagate future-state cotangents; their
-  presence does not mean the released training path selects them. Appendix C.4
-  distinguishes prescribed gradient truncation from exact replay.
-- **Compatibility.** Resource-pack keys, checkpoint parameter names and legacy
-  configuration fields remain supported. Their historical wording is not an
-  additional training objective or a requirement to instantiate score models.
-
-## Instrumentation and compatibility names
-
-Old imports and direct calls remain available as aliases. New training code
-calls the readable method names, so observers that intercept methods should
-patch `_collect_rollout_microbatch`, `_run_distribution_matching_step`,
-`collect_rollout_features`, `prepare_latent_gradients`, and
-`replay_generator_gradients`. Patching an old method alias does not intercept
-calls dispatched through the new name.
-
-Edit the public YAML when changing experiment settings. `prepare_config.py`
-resolves those readable fields and writes the legacy runtime schema to
-`configs/local.yaml`; this preserves resource and checkpoint identities.