TurboT2VA-Ovi
Experimental four-step, text-conditioned audio/video distillation snapshots. This includes DCM and independent corrected sCM research snapshots, plus explicitly archival DMD-only and legacy rCM ablations. It is NOT a validated final rCM model, an upstream official release or a drop-in Diffusers pipeline.
Code: liuyuxiang1021/ovi-t2av-distillation.
HF target: luyu1021/turbot2va-ovi,
public with explicit user approval on 2026-11-19. The initial DCM500/sCM300
release passed remote verification; corrected sCM343 was subsequently uploaded
and verified as a later safe-stop snapshot without a same-step benchmark.
That initial three-student package was about140.17GB. The current five-student
package including experimental DMD/rCM is about233.67GB; all optimizers remain local.
GitHub remains private, so accessing the matching training/inference code
requires repository access; this is not a self-contained public inference bundle.
The expanded weight-upload commit is 44d049be9ce3686500cedff1a025fbc2940f4c62.
The DMD2000/rCM1663 extension was uploaded under the user's explicit request.
These are experimental archives, not recommended quality weights. The expanded
weight-upload commit 2a666163401f3637a84d354fc078fb95a60274b2 passed remote
size/SHA-256 verification of all84 new model/evaluation artifacts. Use the current
release manifest for exact
available files, byte counts, evaluation steps and per-snapshot provenance.
Check release_manifest.json for the exact source code commit, included files,
SHA-256 hashes, byte counts and optimizer availability. This card is also kept
in the code repository. code_commit identifies the unchanged training/release
implementation; documentation_commit, when present, identifies the later
documentation-only update. The manifest and checksums include those updated docs.
Snapshots
| HF Folder | Purpose | Full Same-Step Benchmark | Optimizer |
|---|---|---|---|
checkpoints/dcm-step500/ |
Anchored DCM result; initializes a new sCM stage | Eight Prompts, DCM500 | Not Included In Default Package |
checkpoints/scm-v2-step300/ |
Latest corrected sCM snapshot with evaluated outputs | Eight Prompts, sCM300 | Not Included In Default Package |
checkpoints/scm-v2-step343/ |
Later safe-stop snapshot; no same-step quality claim | None; nearest benchmark is300 | Not Included |
checkpoints/dmd-reference-step2000/ |
Archival reference-objective text-only DMD | Eight Prompts, DMD2000 | Not Included |
checkpoints/rcm-av3to2-step1663/ |
Archival legacy combined3:2 safe stop, NOT corrected-sCM continuation | None; included evaluation is1650 | Not Included |
Each student comprises eight PyTorch DCP .distcp shards and .metadata, about
46.67 GB decimal in total, plus original meta.json. Do not load a single shard
or rename DCP as safetensors. Five students total about233.36GB. An optional
sCM343 resume layer adds eight AdamW shards, about93.29 GB. Exact optimizer
continuation requires that layer AND matching inputs/runtime/settings; model-only
snapshots are not full resume packages. The historical two-model private package
was about 93.5 GB. The user approved public HF visibility to expand to three
model snapshots, about 140.17 GB, then separately requested archival DMD/rCM.
No critic states or optimizer files are included for either experiment; exact
continuation would additionally require those full states and validated runtime
compatibility. Complete source states remain local.
evaluations/dcm-step500/ and evaluations/scm-v2-step300/ contain eight MP4s,
generated student latents, and per-source latent/media metrics. No raw training
media, training latents/text embeddings, train CSV, held-out input embeddings,
teacher model, teacher videos, decoders or tokens are bundled.
The extension includes evaluations/dmd-reference-step2000/ and
evaluations/rcm-av3to2-step1650/, each eight videos, generated latents and metrics.
The latter is NOT an evaluation of1663. Parameter/reproduction/loading details
are in docs/DMD_RCM_EXPERIMENTS.md; historical rCM critical source is retained
under docs/provenance/rcm-av3to2/.
Recipe
The following recipe describes DCM/corrected sCM only. DMD uses text-only audited inputs, AdamW beta1=0, student/critic LR2.5e-6/5e-7 and critic-every-step reference cadence. Legacy rCM uses paired inputs, beta1=0.9, LR2e-6/2e-6, exclusive1:4, AV weights0.60/0.40, old sCM objective and DMD coefficient0.002. These objectives and schedules are not interchangeable. See the experiment parameter table.
Eight H20 GPUs; 20,000 audited disjoint paired training samples; seed1234; AdamW betas0.9/0.999, default weight decay0.01, grad clip1; BF16 ordinary forwards with FP32 student weights and gradient reductions; optimizer CPU offload. Video/audio loss weights0.85/0.15.
DCM forks the original unanchored step50 checkpoint, adding conditional teacher
anchor0.25; base LR2e-6, LR warmup100, cosine horizon2000/floor0.1; trig grid48,
skip1, teacher CFG4/3, loss scale100. Corrected sCM starts a NEW stage from DCM500:
fresh AdamW, LR2e-6, LR warmup500 AND both tangent terms warmup500, exact
FSDP-compatible unsharded JVP, p_g mean1.5, tangent p99 clip20, loss scale100.
Objective ID: rcm_tangent_warmup_v2_unsharded.
The old target incorrectly omitted warmup from the JVP term. Old high-LR sCM branches showed cross-prompt audio-latent collapse and are NOT included here. The code fix is established by regression tests; there is no controlled proof that the bug alone caused collapse because LR also varied.
Evaluation And Limitations
Eight fixed held-out prompts, seed103, matching initial noise, four-step student versus50-step recipe teacher. These are paired diagnostic means, not perceptual or large-scale benchmark scores:
| Snapshot | Video Latent Cosine | Audio Latent Cosine | RGB SSIM |
|---|---|---|---|
| DCM500 / sCM0 | 0.60283 | 0.45035 | 0.41159 |
| Corrected sCM300 | 0.62233 | 0.44530 | 0.36119 |
| Archival DMD2000 | 0.21951 | 0.29864 | 0.11028 |
| Archival rCM1650 (NOT1663) | 0.47028 | 0.31574 | 0.22518 |
At sCM300 cross-prompt audio cosine is0.671 and mean-centered residual/audio norm ratio0.521. The previously observed near-identical audio-latent collapse has not recurred up to this evaluation, but visual SSIM fell and subjective sound/video quality still needs inspection. Step343 has no same-step benchmark; do NOT label step300 metrics as step343 measurements. Neither sound fidelity nor AV sync is established by latent cosine. Later training stability is unknown.
The legacy combined rCM3:2 ablation was stopped at1663. Its latest1650 audio
alignment deteriorated despite increased RMS; it is not a recommended weight
release and is documented separately in docs/RCM_EXPERIMENTS_2026-11-19.md.
It is now included only as an explicitly requested archival ablation. DMD2000
also deteriorated relative to its pre-training baseline; neither extension
establishes improved subjective sound fidelity, visual quality or AV sync.
Loading And Provenance
Use the matching code repository, upstream Ovi model configuration and teacher
initialization. compare_ovi_inference.py loads DCP into an ordinary FP32 model
with CPU resharding; use explicit --preview-stage dcm or --preview-stage scm.
The loader then requires original Ovi decoders and your own input text/latents
to render outputs. See docs/TRAINING_GUIDE.md for commands and prerequisites.
For DMD student loading explicitly add --diagnostic-dmd; for combined rCM
omit that flag and --preview-stage. DMD's original metadata predates newer
strict trainer fields: exact optimizer continuation is unverified and cannot
be enabled by silently rewriting the published provenance.
Original meta.json includes the source machine's absolute paths and is preserved
unchanged. Strict loader/trainer checks require matching teacher/index/benchmark
identity. Controlled cross-machine path remapping, full-weight GPU recovery and
exact resume on a different host have NOT been validated. File completeness and
SHA-256 verification are not a full-weight resume test.
Ovi source and upstream model card identify Apache-2.0; attribution and license are retained. Decoder/encoder components keep their own upstream licenses. This release does not relicense input datasets or redistribute upstream decoder weights. Avoid presenting generated videos/audio as real recordings; subject to applicable rights and policies, use these experimental artifacts responsibly.