George0667/MiniMax_Temporal_Upscaler

Minimax H3 Latent Upscaler v2

A latent-space upscaler for MiniMax-H3 video generation. It works directly on H3's 24-channel VAE latents, so the VAE's decode → pixel upscale → encode round trip is not needed.

• Spatial 2–4× (H×W). It is a drop-in replacement for v1 in the ComfyUI node.
• Frame rate 8 / 12 / 16 → 24 fps, alone or together with a 2–4× spatial upscale. This needs the _temporal files and the code in code/. The ComfyUI node does not support it.

The architecture and checkpoint layout are those of Minimax H3 Latent Upscaler v1: 3D-conv LatentResizer3D, 512 channels, 12 + 12 blocks, a temporal conv every 2 blocks, 345M parameters. v2 was trained from scratch. The frame-rate variant adds one small embedding of the frame-rate factor.

Files

Folder	Use	Precision	Size
minimax_h3_latent_upscaler_3d_conv_v2/	Spatial 2–4×; loads in the ComfyUI node	fp16, bf16 safetensors	659 MB each
minimax_h3_latent_upscaler_3d_conv_v2_temporal/	Spatial 2–4× and 8/12/16 → 24 fps; needs code/model.py	fp16, bf16 safetensors	659 MB each
code/	model.py, stats.py (PyTorch only)

At equal input and output frame rates, the two variants compute the same function. The spatial files are the temporal checkpoint with its frame-rate embedding at factor 1 folded into the scale embedding's bias.

Usage

ComfyUI (spatial): copy minimax_h3_latent_upscaler_3d_conv_v2/minimax_h3_latent_upscaler_3d_conv_v2_fp16.safetensors (or _bf16) into ComfyUI/models/latent_upscale_models/. Then pick it in the Minimax H3 Latent Upscaler 3D node, as you would v1.

Python (spatial and frame rate):

import torch
from model import MiniMaxH3LatentUpscaler  # code/model.py, with code/stats.py next to it

m = MiniMaxH3LatentUpscaler.from_file(
    "minimax_h3_latent_upscaler_3d_conv_v2_temporal/minimax_h3_latent_upscaler_3d_conv_v2_temporal_bf16.safetensors",
    device="cuda", dtype=torch.bfloat16)

# z: [B, 24, T, h, w], latents as the H3 VAE encoder / pipeline returns them (normalized)
hi = m.upscale(z, 2.0, (T, 2 * h, 2 * w))                                      # spatial 2x
hi = m.upscale(z8, 2.0, (17, 2 * h, 2 * w), chunk=None, fps=(8, 24))           # 22 frames @ 8 fps -> 56 @ 24 fps, 2x
hi = m.upscale(z8, 1.0, (17, h, w), chunk=None, fps=(8, 24))                   # frame rate only

• upscale() applies the extra mean/std normalization the network was trained in. Do not call forward() on VAE latents directly: that gives a magenta dot lattice.
• For a frame-rate upscale, both clips start at the same instant. The 24 fps target covers the low clip's span: 17n + 5 frames, i.e. 5n + 2 latents.
• Frame-rate upscaling runs the whole clip at once (chunk=None).

Evaluation

All numbers are on 6 held-out clips (8 aspect ratios, high-res canvas about 1 MP, 39 frames at 24 fps). The clips' source videos were not used in training. Metrics: PSNR (dB, higher is better), LPIPS and tLP (temporal LPIPS) against the original frames, after decoding with the H3 VAE (lower is better).

Spatial upscaling (PSNR / LPIPS):

Method	2×	3×	4×
H3 VAE reconstruction (ceiling)	38.08 / 0.032	38.08 / 0.032	38.08 / 0.032
v2	31.27 / 0.106	29.45 / 0.149	28.27 / 0.191
v1 (minimax_h3_latent_upscaler_3d_conv_v1_fp16)	29.35 / 0.129	28.36 / 0.186	27.57 / 0.235
low-res decode + bicubic	34.60 / 0.095	32.47 / 0.178	30.52 / 0.253

Frame-rate upscaling to 24 fps (PSNR / LPIPS / tLP). The baselines decode the low clip, upscale it bicubically in space, then interpolate in time. RIFE is RIFE 4.7, as ComfyUI-Frame-Interpolation runs it by default.

Input	Spatial	v2	RIFE 4.7 (pixels)	linear blend (pixels)	latent interpolation
8 fps	1×	28.12 / 0.092 / 0.0126	31.68 / 0.040 / 0.0116	29.90 / 0.058 / 0.0126	22.18 / 0.145 / 0.0402
8 fps	2×	27.28 / 0.113 / 0.0126	30.10 / 0.095 / 0.0132	28.87 / 0.107 / 0.0133	19.41 / 0.376 / 0.0438
8 fps	3×	26.13 / 0.153 / 0.0131	28.79 / 0.165 / 0.0143	27.94 / 0.174 / 0.0137	18.13 / 0.478 / 0.0531
12 fps	1×	29.71 / 0.076 / 0.0111	34.09 / 0.032 / 0.0088	32.30 / 0.046 / 0.0154	23.51 / 0.112 / 0.0260
12 fps	2×	28.44 / 0.097 / 0.0112	31.77 / 0.087 / 0.0105	30.67 / 0.096 / 0.0133	19.83 / 0.358 / 0.0367
12 fps	3×	27.07 / 0.138 / 0.0119	29.89 / 0.158 / 0.0117	29.27 / 0.165 / 0.0118	18.38 / 0.463 / 0.0443

In latent space, v2 is far better than latent interpolation. In pixel space, RIFE still wins on PSNR at every setting, and on everything at 1× and 2×. v2 has better LPIPS from 3× and less flicker at 8 fps with a spatial upscale. 16 fps → 24 fps was trained but not evaluated.

Training

• Data: about 26k clips from OpenVid-1M (HD subset) and UltraVideo, encoded once with the H3 VAE.
• Pairs: high-res latents with low-res latents at 2–4× (24 fps), and at 8/12/16 fps (2–4×, plus 1× at 8/12 fps).
• Losses: latent L1 and FFT, plus pixel L1 / FFT / LPIPS / temporal LPIPS on decoded crops.
• Training run: 8,400 steps in total (AdamW).

License

The weights are released under CC BY-NC 4.0: free for non-commercial use with attribution.

• code/model.py adapts the network of Comfyui_Minimax_h3_latent_Upscaler (MIT License), as its header says.
• The model works on MiniMax-H3 VAE latents. Using it with MiniMax-H3 is also subject to MiniMax-H3's own license.
• Training data includes OpenVid-1M (CC BY 4.0).