Image-Text-to-Video
Diffusers
Safetensors
MiniMaxH3ModularPipeline
text-to-video
image-to-video
video-to-video
text-to-audio-video
image-to-audio-video
image-text-to-audio-video
video-to-audio-video
audio-to-audio-video
audio-video-generation
multimodal
synchronized-audio-video
reference-to-audio-video
Instructions to use MiniMaxAI/MiniMax-H3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use MiniMaxAI/MiniMax-H3 with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("MiniMaxAI/MiniMax-H3", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Scope the download command to the original checkpoint folders
#16
by multimodalart HF Staff - opened
- README.md +16 -14
- assets/action-reference.mov +3 -0
- assets/character-action-reference.png +3 -0
- assets/character-replacement-action-reference.mp4 +3 -0
- assets/fashion-glasses-ad.mp4 +3 -0
- assets/fashion-glasses-reference-1.png +3 -0
- assets/fashion-glasses-reference-2.png +3 -0
- assets/fashion-glasses-reference-3.png +3 -0
- assets/fashion-glasses-reference-4.png +3 -0
- assets/fl2va-clay-fox-reference.png +3 -0
- assets/fl2va-clay-fox.mp4 +3 -0
- assets/h3-architecture.png +3 -0
- assets/h3-cinematic-shot.mp4 +3 -0
- assets/h3-suspense-title.mp4 +3 -0
- assets/logo.svg +16 -0
- assets/reference-image-1.png +3 -0
- assets/reference-image-2.png +3 -0
- assets/robot-arm-red-cube.mp4 +3 -0
- assets/t2va-768p-demo.mp4 +3 -0
- model_index.json +0 -139
README.md
CHANGED
|
@@ -38,16 +38,6 @@ tags:
|
|
| 38 |
|
| 39 |
# MiniMax H3
|
| 40 |
|
| 41 |
-
## Online API
|
| 42 |
-
Use MiniMax\-H3 directly via API\.
|
| 43 |
-
- Global: [platform\.minimax\.io](https://platform.minimax.io/docs/api-reference/video-generation-v2-create) \| CN: [platform\.minimaxi\.com](https://platform.minimaxi.com/docs/api-reference/video-generation-v2-create)
|
| 44 |
-
|
| 45 |
-
## Online App
|
| 46 |
-
Use MiniMax\-H3 directly via App\.
|
| 47 |
-
- WebApp Global: [hailuoai\.video](https://hailuoai.video) \| CN: [hailuoai\.com](https://hailuoai.com/)
|
| 48 |
-
- Desktop Global: [hub\.minimax\.io](https://hub.minimax.io/) \| CN: [hub\.minimaxi\.com](https://hub.minimaxi.com/)
|
| 49 |
-
|
| 50 |
-
|
| 51 |
## System Overview
|
| 52 |
MiniMax H3 is a general-purpose, omni-modal generative system. It supports unified understanding of multimodal contexts composed of text, images, video, and audio, and can generate video with native stereo audio at resolutions up to 2K and durations of up to 15 seconds. Thanks to its task-generalization-oriented system design, H3 already possesses broad multimodal context understanding and generation capabilities at the pre-training stage, enabling outstanding performance in following complex multimodal instructions.
|
| 53 |
|
|
@@ -76,6 +66,20 @@ The complete H3 system consists of the following three modules:
|
|
| 76 |
- H3-Base: Generates audio and video based on the H3-Context-IR output, producing results at 768p resolution.
|
| 77 |
- H3-Regenerate-2K: Feeds the 768p result together with the original context back into H3 to regenerate the output at 2K resolution. This process leverages both H3’s powerful generative capabilities and the rich information contained in the original context, enabling it to produce high-resolution outputs with more accurate details and greater visual fidelity.
|
| 78 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 79 |
## Model Architecture
|
| 80 |
|
| 81 |
### H3\-Context\-IR
|
|
@@ -190,14 +194,12 @@ Each checkpoint is distributed as a self\-contained Hugging Face\-style reposito
|
|
| 190 |
|
| 191 |
Download the model. The repository hosts the original checkpoint (`FL2VA/`, `Ref2VA/`) and the diffusers format side by side, so scope the download to what your framework needs:
|
| 192 |
|
| 193 |
-
`model_index.json` is the repository-level modular index. The task-family-specific diffusers indexes remain under `FL2VA/model_index.json` and `Ref2VA/model_index.json`.
|
| 194 |
-
|
| 195 |
```bash
|
| 196 |
# Original checkpoint, both task families (SGLang, vLLM):
|
| 197 |
-
hf download MiniMaxAI/MiniMax-H3 --include "
|
| 198 |
|
| 199 |
# Or a single task family:
|
| 200 |
-
hf download MiniMaxAI/MiniMax-H3 --include "
|
| 201 |
```
|
| 202 |
|
| 203 |
diffusers users do not need a manual download: `ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3")` fetches exactly the components it needs. See the [diffusers documentation](https://github.com/huggingface/diffusers/blob/minimax-h3/docs/source/en/api/pipelines/minimax_h3.md) for loading recipes.
|
|
|
|
| 38 |
|
| 39 |
# MiniMax H3
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
## System Overview
|
| 42 |
MiniMax H3 is a general-purpose, omni-modal generative system. It supports unified understanding of multimodal contexts composed of text, images, video, and audio, and can generate video with native stereo audio at resolutions up to 2K and durations of up to 15 seconds. Thanks to its task-generalization-oriented system design, H3 already possesses broad multimodal context understanding and generation capabilities at the pre-training stage, enabling outstanding performance in following complex multimodal instructions.
|
| 43 |
|
|
|
|
| 66 |
- H3-Base: Generates audio and video based on the H3-Context-IR output, producing results at 768p resolution.
|
| 67 |
- H3-Regenerate-2K: Feeds the 768p result together with the original context back into H3 to regenerate the output at 2K resolution. This process leverages both H3’s powerful generative capabilities and the rich information contained in the original context, enabling it to produce high-resolution outputs with more accurate details and greater visual fidelity.
|
| 68 |
|
| 69 |
+
## Online API
|
| 70 |
+
|
| 71 |
+
Use MiniMax\-H3 directly via API\.
|
| 72 |
+
|
| 73 |
+
- Global: [platform\.minimax\.io](https://platform.minimax.io/docs/api-reference/video-generation-v2-create) \| CN: [platform\.minimaxi\.com](https://platform.minimaxi.com/docs/api-reference/video-generation-v2-create)
|
| 74 |
+
|
| 75 |
+
## Online App
|
| 76 |
+
|
| 77 |
+
Use MiniMax\-H3 directly via App\.
|
| 78 |
+
|
| 79 |
+
- WebApp Global: [hailuoai\.video](https://hailuoai.video) \| CN: [hailuoai\.com](https://hailuoai.com/)
|
| 80 |
+
|
| 81 |
+
- Desktop Global: [hub\.minimax\.io](https://hub.minimax.io/) \| CN: [hub\.minimaxi\.com](https://hub.minimaxi.com/)
|
| 82 |
+
|
| 83 |
## Model Architecture
|
| 84 |
|
| 85 |
### H3\-Context\-IR
|
|
|
|
| 194 |
|
| 195 |
Download the model. The repository hosts the original checkpoint (`FL2VA/`, `Ref2VA/`) and the diffusers format side by side, so scope the download to what your framework needs:
|
| 196 |
|
|
|
|
|
|
|
| 197 |
```bash
|
| 198 |
# Original checkpoint, both task families (SGLang, vLLM):
|
| 199 |
+
hf download MiniMaxAI/MiniMax-H3 --include "FL2VA/*" "Ref2VA/*" --local-dir MiniMax-H3
|
| 200 |
|
| 201 |
# Or a single task family:
|
| 202 |
+
hf download MiniMaxAI/MiniMax-H3 --include "FL2VA/*" --local-dir MiniMax-H3
|
| 203 |
```
|
| 204 |
|
| 205 |
diffusers users do not need a manual download: `ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3")` fetches exactly the components it needs. See the [diffusers documentation](https://github.com/huggingface/diffusers/blob/minimax-h3/docs/source/en/api/pipelines/minimax_h3.md) for loading recipes.
|
assets/action-reference.mov
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:116e0f08a399834e7ffc3472d036659b33250f4ba4f0b7e63f17ec07cb58e4dc
|
| 3 |
+
size 31446151
|
assets/character-action-reference.png
ADDED
|
Git LFS Details
|
assets/character-replacement-action-reference.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:796c264126110d87992dcb54213ac0697920cb4ddf3d5a06aa36069386fc4fa1
|
| 3 |
+
size 8223683
|
assets/fashion-glasses-ad.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a11e6b0095dd7766b724bb886edf4d7d9930af44da6839ee195269ac4fc60ba4
|
| 3 |
+
size 10938085
|
assets/fashion-glasses-reference-1.png
ADDED
|
Git LFS Details
|
assets/fashion-glasses-reference-2.png
ADDED
|
Git LFS Details
|
assets/fashion-glasses-reference-3.png
ADDED
|
Git LFS Details
|
assets/fashion-glasses-reference-4.png
ADDED
|
Git LFS Details
|
assets/fl2va-clay-fox-reference.png
ADDED
|
Git LFS Details
|
assets/fl2va-clay-fox.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4604b0838b736ecaa209c978f880bfacbb3a0fc82a8a0517c1f5aa16454b50ef
|
| 3 |
+
size 10931936
|
assets/h3-architecture.png
ADDED
|
Git LFS Details
|
assets/h3-cinematic-shot.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:436defc81cfa7d53aef423f368be82ead20056088087e0be308b6e3865e8fb81
|
| 3 |
+
size 4361547
|
assets/h3-suspense-title.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:90ebbd7edc71c9a0151c3064126acd2dbb5b7cba58ce459e223e29f2c04f9186
|
| 3 |
+
size 16809220
|
assets/logo.svg
ADDED
|
|
assets/reference-image-1.png
ADDED
|
Git LFS Details
|
assets/reference-image-2.png
ADDED
|
Git LFS Details
|
assets/robot-arm-red-cube.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1a751e6100dbf6502a99a2adc0b12171304ca7b17eecea70e2ed035a73ea692c
|
| 3 |
+
size 2259732
|
assets/t2va-768p-demo.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d66903241362e224085cc93f7a5e70fba6ab378d0ac6ff6af83acf2559849a42
|
| 3 |
+
size 1637373
|
model_index.json
DELETED
|
@@ -1,139 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"_class_name": "MiniMaxH3ModularPipeline",
|
| 3 |
-
"_diffusers_version": "0.36.0.dev0",
|
| 4 |
-
"_blocks_class_name": "MiniMaxH3Blocks",
|
| 5 |
-
"_minimax_h3": {
|
| 6 |
-
"schema_version": 1,
|
| 7 |
-
"index_scope": "repository",
|
| 8 |
-
"task_family_indexes": {
|
| 9 |
-
"fl2va": "FL2VA/model_index.json",
|
| 10 |
-
"ref2va": "Ref2VA/model_index.json"
|
| 11 |
-
}
|
| 12 |
-
},
|
| 13 |
-
"text_encoder": [
|
| 14 |
-
"transformers",
|
| 15 |
-
"Qwen3VLForConditionalGeneration",
|
| 16 |
-
{
|
| 17 |
-
"type_hint": [
|
| 18 |
-
"transformers",
|
| 19 |
-
"Qwen3VLForConditionalGeneration"
|
| 20 |
-
],
|
| 21 |
-
"pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
|
| 22 |
-
"subfolder": "text_encoder",
|
| 23 |
-
"variant": null,
|
| 24 |
-
"revision": null
|
| 25 |
-
}
|
| 26 |
-
],
|
| 27 |
-
"tokenizer": [
|
| 28 |
-
"transformers",
|
| 29 |
-
"Qwen2TokenizerFast",
|
| 30 |
-
{
|
| 31 |
-
"type_hint": [
|
| 32 |
-
"transformers",
|
| 33 |
-
"Qwen2TokenizerFast"
|
| 34 |
-
],
|
| 35 |
-
"pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
|
| 36 |
-
"subfolder": "tokenizer",
|
| 37 |
-
"variant": null,
|
| 38 |
-
"revision": null
|
| 39 |
-
}
|
| 40 |
-
],
|
| 41 |
-
"processor": [
|
| 42 |
-
"transformers",
|
| 43 |
-
"Qwen3VLProcessor",
|
| 44 |
-
{
|
| 45 |
-
"type_hint": [
|
| 46 |
-
"transformers",
|
| 47 |
-
"Qwen3VLProcessor"
|
| 48 |
-
],
|
| 49 |
-
"pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
|
| 50 |
-
"subfolder": "processor",
|
| 51 |
-
"variant": null,
|
| 52 |
-
"revision": null
|
| 53 |
-
}
|
| 54 |
-
],
|
| 55 |
-
"vae": [
|
| 56 |
-
"diffusers",
|
| 57 |
-
"AutoencoderKLMiniMaxH3",
|
| 58 |
-
{
|
| 59 |
-
"type_hint": [
|
| 60 |
-
"diffusers",
|
| 61 |
-
"AutoencoderKLMiniMaxH3"
|
| 62 |
-
],
|
| 63 |
-
"pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
|
| 64 |
-
"subfolder": "vae",
|
| 65 |
-
"variant": null,
|
| 66 |
-
"revision": null
|
| 67 |
-
}
|
| 68 |
-
],
|
| 69 |
-
"audio_vae": [
|
| 70 |
-
"diffusers",
|
| 71 |
-
"AutoencoderKLMiniMaxH3Audio",
|
| 72 |
-
{
|
| 73 |
-
"type_hint": [
|
| 74 |
-
"diffusers",
|
| 75 |
-
"AutoencoderKLMiniMaxH3Audio"
|
| 76 |
-
],
|
| 77 |
-
"pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
|
| 78 |
-
"subfolder": "audio_vae",
|
| 79 |
-
"variant": null,
|
| 80 |
-
"revision": null
|
| 81 |
-
}
|
| 82 |
-
],
|
| 83 |
-
"transformer": [
|
| 84 |
-
"diffusers",
|
| 85 |
-
"MiniMaxH3Transformer3DModel",
|
| 86 |
-
{
|
| 87 |
-
"type_hint": [
|
| 88 |
-
"diffusers",
|
| 89 |
-
"MiniMaxH3Transformer3DModel"
|
| 90 |
-
],
|
| 91 |
-
"pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
|
| 92 |
-
"subfolder": "transformer",
|
| 93 |
-
"variant": null,
|
| 94 |
-
"revision": null
|
| 95 |
-
}
|
| 96 |
-
],
|
| 97 |
-
"transformer_ref": [
|
| 98 |
-
"diffusers",
|
| 99 |
-
"MiniMaxH3Transformer3DModel",
|
| 100 |
-
{
|
| 101 |
-
"type_hint": [
|
| 102 |
-
"diffusers",
|
| 103 |
-
"MiniMaxH3Transformer3DModel"
|
| 104 |
-
],
|
| 105 |
-
"pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
|
| 106 |
-
"subfolder": "transformer_ref",
|
| 107 |
-
"variant": null,
|
| 108 |
-
"revision": null
|
| 109 |
-
}
|
| 110 |
-
],
|
| 111 |
-
"scheduler": [
|
| 112 |
-
"diffusers",
|
| 113 |
-
"MiniMaxH3Scheduler",
|
| 114 |
-
{
|
| 115 |
-
"type_hint": [
|
| 116 |
-
"diffusers",
|
| 117 |
-
"MiniMaxH3Scheduler"
|
| 118 |
-
],
|
| 119 |
-
"pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
|
| 120 |
-
"subfolder": "scheduler",
|
| 121 |
-
"variant": null,
|
| 122 |
-
"revision": null
|
| 123 |
-
}
|
| 124 |
-
],
|
| 125 |
-
"audio_scheduler": [
|
| 126 |
-
"diffusers",
|
| 127 |
-
"MiniMaxH3Scheduler",
|
| 128 |
-
{
|
| 129 |
-
"type_hint": [
|
| 130 |
-
"diffusers",
|
| 131 |
-
"MiniMaxH3Scheduler"
|
| 132 |
-
],
|
| 133 |
-
"pretrained_model_name_or_path": "MiniMaxAI/MiniMax-H3",
|
| 134 |
-
"subfolder": "audio_scheduler",
|
| 135 |
-
"variant": null,
|
| 136 |
-
"revision": null
|
| 137 |
-
}
|
| 138 |
-
]
|
| 139 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|