Text-to-Video
Diffusers
diffusers-training
lora
mochi-1-preview
mochi-1-preview-diffusers
template:sd-lora
Instructions to use weathon/mochi-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use weathon/mochi-lora with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import export_to_video # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("genmo/mochi-1-preview", dtype=torch.bfloat16, device_map="cuda") pipe.load_lora_weights("weathon/mochi-lora") prompt = "There is a *crab* blending into a +rocky ocean floor+ where the crab's mottled brown shell, rough texture, and uneven shape closely match the scattered rocks and coarse sand, all in muted brown and grey tones. The crab moves slowly and subtly, making it difficult to distinguish as its rough brown pattern looks just like a piece of rock among the uneven, similarly colored stones and patches of sand." output = pipe(prompt=prompt).frames[0] export_to_video(output, "output.mp4") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- Draw Things
End of training
Browse files- .gitattributes +3 -0
- README.md +92 -0
- checkpoint-100.pt +3 -0
- checkpoint-1000.pt +3 -0
- checkpoint-1100.pt +3 -0
- checkpoint-1200.pt +3 -0
- checkpoint-1300.pt +3 -0
- checkpoint-1400.pt +3 -0
- checkpoint-1500.pt +3 -0
- checkpoint-1600.pt +3 -0
- checkpoint-1700.pt +3 -0
- checkpoint-1800.pt +3 -0
- checkpoint-1900.pt +3 -0
- checkpoint-200.pt +3 -0
- checkpoint-2000.pt +3 -0
- checkpoint-2100.pt +3 -0
- checkpoint-2200.pt +3 -0
- checkpoint-2300.pt +3 -0
- checkpoint-2400.pt +3 -0
- checkpoint-2500.pt +3 -0
- checkpoint-2600.pt +3 -0
- checkpoint-2700.pt +3 -0
- checkpoint-2800.pt +3 -0
- checkpoint-2900.pt +3 -0
- checkpoint-300.pt +3 -0
- checkpoint-3000.pt +3 -0
- checkpoint-400.pt +3 -0
- checkpoint-500.pt +3 -0
- checkpoint-600.pt +3 -0
- checkpoint-700.pt +3 -0
- checkpoint-800.pt +3 -0
- checkpoint-900.pt +3 -0
- final_video_0.mp4 +3 -0
- pytorch_lora_weights.safetensors +3 -0
- test_video_0_There_is_a_*crab*_blendin.mp4 +3 -0
- validation_video_0_There_is_a_*crab*_blendin.mp4 +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
final_video_0.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
test_video_0_There_is_a_\*crab\*_blendin.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
validation_video_0_There_is_a_\*crab\*_blendin.mp4 filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,92 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
base_model: genmo/mochi-1-preview
|
| 3 |
+
library_name: diffusers
|
| 4 |
+
license: apache-2.0
|
| 5 |
+
instance_prompt: There is a *crab* blending into a +rocky ocean floor+ where the crab's
|
| 6 |
+
mottled brown shell, rough texture, and uneven shape closely match the scattered
|
| 7 |
+
rocks and coarse sand, all in muted brown and grey tones. The crab moves slowly
|
| 8 |
+
and subtly, making it difficult to distinguish as its rough brown pattern looks
|
| 9 |
+
just like a piece of rock among the uneven, similarly colored stones and patches
|
| 10 |
+
of sand.
|
| 11 |
+
widget:
|
| 12 |
+
- text: There is a *crab* blending into a +rocky ocean floor+ where the crab's mottled
|
| 13 |
+
brown shell, rough texture, and uneven shape closely match the scattered rocks
|
| 14 |
+
and coarse sand, all in muted brown and grey tones. The crab moves slowly and
|
| 15 |
+
subtly, making it difficult to distinguish as its rough brown pattern looks just
|
| 16 |
+
like a piece of rock among the uneven, similarly colored stones and patches of
|
| 17 |
+
sand.
|
| 18 |
+
output:
|
| 19 |
+
url: final_video_0.mp4
|
| 20 |
+
tags:
|
| 21 |
+
- text-to-video
|
| 22 |
+
- diffusers-training
|
| 23 |
+
- diffusers
|
| 24 |
+
- lora
|
| 25 |
+
- mochi-1-preview
|
| 26 |
+
- mochi-1-preview-diffusers
|
| 27 |
+
- template:sd-lora
|
| 28 |
+
---
|
| 29 |
+
|
| 30 |
+
<!-- This model card has been generated automatically according to the information the training script had access to. You
|
| 31 |
+
should probably proofread and complete it, then remove this comment. -->
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
# Mochi-1 Preview LoRA Finetune
|
| 35 |
+
|
| 36 |
+
<Gallery />
|
| 37 |
+
|
| 38 |
+
## Model description
|
| 39 |
+
|
| 40 |
+
This is a lora finetune of the Mochi-1 preview model `genmo/mochi-1-preview`.
|
| 41 |
+
|
| 42 |
+
The model was trained using [CogVideoX Factory](https://github.com/a-r-r-o-w/cogvideox-factory) - a repository containing memory-optimized training scripts for the CogVideoX and Mochi family of models using [TorchAO](https://github.com/pytorch/ao) and [DeepSpeed](https://github.com/microsoft/DeepSpeed). The scripts were adopted from [CogVideoX Diffusers trainer](https://github.com/huggingface/diffusers/blob/main/examples/cogvideo/train_cogvideox_lora.py).
|
| 43 |
+
|
| 44 |
+
## Download model
|
| 45 |
+
|
| 46 |
+
[Download LoRA](weathon/mochi-lora/tree/main) in the Files & Versions tab.
|
| 47 |
+
|
| 48 |
+
## Usage
|
| 49 |
+
|
| 50 |
+
Requires the [🧨 Diffusers library](https://github.com/huggingface/diffusers) installed.
|
| 51 |
+
|
| 52 |
+
```py
|
| 53 |
+
from diffusers import MochiPipeline
|
| 54 |
+
from diffusers.utils import export_to_video
|
| 55 |
+
import torch
|
| 56 |
+
|
| 57 |
+
pipe = MochiPipeline.from_pretrained("genmo/mochi-1-preview")
|
| 58 |
+
pipe.load_lora_weights("CHANGE_ME")
|
| 59 |
+
pipe.enable_model_cpu_offload()
|
| 60 |
+
|
| 61 |
+
with torch.autocast("cuda", torch.bfloat16):
|
| 62 |
+
video = pipe(
|
| 63 |
+
prompt="CHANGE_ME",
|
| 64 |
+
guidance_scale=6.0,
|
| 65 |
+
num_inference_steps=64,
|
| 66 |
+
height=480,
|
| 67 |
+
width=848,
|
| 68 |
+
max_sequence_length=256,
|
| 69 |
+
output_type="np"
|
| 70 |
+
).frames[0]
|
| 71 |
+
export_to_video(video)
|
| 72 |
+
```
|
| 73 |
+
|
| 74 |
+
For more details, including weighting, merging and fusing LoRAs, check the [documentation](https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters) on loading LoRAs in diffusers.
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
## Intended uses & limitations
|
| 79 |
+
|
| 80 |
+
#### How to use
|
| 81 |
+
|
| 82 |
+
```python
|
| 83 |
+
# TODO: add an example code snippet for running this diffusion pipeline
|
| 84 |
+
```
|
| 85 |
+
|
| 86 |
+
#### Limitations and bias
|
| 87 |
+
|
| 88 |
+
[TODO: provide examples of latent issues and potential remediations]
|
| 89 |
+
|
| 90 |
+
## Training details
|
| 91 |
+
|
| 92 |
+
[TODO: describe the data used to train the model]
|
checkpoint-100.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8fc586abb7df0ed6e936e635a8511c0b7e7ef3b0531d1b3b077f504249bc1a50
|
| 3 |
+
size 456595953
|
checkpoint-1000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b2e70580217a5ab651dde29b3a6e904a7d8d10012411a251d4d810e4bfcd2b04
|
| 3 |
+
size 456597695
|
checkpoint-1100.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9e96f282059ad0dda5b6522b573d7b72fdcea5cd87dc24f7f98e5c8572b6940c
|
| 3 |
+
size 456597695
|
checkpoint-1200.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8ad8dec239df77d1b93ba4896db3495d3be2905b1ee8c4a8fdd35bde9a0a92df
|
| 3 |
+
size 456597695
|
checkpoint-1300.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5ac71208177601ef6cbf0304b100d59ac4c5092103dbef5b65e2d288d70517a2
|
| 3 |
+
size 456597695
|
checkpoint-1400.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0bc3d7655063cc009bc1f5864760fcfbbdcdbc32051779fb2a21888df25a55d1
|
| 3 |
+
size 456597695
|
checkpoint-1500.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bf54fa5119d91c8b8d07b8d7a6e1c8fdd3f8512d2ed8681b50eaf44d9c415c38
|
| 3 |
+
size 456597695
|
checkpoint-1600.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c8c7f5cff56ac0848f12a0b84d01eaeb390a09547b942774f26fa3588e9d08ed
|
| 3 |
+
size 456597695
|
checkpoint-1700.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:769bcbfc2e6d04c8e77c5fbe3aeb8e0ede75ec0f1574e589a078216b61b4b435
|
| 3 |
+
size 456597695
|
checkpoint-1800.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ca2ab4a1e70f5a54e1754cd59f2770ab1babbc0a55cee210f189d29a333c6274
|
| 3 |
+
size 456597695
|
checkpoint-1900.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b7767d4d4c6f86162b8fa0f30476d989ffec3f551bc98a6a2443fb2c42a25002
|
| 3 |
+
size 456597695
|
checkpoint-200.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:590067e7be24e448b42efde1b0edb4488b46c494f11e27fd5a1057a072f6d810
|
| 3 |
+
size 456596145
|
checkpoint-2000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8773d0bb9f700b71418e7747d07173eeb722e4432efc6d1bfe0e771d24655f9e
|
| 3 |
+
size 456597695
|
checkpoint-2100.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0e9b3440b95600c9816f36d07a72c164ff1eab7b67873d439f65a9c851dc77f4
|
| 3 |
+
size 456597695
|
checkpoint-2200.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:55a803e6a4efded6f188aad2b92eac21a3a24814cf95dab86fdd84cf50638007
|
| 3 |
+
size 456597695
|
checkpoint-2300.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d50faa26506adb4cdd3eba5f9da194d3a58b72127faffc886fd9831dbaa5e382
|
| 3 |
+
size 456597695
|
checkpoint-2400.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:104949ad709d50e3ab5427099cab7e4ac2c3a759c625b6949142b6880f2bf317
|
| 3 |
+
size 456597695
|
checkpoint-2500.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f0a4ef871df7416db557c171d5e73cd79995d69cb9778aada8d3d6565cf3f80e
|
| 3 |
+
size 456597695
|
checkpoint-2600.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b6054ad882fdd74bfa935e9a6d248845efafe91579f1a20e0243fd6f2d7c957d
|
| 3 |
+
size 456597695
|
checkpoint-2700.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3212f63cd1da55666cf2f5ea2578e593b4241ff691e2489f448d5318625d317f
|
| 3 |
+
size 456597695
|
checkpoint-2800.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e848eb6e706071cbda0388b862bd458f47ed6c2d34f7561085ef3436540b979d
|
| 3 |
+
size 456597695
|
checkpoint-2900.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:38fc475ad96271e70f93e288f7602dfc229d2c98ce467b2e09fde075c82a32a9
|
| 3 |
+
size 456597695
|
checkpoint-300.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fc7997757cde3e18d3d9a0309e95efa5212d5e51bf7ef77d09ce8d4761fefd98
|
| 3 |
+
size 456596145
|
checkpoint-3000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8778239ed8eb1a8f954e77a90a15f4ae294c07fe1193e72e563ae01fa7b634ee
|
| 3 |
+
size 456597695
|
checkpoint-400.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d3ae3998884220c5470de4d755cc30da40c1d2a229566d26fd6c811f6a9f97f9
|
| 3 |
+
size 456596145
|
checkpoint-500.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1c77c1e737d36eeb3bb47006f3fe9e74c474a724875ac10c0b367958262eb6ce
|
| 3 |
+
size 456596145
|
checkpoint-600.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:910dead2effbfc4c2284fc55162a0e2ad60fd7b07d141d36c9d06c628a1a16cc
|
| 3 |
+
size 456596145
|
checkpoint-700.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d32f05af5525eb5b5c37768dc15ae894651ec91711995b7682f52fcef247b7de
|
| 3 |
+
size 456596145
|
checkpoint-800.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b064b86d1fffb09d3315bf23a39225cc3080440c7ea60b8401cf74acbf1a89cd
|
| 3 |
+
size 456596145
|
checkpoint-900.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d281b8762f97f49054a1fb6c6ed26c9a13b3540413f8b1d88992534c78018df4
|
| 3 |
+
size 456596145
|
final_video_0.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:36001da633ca89baa7cc42870c6b64a9bd9038ce2ca2f403ad2491bbc984041c
|
| 3 |
+
size 309566
|
pytorch_lora_weights.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:648be1860c65d9d4e3e80a2861367f814b8a7d5fa908b05d80e0530862eb5100
|
| 3 |
+
size 152094232
|
test_video_0_There_is_a_*crab*_blendin.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:36001da633ca89baa7cc42870c6b64a9bd9038ce2ca2f403ad2491bbc984041c
|
| 3 |
+
size 309566
|
validation_video_0_There_is_a_*crab*_blendin.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4752bd902690a999e279beccf3480614338406b61f3612617820952599b76a5a
|
| 3 |
+
size 165240
|