Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: diffsynth
|
| 4 |
+
pipeline_tag: text-to-video
|
| 5 |
+
tags:
|
| 6 |
+
- wan
|
| 7 |
+
- wan2.1
|
| 8 |
+
- echo-memory
|
| 9 |
+
- text-to-video
|
| 10 |
+
base_model: Wan-AI/Wan2.1-T2V-1.3B
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# Echo-Memory `context_k1` — complete Wan 2.1 1.3B DiT
|
| 14 |
+
|
| 15 |
+
DiffSynth-facing **full DiT** for the released Echo-Memory `context_k1` row.
|
| 16 |
+
|
| 17 |
+
- Paper: [arXiv:2606.09803](https://arxiv.org/abs/2606.09803)
|
| 18 |
+
- Code: [Echo-Team-Joy-Future-Academy-JD/Echo-Memory](https://github.com/Echo-Team-Joy-Future-Academy-JD/Echo-Memory)
|
| 19 |
+
- Research overlay (HF): [Echo-Team/Echo-Memory](https://huggingface.co/Echo-Team/Echo-Memory) `context_k1/epoch-0.safetensors`
|
| 20 |
+
|
| 21 |
+
## What this file is
|
| 22 |
+
|
| 23 |
+
`diffusion_pytorch_model.safetensors` is a **complete Wan 2.1 T2V 1.3B DiT** (825 / 825 official keys).
|
| 24 |
+
|
| 25 |
+
It is official [Wan-AI/Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) DiT keys with the Echo-Memory `context_k1` overlay already merged. Research extras (`action_mlp`, `self_attn_with_action`, SSM / spatial slots) are **not** included, so the file loads like a normal Wan transformer (`strict=True` on `pipe.dit`).
|
| 26 |
+
|
| 27 |
+
T5 and VAE stay the official Wan files:
|
| 28 |
+
|
| 29 |
+
- `Wan-AI/Wan2.1-T2V-1.3B` `models_t5_umt5-xxl-enc-bf16.pth`
|
| 30 |
+
- `Wan-AI/Wan2.1-T2V-1.3B` `Wan2.1_VAE.pth`
|
| 31 |
+
|
| 32 |
+
## DiffSynth
|
| 33 |
+
|
| 34 |
+
```python
|
| 35 |
+
import torch
|
| 36 |
+
from diffsynth.pipelines.wan_video import WanVideoPipeline, ModelConfig
|
| 37 |
+
from diffsynth.utils.data import save_video
|
| 38 |
+
|
| 39 |
+
pipe = WanVideoPipeline.from_pretrained(
|
| 40 |
+
torch_dtype=torch.bfloat16,
|
| 41 |
+
device="cuda",
|
| 42 |
+
model_configs=[
|
| 43 |
+
ModelConfig(model_id="Wayne-King/echo-memory-context-k1-full", origin_file_pattern="diffusion_pytorch_model.safetensors"),
|
| 44 |
+
ModelConfig(model_id="Wan-AI/Wan2.1-T2V-1.3B", origin_file_pattern="models_t5_umt5-xxl-enc-bf16.pth"),
|
| 45 |
+
ModelConfig(model_id="Wan-AI/Wan2.1-T2V-1.3B", origin_file_pattern="Wan2.1_VAE.pth"),
|
| 46 |
+
],
|
| 47 |
+
)
|
| 48 |
+
video = pipe(prompt="A toy bear on a table, the camera rotates around it", seed=42)
|
| 49 |
+
save_video(video, "echo_memory_context_k1.mp4")
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
After this repo is mirrored on ModelScope, replace the DiT `model_id` with that ModelScope id (DiffSynth downloads from ModelScope by default).
|
| 53 |
+
|
| 54 |
+
## Notes
|
| 55 |
+
|
| 56 |
+
- 30,000-step `epoch-0` fine-tune, 640×352, 81-frame chunks.
|
| 57 |
+
- Camera-action / multi-chunk revisit protocol stays in the Echo-Memory repo; this file is the Wan DiT overlay only.
|
| 58 |
+
- Apache-2.0. Please cite the Echo-Memory paper if you use it.
|