Expand model card: VTuber pose-transfer beta docs
Browse files
README.md
CHANGED
|
@@ -3,6 +3,7 @@ license: apache-2.0
|
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
pipeline_tag: image-to-image
|
|
|
|
| 6 |
tags:
|
| 7 |
- image-to-image
|
| 8 |
- pose-transfer
|
|
@@ -10,4 +11,129 @@ tags:
|
|
| 10 |
- human-pose
|
| 11 |
- keypoints
|
| 12 |
- diffusion
|
| 13 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
pipeline_tag: image-to-image
|
| 6 |
+
library_name: pytorch
|
| 7 |
tags:
|
| 8 |
- image-to-image
|
| 9 |
- pose-transfer
|
|
|
|
| 11 |
- human-pose
|
| 12 |
- keypoints
|
| 13 |
- diffusion
|
| 14 |
+
- vtuber
|
| 15 |
+
- anime
|
| 16 |
+
- real-time
|
| 17 |
+
- windows
|
| 18 |
+
- cuda
|
| 19 |
+
---
|
| 20 |
+
|
| 21 |
+
# VTM-Elf 0.01
|
| 22 |
+
|
| 23 |
+
**A from-scratch pose-transfer + image-generation model built for VTubers.**
|
| 24 |
+
|
| 25 |
+
VTM-Elf takes a live pose (from your camera / skeleton) and renders a character image in that pose. Pose transfer and image generation live in the **same model base** β not a bolted-on ControlNet stack on top of a generic SD checkpoint.
|
| 26 |
+
|
| 27 |
+
> **Status: Beta**
|
| 28 |
+
> Image quality is still limited. That is expected at this stage and mostly comes down to **model size** and a **small training dataset** β not a permanent ceiling.
|
| 29 |
+
|
| 30 |
+
---
|
| 31 |
+
|
| 32 |
+
## What this model is for
|
| 33 |
+
|
| 34 |
+
| | |
|
| 35 |
+
|---|---|
|
| 36 |
+
| **Audience** | VTubers / live avatar pipelines |
|
| 37 |
+
| **Task** | Pose transfer β character image |
|
| 38 |
+
| **Input** | Pose (keypoints / skeleton derived from camera) + identity/reference context |
|
| 39 |
+
| **Output** | Character image matching the given pose |
|
| 40 |
+
| **Style** | Anime / VTuber-oriented |
|
| 41 |
+
|
| 42 |
+
### Pipeline (high level)
|
| 43 |
+
|
| 44 |
+
```text
|
| 45 |
+
Camera frame
|
| 46 |
+
β
|
| 47 |
+
Skeleton / keypoints
|
| 48 |
+
β
|
| 49 |
+
Pose normalizer (scale, center, consistent body frame)
|
| 50 |
+
β
|
| 51 |
+
VTM-Elf (pose-conditioned generation)
|
| 52 |
+
β
|
| 53 |
+
Character image in your pose
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
The normalizer is important: it averages and remaps joints into a consistent space so the character comes back with **correct proportions / dimensions**, not a warped copy of your raw camera pose.
|
| 57 |
+
|
| 58 |
+
---
|
| 59 |
+
|
| 60 |
+
## Getting started
|
| 61 |
+
|
| 62 |
+
### Requirements (current)
|
| 63 |
+
|
| 64 |
+
| Requirement | Support |
|
| 65 |
+
|---|---|
|
| 66 |
+
| **OS** | **Windows only** (other platforms not supported yet) |
|
| 67 |
+
| **GPU** | **NVIDIA + CUDA only** |
|
| 68 |
+
| **CPU / macOS / AMD** | Not supported |
|
| 69 |
+
|
| 70 |
+
If you are not on Windows with an NVIDIA CUDA GPU, this release will not run for you yet.
|
| 71 |
+
|
| 72 |
+
### Download the weights
|
| 73 |
+
|
| 74 |
+
```bash
|
| 75 |
+
hf download sinBoo1/VTM-Elf-0.01 VTM-elf.pt-000060000 --local-dir ./VTM-Elf-0.01
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
Or with Python:
|
| 79 |
+
|
| 80 |
+
```python
|
| 81 |
+
from huggingface_hub import hf_hub_download
|
| 82 |
+
|
| 83 |
+
ckpt = hf_hub_download(
|
| 84 |
+
repo_id="sinBoo1/VTM-Elf-0.01",
|
| 85 |
+
filename="VTM-elf.pt-000060000",
|
| 86 |
+
)
|
| 87 |
+
print(ckpt)
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
### Files
|
| 91 |
+
|
| 92 |
+
| File | Notes |
|
| 93 |
+
|---|---|
|
| 94 |
+
| `VTM-elf.pt-000060000` | Main checkpoint (~436 MB) β step 60k |
|
| 95 |
+
| `README.md` | This model card |
|
| 96 |
+
|
| 97 |
+
---
|
| 98 |
+
|
| 99 |
+
## Dataset & quality notes
|
| 100 |
+
|
| 101 |
+
This model was trained on a **small dedicated VTuber/character dataset**:
|
| 102 |
+
|
| 103 |
+
- about **1,500 characters**
|
| 104 |
+
- roughly **16β30 images per character**
|
| 105 |
+
|
| 106 |
+
That scale is the main reason generation quality is still rough in beta, together with the current model size. More data and a larger model should improve fidelity, identity consistency, and overall look over time.
|
| 107 |
+
|
| 108 |
+
**Please treat outputs as experimental** β fine for testing pose drive and pipeline work, not as a finished production look.
|
| 109 |
+
|
| 110 |
+
---
|
| 111 |
+
|
| 112 |
+
## Limitations
|
| 113 |
+
|
| 114 |
+
- **Beta quality** β soft detail, identity drift, and artifacts can happen
|
| 115 |
+
- **Small dataset** β limited character coverage and pose variety
|
| 116 |
+
- **Windows + NVIDIA CUDA only** for now
|
| 117 |
+
- Not a general-purpose text-to-image model β it is a **pose-driven VTuber renderer**
|
| 118 |
+
|
| 119 |
+
---
|
| 120 |
+
|
| 121 |
+
## Intended use
|
| 122 |
+
|
| 123 |
+
- Live / near-live VTuber pose driving
|
| 124 |
+
- Research and prototyping of pose β character pipelines
|
| 125 |
+
- Building on top of a VTuber-first backbone instead of generic SD pose tools
|
| 126 |
+
|
| 127 |
+
**Not intended for:** photoreal people generation, production-final art, or non-CUDA / non-Windows setups (yet).
|
| 128 |
+
|
| 129 |
+
---
|
| 130 |
+
|
| 131 |
+
## Citation / credit
|
| 132 |
+
|
| 133 |
+
If you use VTM-Elf in a project, a link back to this repo is appreciated:
|
| 134 |
+
|
| 135 |
+
[`https://huggingface.co/sinBoo1/VTM-Elf-0.01`](https://huggingface.co/sinBoo1/VTM-Elf-0.01)
|
| 136 |
+
|
| 137 |
+
---
|
| 138 |
+
|
| 139 |
+
*Built from scratch for VTubers. Still early β quality will improve with more data and scale.*
|