sinBoo1 commited on
Commit
2081f54
Β·
verified Β·
1 Parent(s): ad43b1b

Expand model card: VTuber pose-transfer beta docs

Browse files
Files changed (1) hide show
  1. README.md +127 -1
README.md CHANGED
@@ -3,6 +3,7 @@ license: apache-2.0
3
  language:
4
  - en
5
  pipeline_tag: image-to-image
 
6
  tags:
7
  - image-to-image
8
  - pose-transfer
@@ -10,4 +11,129 @@ tags:
10
  - human-pose
11
  - keypoints
12
  - diffusion
13
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  language:
4
  - en
5
  pipeline_tag: image-to-image
6
+ library_name: pytorch
7
  tags:
8
  - image-to-image
9
  - pose-transfer
 
11
  - human-pose
12
  - keypoints
13
  - diffusion
14
+ - vtuber
15
+ - anime
16
+ - real-time
17
+ - windows
18
+ - cuda
19
+ ---
20
+
21
+ # VTM-Elf 0.01
22
+
23
+ **A from-scratch pose-transfer + image-generation model built for VTubers.**
24
+
25
+ VTM-Elf takes a live pose (from your camera / skeleton) and renders a character image in that pose. Pose transfer and image generation live in the **same model base** β€” not a bolted-on ControlNet stack on top of a generic SD checkpoint.
26
+
27
+ > **Status: Beta**
28
+ > Image quality is still limited. That is expected at this stage and mostly comes down to **model size** and a **small training dataset** β€” not a permanent ceiling.
29
+
30
+ ---
31
+
32
+ ## What this model is for
33
+
34
+ | | |
35
+ |---|---|
36
+ | **Audience** | VTubers / live avatar pipelines |
37
+ | **Task** | Pose transfer β†’ character image |
38
+ | **Input** | Pose (keypoints / skeleton derived from camera) + identity/reference context |
39
+ | **Output** | Character image matching the given pose |
40
+ | **Style** | Anime / VTuber-oriented |
41
+
42
+ ### Pipeline (high level)
43
+
44
+ ```text
45
+ Camera frame
46
+ ↓
47
+ Skeleton / keypoints
48
+ ↓
49
+ Pose normalizer (scale, center, consistent body frame)
50
+ ↓
51
+ VTM-Elf (pose-conditioned generation)
52
+ ↓
53
+ Character image in your pose
54
+ ```
55
+
56
+ The normalizer is important: it averages and remaps joints into a consistent space so the character comes back with **correct proportions / dimensions**, not a warped copy of your raw camera pose.
57
+
58
+ ---
59
+
60
+ ## Getting started
61
+
62
+ ### Requirements (current)
63
+
64
+ | Requirement | Support |
65
+ |---|---|
66
+ | **OS** | **Windows only** (other platforms not supported yet) |
67
+ | **GPU** | **NVIDIA + CUDA only** |
68
+ | **CPU / macOS / AMD** | Not supported |
69
+
70
+ If you are not on Windows with an NVIDIA CUDA GPU, this release will not run for you yet.
71
+
72
+ ### Download the weights
73
+
74
+ ```bash
75
+ hf download sinBoo1/VTM-Elf-0.01 VTM-elf.pt-000060000 --local-dir ./VTM-Elf-0.01
76
+ ```
77
+
78
+ Or with Python:
79
+
80
+ ```python
81
+ from huggingface_hub import hf_hub_download
82
+
83
+ ckpt = hf_hub_download(
84
+ repo_id="sinBoo1/VTM-Elf-0.01",
85
+ filename="VTM-elf.pt-000060000",
86
+ )
87
+ print(ckpt)
88
+ ```
89
+
90
+ ### Files
91
+
92
+ | File | Notes |
93
+ |---|---|
94
+ | `VTM-elf.pt-000060000` | Main checkpoint (~436 MB) β€” step 60k |
95
+ | `README.md` | This model card |
96
+
97
+ ---
98
+
99
+ ## Dataset & quality notes
100
+
101
+ This model was trained on a **small dedicated VTuber/character dataset**:
102
+
103
+ - about **1,500 characters**
104
+ - roughly **16–30 images per character**
105
+
106
+ That scale is the main reason generation quality is still rough in beta, together with the current model size. More data and a larger model should improve fidelity, identity consistency, and overall look over time.
107
+
108
+ **Please treat outputs as experimental** β€” fine for testing pose drive and pipeline work, not as a finished production look.
109
+
110
+ ---
111
+
112
+ ## Limitations
113
+
114
+ - **Beta quality** β€” soft detail, identity drift, and artifacts can happen
115
+ - **Small dataset** β€” limited character coverage and pose variety
116
+ - **Windows + NVIDIA CUDA only** for now
117
+ - Not a general-purpose text-to-image model β€” it is a **pose-driven VTuber renderer**
118
+
119
+ ---
120
+
121
+ ## Intended use
122
+
123
+ - Live / near-live VTuber pose driving
124
+ - Research and prototyping of pose β†’ character pipelines
125
+ - Building on top of a VTuber-first backbone instead of generic SD pose tools
126
+
127
+ **Not intended for:** photoreal people generation, production-final art, or non-CUDA / non-Windows setups (yet).
128
+
129
+ ---
130
+
131
+ ## Citation / credit
132
+
133
+ If you use VTM-Elf in a project, a link back to this repo is appreciated:
134
+
135
+ [`https://huggingface.co/sinBoo1/VTM-Elf-0.01`](https://huggingface.co/sinBoo1/VTM-Elf-0.01)
136
+
137
+ ---
138
+
139
+ *Built from scratch for VTubers. Still early β€” quality will improve with more data and scale.*