Revise model card: architecture, 7 FPS note, method figure

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 VTM-elf.pt-000060000 filter=lfs diff=lfs merge=lfs -text
+vtm-elf-method.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -11,72 +11,96 @@ tags:
 - human-pose
 - keypoints
 - diffusion
+- dit
+- rectified-flow
 - vtuber
 - anime
-- real-time
 - windows
 - cuda
 ---
 
-# VTM-Elf 0.01
+<div align="center">
+  <img src="vtm-elf-method.png" alt="VTM-Elf method overview: pose keypoints → normalized pose → generated character" width="100%"/>
+</div>
 
-**A from-scratch pose-transfer + image-generation model built for VTubers.**
+# VTM-Elf 0.01
 
-VTM-Elf takes a live pose (from your camera / skeleton) and renders a character image in that pose. Pose transfer and image generation live in the **same model base** — not a bolted-on ControlNet stack on top of a generic SD checkpoint.
+Pose-conditioned character generation for VTuber pipelines.  
+**Status:** beta · **Platform:** Windows + NVIDIA CUDA only
 
-> **Status: Beta**  
-> Image quality is still limited. That is expected at this stage and mostly comes down to **model size** and a **small training dataset** — not a permanent ceiling.
+VTM-Elf maps a body pose to a character image. Pose control and appearance generation are trained in one backbone (not a separate ControlNet on a frozen SD UNet).
 
 ---
 
-## What this model is for
+## Architecture
 
-| | |
+| Component | Detail |
 |---|---|
-| **Audience** | VTubers / live avatar pipelines |
-| **Task** | Pose transfer → character image |
-| **Input** | Pose (keypoints / skeleton derived from camera) + identity/reference context |
-| **Output** | Character image matching the given pose |
-| **Style** | Anime / VTuber-oriented |
+| Backbone | Dual-stream DiT (`i1DiT` / MM-DiT-style joint attention) |
+| Size preset | **DiT-30M** — depth 10, hidden size 320, 5 heads, MLP ratio 4.0 |
+| Transport | Rectified flow, **velocity** prediction |
+| VAE | SD VAE (4-channel latents), inference at **768×768** |
+| Pose input | 37 keypoints × 4 (`x, y, score, visible`), rasterized pose maps; optional keypoint RoPE |
+| Identity | Reference latent tokens (whole-body + face crop tokens) |
+| Conditioning | AdaLN; independent dropout on pose / ref / text during training |
+| Other | Patch size 2, SwiGLU, RMSNorm, QK-norm, sinusoidal + RoPE positions |
+
+Training recipe (from `scratch_keypoint_ref`): keypoint conditioning + ref tokens, face-weighted velocity loss, optional VAE-REPA early in training, EMA.
 
-### Pipeline (high level)
+**Checkpoint:** `VTM-elf.pt-000060000` (~436 MB, step 60k).
+
+---
+
+## Method (short)
 
 ```text
-Camera frame
-    ↓
-Skeleton / keypoints
-    ↓
-Pose normalizer  (scale, center, consistent body frame)
-    ↓
-VTM-Elf  (pose-conditioned generation)
-    ↓
-Character image in your pose
+camera frame
+  → keypoint / skeleton estimate
+  → pose normalizer (consistent crop / scale / body frame)
+  → Dual-Stream DiT-30M  (pose maps + ref identity tokens)
+  → SD VAE decode
+  → character image
 ```
 
-The normalizer is important: it averages and remaps joints into a consistent space so the character comes back with **correct proportions / dimensions**, not a warped copy of your raw camera pose.
+The normalizer remaps raw tracker joints into model space (`norm_crop`, roughly `[-1, 1]` after pad-to-square) so scale and framing stay consistent between training and live drive.
 
 ---
 
-## Getting started
+## Measured latency
 
-### Requirements (current)
+| Hardware | Resolution | Observed rate |
+|---|---|---|
+| **RTX 5060 Ti** | 768×768 (current live path) | **~7 FPS** |
 
-| Requirement | Support |
-|---|---|
-| **OS** | **Windows only** (other platforms not supported yet) |
-| **GPU** | **NVIDIA + CUDA only** |
-| **CPU / macOS / AMD** | Not supported |
+This is the current end-to-end live figure on that GPU, not a theoretical peak. Throughput depends on sampling steps, CFG, and tracker overhead.
 
-If you are not on Windows with an NVIDIA CUDA GPU, this release will not run for you yet.
+---
+
+## Data
+
+Training used a small private character set:
+
+- ~**1,500** characters  
+- ~**16–30** images per character  
+
+Generation quality in this release is limited mainly by **model capacity (DiT-30M)** and **dataset scale**, not by the pose-conditioning design itself.
+
+---
+
+## Getting started
 
-### Download the weights
+### Requirements
+
+- **Windows**
+- **NVIDIA GPU + CUDA**
+- Other OS / AMD / CPU: **not supported** in this release
+
+### Download
 
 ```bash
 hf download sinBoo1/VTM-Elf-0.01 VTM-elf.pt-000060000 --local-dir ./VTM-Elf-0.01
 ```
 
-Or with Python:
-
 ```python
 from huggingface_hub import hf_hub_download
 
@@ -84,56 +108,38 @@ ckpt = hf_hub_download(
     repo_id="sinBoo1/VTM-Elf-0.01",
     filename="VTM-elf.pt-000060000",
 )
-print(ckpt)
 ```
 
 ### Files
 
-| File | Notes |
+| File | Description |
 |---|---|
-| `VTM-elf.pt-000060000` | Main checkpoint (~436 MB) — step 60k |
-| `README.md` | This model card |
-
----
-
-## Dataset & quality notes
-
-This model was trained on a **small dedicated VTuber/character dataset**:
-
-- about **1,500 characters**
-- roughly **16–30 images per character**
+| `VTM-elf.pt-000060000` | DiT-30M weights (step 60k) |
+| `vtm-elf-method.png` | Method overview figure |
+| `README.md` | This card |
 
-That scale is the main reason generation quality is still rough in beta, together with the current model size. More data and a larger model should improve fidelity, identity consistency, and overall look over time.
-
-**Please treat outputs as experimental** — fine for testing pose drive and pipeline work, not as a finished production look.
+Runtime code for live camera → keypoints → inference is separate from this weight repo.
 
 ---
 
 ## Limitations
 
-- **Beta quality** — soft detail, identity drift, and artifacts can happen
-- **Small dataset** — limited character coverage and pose variety
-- **Windows + NVIDIA CUDA only** for now
-- Not a general-purpose text-to-image model — it is a **pose-driven VTuber renderer**
+- Beta: soft detail, identity drift, and pose errors are common  
+- Small data + DiT-30M capacity constrain image quality  
+- ~7 FPS on RTX 5060 Ti at 768 — not real-time 30 FPS yet  
+- Windows + CUDA only  
+- Not a general text-to-image model; pose + reference driven
 
 ---
 
 ## Intended use
 
-- Live / near-live VTuber pose driving
-- Research and prototyping of pose → character pipelines
-- Building on top of a VTuber-first backbone instead of generic SD pose tools
-
-**Not intended for:** photoreal people generation, production-final art, or non-CUDA / non-Windows setups (yet).
+Research and prototyping of VTuber pose → image pipelines (live drive, pose retarget experiments). Not a finished production renderer.
 
 ---
 
-## Citation / credit
-
-If you use VTM-Elf in a project, a link back to this repo is appreciated:
-
-[`https://huggingface.co/sinBoo1/VTM-Elf-0.01`](https://huggingface.co/sinBoo1/VTM-Elf-0.01)
+## BibTeX / link
 
----
-
-*Built from scratch for VTubers. Still early — quality will improve with more data and scale.*
+```
+https://huggingface.co/sinBoo1/VTM-Elf-0.01
+```