rzgar commited on
Commit
da49569
·
verified ·
1 Parent(s): 9398abd

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +59 -0
README.md CHANGED
@@ -1,3 +1,62 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ base_model:
4
+ - ByteDance/Bernini-R
5
+ tags:
6
+ - text-to-video
7
+ - image-to-video
8
+ - video-to-video
9
+ - image-to-image
10
+ - audio-to-video
11
+ - wan-s2v
12
+ - bernini-r-s2v
13
  ---
14
+
15
+ # Bernini-R-S2V
16
+
17
+ <video controls width="100%" height="480">
18
+ <source src="https://huggingface.co/rzgar/Bernini-R-S2V/resolve/main/video/ComfyUI__00002-audio.mp4" type="video/mp4">
19
+ Your browser does not support the video tag.
20
+ </video>
21
+
22
+ *Speech-driven video on Bernini-R , T2V, I2V, and V2V with lip-sync.*
23
+
24
+ This model adds single-speaker audio support to Bernini-R, so you can drive video with speech in text-to-video, image-to-video, and video-to-video setups.
25
+ It is not state-of-the-art audio-to-video, but it removes the need for post-processing or extra models just to add speech to Wan videos.
26
+ For basic talking-head work, or longer videos built from short sequences, it is a handy all-in-one option on top of Bernini's motion and editing strengths.
27
+
28
+
29
+ | File | Role |
30
+ |---|---|
31
+ | '[wan2.2_bernini_r_high_noise_int8_convrot_s2v.safetensors](https://huggingface.co/rzgar/Bernini-R-S2V/resolve/main/Bernini-R-S2V-int8-convrot/wan2.2_bernini_r_high_noise_int8_convrot_s2v.safetensors)' | High noise |
32
+ | '[wan2.2_bernini_r_low_noise_int8_convrot_s2v.safetensors](https://huggingface.co/rzgar/Bernini-R-S2V/resolve/main/Bernini-R-S2V-int8-convrot/wan2.2_bernini_r_low_noise_int8_convrot_s2v.safetensors)' | Low noise |
33
+
34
+ ComfyUI detects these as **`WAN22_S2V`** / 'WanModel_S2V' (audio keys trigger S2V model type).
35
+
36
+ ## Usage
37
+
38
+ 1. Download the checkpoints.
39
+ 2. Place in:
40
+ ```
41
+ ComfyUI/models/diffusion_models/
42
+ ```
43
+ 3. Add [wav2vec2](https://huggingface.co/rzgar/Bernini-R-S2V/resolve/main/audio_encoders/wav2vec2_large_english_fp16.safetensors) to:
44
+ ```
45
+ ComfyUI/models/audio_encoders/
46
+ ```
47
+ 4. Install **[ComfyUI-WanBerniniS2V](https://huggingface.co/rzgar/Bernini-R-S2V/tree/main/ComfyUI-WanBerniniS2V)** custom node:
48
+ ```
49
+ (Create a folder in 'ComfyUI/custom_nodes/' named 'ComfyUI-WanBerniniS2V', then save the Python files in that folder.)
50
+ ```
51
+ 5. Restart ComfyUI.
52
+ 6. Search for the Bernini S2V Conditioning node or use the example [workfllow](https://huggingface.co/rzgar/Bernini-R-S2V/blob/main/Workflow/Bernini-R-S2V_Workflow.json)
53
+
54
+ ## Audio tips
55
+
56
+ | Setting | Recommendation |
57
+ |---|---|
58
+ | **Channels** | **Mono**, wav2vec2 downmixes stereo internally |
59
+ | **Sample rate** | 44.1 khz or 48 khz (resampled to 16 kHz) |
60
+ | **Content** | Clear speech, less background music = better sync |
61
+ | **Length** | in my testing, max 9 to 15 seconds |
62
+