thedeoxen commited on
Commit
b4f9c29
Β·
verified Β·
1 Parent(s): 7ac84c4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +115 -38
README.md CHANGED
@@ -1,52 +1,129 @@
1
  ---
 
 
 
 
2
  tags:
3
- - text-to-image
 
 
 
 
 
 
 
 
 
4
  - lora
5
- - diffusers
6
- - template:diffusion-lora
7
- widget:
8
- - output:
9
- url: images/ComfyUI_00165_.png
10
- text: '-'
11
- - output:
12
- url: images/ComfyUI_00166_.png
13
- text: '-'
14
- - output:
15
- url: images/ComfyUI_00167_.png
16
- text: '-'
17
- - output:
18
- url: images/ComfyUI_00170_.png
19
- text: '-'
20
- - output:
21
- url: images/ComfyUI_00183_.png
22
- text: '-'
23
- - output:
24
- url: images/ComfyUI_00198_.png
25
- text: '-'
26
- - output:
27
- url: images/ComfyUI_00211_.png
28
- text: '-'
29
- - output:
30
- url: images/ComfyUI_00217_.png
31
- text: '-'
32
- - output:
33
- url: images/ComfyUI_00219_.png
34
- text: '-'
35
- base_model: black-forest-labs/FLUX.2-klein-base-9B
36
- instance_prompt: refcontrol
37
- license: apache-2.0
38
  ---
 
39
  # RefControl FLUX.2 Klein 9B – Reference Pose LoRA
40
 
 
 
 
 
 
 
41
  <Gallery />
42
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
43
 
44
- ## Trigger words
 
45
 
46
- You should use `refcontrol` to trigger the image generation.
47
 
 
 
 
48
 
49
- ## Download model
50
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
51
 
52
- [Download](/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora/tree/main) them in the Files & versions tab.
 
 
 
 
1
  ---
2
+ license: apache-2.0
3
+ base_model:
4
+ - black-forest-labs/FLUX.2-klein-base-9B
5
+ pipeline_tag: image-to-image
6
  tags:
7
+ - flux
8
+ - flux2
9
+ - klein
10
+ - pose
11
+ - openpose
12
+ - controlnet
13
+ - refcontrol
14
+ - img2img
15
+ - image
16
+ - editing
17
  - lora
18
+ library_name: diffusers
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  ---
20
+
21
  # RefControl FLUX.2 Klein 9B – Reference Pose LoRA
22
 
23
+ ## πŸ“ Short description
24
+ A LoRA for **FLUX.2 Klein 9B Base** that fuses a **reference image** (identity) with a **pose control map** (skeleton / keypoints).
25
+ It preserves **identity and style** from the reference while following the **body pose and structure** from the pose map.
26
+
27
+ **Trigger word:** `refcontrol`
28
+
29
  <Gallery />
30
 
31
+ ---
32
+
33
+ ## πŸ“Š Examples
34
+
35
+ Each preview is a **single combined image** from ComfyUI: **Pose β†’ Reference β†’ Result** (left to right).
36
+
37
+ <table>
38
+ <thead>
39
+ <tr>
40
+ <th>Pose β†’ Reference β†’ Result</th>
41
+ </tr>
42
+ </thead>
43
+ <tbody>
44
+ <tr>
45
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora/resolve/main/images/ComfyUI_00165_.png" width="960" /></td>
46
+ </tr>
47
+ <tr>
48
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora/resolve/main/images/ComfyUI_00166_.png" width="960" /></td>
49
+ </tr>
50
+ <tr>
51
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora/resolve/main/images/ComfyUI_00167_.png" width="960" /></td>
52
+ </tr>
53
+ <tr>
54
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora/resolve/main/images/ComfyUI_00170_.png" width="960" /></td>
55
+ </tr>
56
+ <tr>
57
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora/resolve/main/images/ComfyUI_00183_.png" width="960" /></td>
58
+ </tr>
59
+ <tr>
60
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora/resolve/main/images/ComfyUI_00211_.png" width="960" /></td>
61
+ </tr>
62
+ <tr>
63
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora/resolve/main/images/ComfyUI_00217_.png" width="960" /></td>
64
+ </tr>
65
+ <tr>
66
+ <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora/resolve/main/images/ComfyUI_00219_.png" width="960" /></td>
67
+ </tr>
68
+ </tbody>
69
+ </table>
70
+ ---
71
+
72
+ ## πŸ“– Extended description
73
+ This LoRA was primarily trained on **humans**, but it also works with **stylized characters** and some objects.
74
+ Its main purpose is to **preserve identity** β€” facial features, hairstyle, clothing, or object details β€” from the reference image, while adapting the subject to the **pose and skeleton structure** defined by the control map.
75
+
76
+ Part of the **RefControl** family: reference + control fusion for consistent, controllable generation on **FLUX.2 Klein 9B Base**.
77
+
78
+ ---
79
+
80
+ ## βš™οΈ How to use
81
+ 1. Use the **first image** as the **pose control map** (skeleton / keypoints).
82
+ 2. Use the **second image** as the **reference** (character, person, or object).
83
+ 3. Add the trigger word **`refcontrol`** in your prompt.
84
+ 4. Adjust LoRA weight (recommended **0.8–1.0**) depending on how strongly you want to preserve identity.
85
 
86
+ ## ComfyUI requirenments
87
+ For extracting pose you need to install nodes with pose estimation in workflow used:
88
 
89
+ https://github.com/Fannovel16/comfyui_controlnet_aux
90
 
91
+ ### Base model
92
+ Use with [`black-forest-labs/FLUX.2-klein-base-9B`](https://huggingface.co/black-forest-labs/FLUX.2-klein-base-9B).
93
+ The undistilled Base variant is intended for LoRA training and custom pipelines (~50 inference steps, `guidance_scale` ~4.0).
94
 
95
+ ### βœ… Example prompt
96
 
97
+ ```
98
+ refcontrol
99
+ ```
100
+
101
+ ---
102
+
103
+ ## 🎯 What it does
104
+ - Preserves **character or object identity** across generations.
105
+ - Adapts the subject to a **new pose or action** defined by the skeleton map.
106
+ - Works best when the pose map has **similar proportions and scale** to the reference.
107
+
108
+ ---
109
+
110
+ ## ⚑ Tips
111
+ - Best results when the pose map is not drastically different in body scale or framing from the reference.
112
+ - Combine with **text prompts** to refine background, lighting, or mood.
113
+ - Can be combined with other **RefControl** LoRAs (depth, lineart, canny) on compatible pipelines for multi-constraint generation.
114
+ - OpenPose / DWPose skeleton maps on a **black background** work well as control input.
115
+
116
+ ---
117
+
118
+ ## πŸ“Œ Use cases
119
+ - Character posing for illustrations, comics, or storyboards.
120
+ - Consistent **character design** across different poses.
121
+ - Re-posing stylized characters while keeping their identity.
122
+ - Creating animation keyframes from static references.
123
+
124
+ ---
125
 
126
+ ## πŸ“¦ Files
127
+ - **Weights:** [`flux2_klein_9b_refcontrol_pose.safetensors`](https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora/blob/main/flux2_klein_9b_refcontrol_pose.safetensors)
128
+ - **ComfyUI workflow:** [`image_flux2_klein_image_edit_9b_refcontrol_pose.json`](https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora/blob/main/image_flux2_klein_image_edit_9b_refcontrol_pose.json)
129
+ - **Repo:** [thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora](https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-pose-lora)