ysmao commited on
Commit
ba150fd
·
verified ·
1 Parent(s): f4039db

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +41 -1
README.md CHANGED
@@ -51,7 +51,7 @@ base_model: black-forest-labs/FLUX.1-dev
51
  instance_prompt: null
52
  license: mit
53
  ---
54
- # twoview-incontext-beta
55
 
56
  <Gallery />
57
 
@@ -59,6 +59,46 @@ license: mit
59
 
60
  Inspired by [In-Context-LoRA](https:&#x2F;&#x2F;github.com&#x2F;ali-vilab&#x2F;In-Context-LoRA), this project aims to generate multi-view images of the same scene or object simultaneously. By using flux with the multiview-incontext-lora, we can divide the images into portions to obtain novel views.
61
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
 
63
  ## Download model
64
 
 
51
  instance_prompt: null
52
  license: mit
53
  ---
54
+ # MultiView-InContext-Lora
55
 
56
  <Gallery />
57
 
 
59
 
60
  Inspired by [In-Context-LoRA](https:&#x2F;&#x2F;github.com&#x2F;ali-vilab&#x2F;In-Context-LoRA), this project aims to generate multi-view images of the same scene or object simultaneously. By using flux with the multiview-incontext-lora, we can divide the images into portions to obtain novel views.
61
 
62
+ > **_NOTE:_** This is a beta release of the model. The consistency between views may not be perfect, and the model might sometimes generate views that don't perfectly align or maintain exact object positions across viewpoints. I am working on improving the geometric consistency and spatial relationships between generated views.
63
+
64
+
65
+ ## Roadmap
66
+
67
+ - [ ] 🔄 Improve the consistency between the two-view images.
68
+ - [ ] 4️⃣ Generate 4 views of a scene in a grid format.
69
+ - [ ] 🧸 Generate 4 canonical coordinates view points of a single object in a grid format.
70
+ - [ ] 🏛️ 3D reconstruction from multi-view images.
71
+
72
+ ## Inference
73
+
74
+ ```python
75
+ import torch
76
+ from diffusers import FluxPipeline
77
+
78
+ pipeline = FluxPipeline.from_pretrained(
79
+ "black-forest-labs/FLUX.1-dev",
80
+ torch_dtype=torch.bfloat16,
81
+ )
82
+ pipeline.load_lora_weights(
83
+ "ysmao/multiview-incontext",
84
+ weight_name="twoview-incontext-b01.safetensors",
85
+ )
86
+ pipeline.fuse_lora()
87
+
88
+ scene_prompt = "a living room with a sofa set with cushions, side tables with table lamps, a flat screen television on a table, houseplants, wall hangings, electric lights, and a carpet on the floor"
89
+ prompt = f"[TWO-VIEWS] This set of two images presents a scene from two different viewpoints. [IMAGE1] The first image shows {scene_prompt}. [IMAGE2] The second image shows the same room but in another viewpoint."
90
+ image_height = 576
91
+ image_width = 864
92
+ output = pipeline(
93
+ prompt=prompt,
94
+ height=int(image_height),
95
+ width=int(image_width * 2),
96
+ num_inference_steps=30,
97
+ guidance_scale=3.5,
98
+ ).images[0]
99
+
100
+ output.save("twoview-incontext-beta.png")
101
+ ```
102
 
103
  ## Download model
104