Create README.md

README.md

@@ -0,0 +1,58 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: diffusers
+---
+# BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing
+
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+Model card for BLIP-Diffusion, a text to image Diffusion model which enables zero-shot subject-driven generation and control-guided zero-shot generation. 
+
+The abstract from the paper is:
+
+*Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we introduce BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representation aligned with the text. Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions. Compared with previous methods such as DreamBooth, our model enables zero-shot subject-driven generation, and efficient fine-tuning for customized subject with up to 20x speedup. We also demonstrate that BLIP-Diffusion can be flexibly combined with existing techniques such as ControlNet and prompt-to-prompt to enable novel subject-driven generation and editing applications.*
+
+The model is created by Dongxu Li, Junnan Li, Steven C.H. Hoi.
+
+### Model Sources
+
+<!-- Provide the basic links for the model. -->
+
+- **Original Repository:** https://github.com/salesforce/LAVIS/tree/main
+- **Project Page:** https://dxli94.github.io/BLIP-Diffusion-website/
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+
+
+## Citation
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+If you find this repository useful in your research, please cite:
+
+```
+@misc{li2023blipdiffusion,
+      title={BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing}, 
+      author={Dongxu Li and Junnan Li and Steven C. H. Hoi},
+      year={2023},
+      eprint={2305.14720},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+