Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,77 @@
+# InterleavedBench
+
+This is the official huggingface repo for the work "Holistic Evaluation for Interleaved Text-and-Image Generation". This is a research preview. More details, including the baseline models' prediction and so on, will be coming soon in the following few weeks.
+
+
+## How to use InterleavedBench
+
+### Repo hierarchy
+
+- `interleaved_bench.json` is the main json file of the dataset. 
+- `zipped_images` is the directory of zipped images for each subset, including the images for the context and ground truths.
+- `src/interleavedeval_gpt4o.py` is the python script for InterleavedEval with GPT-4o. Its input is the model prediction file.
+
+### To get started
+
+- unzip the images files under `zipped_images`.
+- Run the inference on `interleaved_bench.json` with your model and get your model output (including text and image).
+- Use the script in `src/interleavedeval_gpt4o.py` to perform evaluation.
+
+
+One example in `interleaved_bench.json` is as follows:
+
+```
+{
+    "id": "wikihow_next_step_0_489157",
+    "image": [
+        "wiki_images_test/489157_0_0.png",
+        "wiki_images_test/489157_0_1.png",
+        "wiki_images_test/489157_0_2.png",
+        "wiki_images_test/489157_0_3.png",
+        "wiki_images_test/489157_0_4.png"
+    ],
+    "task_name": "wikihow_next_step",
+    "conversations": [
+        {
+            "from": "human",
+            "value": "In this task, you are given a high-level goal 'How to Make a Banana Shake': Banana shakes are a tasty way to get a lot of nutrients all at once. Bananas provide a creamy, smooth texture when turned into a drink. Bananas also fill empty stomachs, staving off hunger pangs and giving you a nice energy burst. In this article you'll find a few ways to make banana shakes, among the many possibilities. \n  You need to assist human user to complete this task via making a banana shake with kefir. Given the previous steps, you need to predict the subsequent 4 steps to help the user to finish the task. The previous steps are: \n <BEGIN>  Put 2 to 3 bananas in a bowl. <image>\n"
+        },
+        {
+            "from": "gpt",
+            "value": "Now put in a liter of kefir and a teaspoon of sugar. <image>\n Put 1 cup milk into the mix. <image>\n Using a blender, blend all ingredients together. <image>\n Relax with your fresh banana smoothie! <image>\n"
+        }
+    ],
+    "goal": "How to Make a Banana Shake",
+    "category": [
+        "Food and Entertaining",
+        "Drinks",
+        "Smoothies Shakes and Milk",
+        "Fruit Based Shakes"
+    ],
+    "dataset_id": "wikihow_selected_test_uni"
+},
+```
+
+### Reference
+
+If you find our work useful or interesting, please cite:
+```
+@article{liu_holistic_2024,
+  author       = {Minqian Liu and
+                  Zhiyang Xu and
+                  Zihao Lin and
+                  Trevor Ashby and
+                  Joy Rimchala and
+                  Jiaxin Zhang and
+                  Lifu Huang},
+  title        = {Holistic Evaluation for Interleaved Text-and-Image Generation},
+  journal      = {CoRR},
+  volume       = {abs/2406.14643},
+  year         = {2024},
+  url          = {https://doi.org/10.48550/arXiv.2406.14643},
+  doi          = {10.48550/ARXIV.2406.14643},
+  eprinttype    = {arXiv},
+  eprint       = {2406.14643},
+  timestamp    = {Tue, 16 Jul 2024 16:17:50 +0200}
+}
+```