Image-Text-to-Text
Transformers
Safetensors
qwen2
text-generation
conversational
text-generation-inference
luzimu commited on
Commit
79960aa
·
verified ·
1 Parent(s): 8261ff9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +67 -3
README.md CHANGED
@@ -1,3 +1,67 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ ---
4
+
5
+ # WebGen-Agent
6
+
7
+ WebGen-Agent is an advanced website generation agent designed to autonomously create websites from natural language instructions. It was introduced in the paper [WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning](fig/WebGen_Agent.pdf).
8
+
9
+ ## Project Overview
10
+
11
+ WebGen-Agent combines state-of-the-art language models with specialized training techniques to create a powerful website generation tool. The agent can understand natural language instructions specifying appearance and functional requirements, iteratively generate website codebases, and refine them using visual and functional feedback.
12
+
13
+ ## Resources
14
+
15
+ Links to the data and model parameters are as follows:
16
+
17
+ | **Data** | **HF Link** |
18
+ |----------|------|
19
+ | **webgen-agent_train_sft** | 🤗 [luzimu/webgen-agent_train_sft](https://huggingface.co/datasets/luzimu/webgen-agent_train_sft) |
20
+ | **webgen-agent_train_step-grpo** | 🤗 [luzimu/webgen-agent_train_step-grpo](https://huggingface.co/datasets/luzimu/webgen-agent_train_step-grpo) |
21
+
22
+ | **Model** | **HF Link** |
23
+ |----------|------|
24
+ | **WebGenAgent-LM-7B-SFT** | 🤗 [luzimu/WebGenAgent-LM-7B-SFT](https://huggingface.co/luzimu/WebGenAgent-LM-7B-SFT) |
25
+ | **WebGenAgent-LM-7B-Step-GRPO** | 🤗 [luzimu/WebGenAgent-LM-7B-Step-GRPO](https://huggingface.co/luzimu/WebGenAgent-LM-7B-Step-GRPO) |
26
+ | **WebGenAgent-LM-8B-SFT** | 🤗 [luzimu/WebGenAgent-LM-8B-SFT](https://huggingface.co/luzimu/WebGenAgent-LM-8B-SFT) |
27
+ | **WebGenAgent-LM-8B-Step-GRPO** | 🤗 [luzimu/WebGenAgent-LM-8B-Step-GRPO](https://huggingface.co/luzimu/WebGenAgent-LM-8B-Step-GRPO) |
28
+
29
+ ## How WebGen-Agent Works
30
+
31
+ WebGen-Agent follows an iterative, multi-step paradigm for website generation:
32
+
33
+ 1. **Code Generation**: The agent generates code to create or edit website files based on natural language instructions
34
+ 2. **Code Execution**: Dependencies are installed and the website service is started
35
+ 3. **Feedback Gathering**:
36
+ - A screenshot of the website is captured
37
+ - A Visual Language Model (VLM) provides appearance feedback and scores
38
+ - A GUI-agent tests the website functionality and provides functional feedback
39
+ 4. **Refinement**: Based on the feedback, the agent continues to improve the website until it meets requirements
40
+
41
+ ![WebGen-Agent Workflow](fig/webgen-agent.png)
42
+
43
+ ## Step-GRPO with Screenshot and GUI-agent Feedback
44
+
45
+ The Step-GRPO with Screenshot and GUI-agent Feedback approach uses the screenshot and GUI-agent scores inherently produced in the WebGen-Agent workflow as step-level rewards:
46
+ - **Screenshot Score**: Quantifies the visual appeal and aesthetics of the website
47
+ - **GUI-agent Score**: Measures how well the website meets functional requirements
48
+
49
+ These dual rewards provide dense, reliable process supervision that significantly improves the model's ability to generate high-quality websites.
50
+
51
+ ![Step-GRPO with Screenshot and GUI-agent Feedback](fig/step-grpo.png)
52
+
53
+ ## Citation
54
+
55
+ If you find our project useful, please cite:
56
+
57
+ ```
58
+ @misc{lu2025webgenbenchevaluatingllmsgenerating,
59
+ title={WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch},
60
+ author={Zimu Lu and Yunqiao Yang and Houxing Ren and Haotian Hou and Han Xiao and Ke Wang and Weikang Shi and Aojun Zhou and Mingjie Zhan and Hongsheng Li},
61
+ year={2025},
62
+ eprint={2505.03733},
63
+ archivePrefix={arXiv},
64
+ primaryClass={cs.CL},
65
+ url={https://arxiv.org/abs/2505.03733},
66
+ }
67
+ ```