| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="description" content="PlanBench: A Comprehensive Benchmark for Evaluating Urban Planning Capabilities in Large Language Models"> |
| <meta name="keywords" content="PlanBench, PlanBench-V, Urban Planning, LLM Benchmark"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>PlanBench</title> |
|
|
| <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet"> |
| <link rel="stylesheet" href="./css/bulma.min.css"> |
| <link rel="stylesheet" href="./css/bulma-carousel.min.css"> |
| <link rel="stylesheet" href="./css/bulma-slider.min.css"> |
| <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0-beta3/css/all.min.css"> |
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> |
| <link rel="stylesheet" href="./css/index.css"> |
|
|
| <style> |
| .hidden { display: none; } |
| .sticky-menu { position: sticky; top: 4rem; } |
| p { margin: 0; } |
| |
| h1.maintitle { |
| font-size: 2.5rem !important; |
| font-weight: bold; |
| color: black; |
| } |
| .publication-links .link-block { |
| margin: 0 0.5rem; |
| display: inline-block; |
| } |
| |
| table { |
| border-collapse: collapse; |
| width: 100%; |
| margin-top: 5px; |
| border: 1px solid #ddd; |
| font-size: 14px; |
| } |
| th, td { text-align: left; padding: 8px; } |
| th { background-color: #f2f2f2; border-bottom: 2px solid #ddd; } |
| |
| .thanks-content { |
| display: flex; justify-content: space-between; align-items: center; |
| flex-wrap: wrap; gap: 20px; |
| } |
| .thanks-text { max-width: calc(100% - 120px); flex: 1 1 300px; } |
| .thanks-image img { width: 400px; height: auto; } |
| |
| @media screen and (max-width: 768px) { |
| h1.maintitle { font-size: 1.9rem !important; } |
| h2.publication-subtitle { font-size: 1.2rem !important; } |
| .content p, .content ul { font-size: 0.8rem; } |
| figcaption { font-size: 0.5rem !important; } |
| .thanks-image img { width: 100%; max-width: 100%; } |
| } |
| </style> |
| </head> |
|
|
| <body> |
| <section class="section"> |
| <div class="container"> |
| <div class="columns"> |
|
|
| |
| <div class="column is-2"> |
| <aside class="menu sticky-menu is-hidden-mobile"> |
| <p class="menu-label">Table of Contents</p> |
| <ul class="menu-list"> |
| <li><a href="#planbench-text">PlanBench (Text)</a></li> |
| <li><a href="#planbench-v">PlanBench-V (Vision)</a></li> |
| <li><a href="#results">Results</a></li> |
| </ul> |
| </aside> |
| </div> |
|
|
| |
| <div class="column is-10"> |
|
|
| |
| <section class="hero"> |
| <div class="hero-body"> |
| <div class="container is-max-desktop"> |
| <div class="columns is-centered"> |
| <div class="column has-text-centered"> |
| <h1 class="maintitle is-1 publication-title is-bold"> |
| <span style="vertical-align: middle">PlanBench: A Comprehensive Benchmark for Urban Planning</span> |
| </h1> |
| <div class="column has-text-centered" style="margin-top: 1rem;"> |
| <div class="publication-links"> |
| <span class="link-block"> |
| <a href="https://arxiv.org/abs/2402.19273" class="external-link button is-normal is-rounded is-dark"> |
| <span class="icon"><i class="fas fa-file-pdf"></i></span> |
| <span>arXiv</span> |
| </a> |
| </span> |
| <span class="link-block"> |
| <a href="https://github.com/zhuchichi56/PlanBench" class="external-link button is-normal is-rounded is-dark"> |
| <span class="icon"><i class="fab fa-github"></i></span> |
| <span>GitHub</span> |
| </a> |
| </span> |
| <span class="link-block"> |
| <a href="https://huggingface.co/datasets/chichi56/PlanBench" class="external-link button is-normal is-rounded is-dark"> |
| <span>🤗 Dataset</span> |
| </a> |
| </span> |
| <span class="link-block"> |
| <a href="https://plangpt.github.io/" class="external-link button is-normal is-rounded is-dark"> |
| <span>🌐 PlanGPT Blog</span> |
| </a> |
| </span> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </section> |
|
|
|
|
| |
| <section class="hero is-light is-small"> |
| <div class="hero-body has-text-centered"> |
| <h1 class="title is-1" id="planbench-text"> |
| <span style="vertical-align: middle">PlanBench: Planning Knowledge Benchmark</span> |
| </h1> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container" style="margin-bottom: 2vh;"> |
| <div class="columns is-centered"> |
| <div class="column is-four-fifths has-text-centered"> |
| <h2 class="subtitle is-3 publication-subtitle" style="color: black; font-weight: bold;"> |
| A Comprehensive Benchmark for Evaluating Urban Planning Capabilities in Large Language Models |
| </h2> |
| <div class="column has-text-centered"> |
| <div class="publication-links"> |
| <span class="link-block"> |
| <a href="https://behavioral-spatial-ai-lab.github.io/plangpt-bench/" class="external-link button is-normal is-rounded is-dark"> |
| <span>🔗Homepage</span> |
| </a> |
| </span> |
| </div> |
| </div> |
| <h3 class="title is-4">Abstract</h3> |
| <div class="content has-text-justified" style="line-height: 2;"> |
| <p> |
| Urban planning, as a highly interdisciplinary and practice-oriented field, requires not only simple recall of knowledge but also complex situational judgment, policy understanding, spatial logical reasoning, and value assessment. Planning texts are characterized by dense terminology, complex structures, and long reasoning chains. Constructing benchmarks can help enhance large models' planning adaptation capabilities in the following aspects: |
| </p> |
| <ul> |
| <li>Deconstruction of planning texts (e.g., regulation breakdown, indicator interpretation)</li> |
| <li>Multi-level spatial governance logic (national - city - community)</li> |
| <li>Situational policy judgment and plan generation (e.g., site selection, land allocation, industry recommendations)</li> |
| </ul> |
| <p> |
| Text-based benchmarks serve as the linguistic foundation for "multimodal urban intelligence." In subsequent integrations with maps, charts, and spatial models, text comprehension capabilities are fundamental for achieving the three-dimensional linkage of "text-image-policy." |
| </p> |
| </div> |
| <h3 class="title is-4">Architecture</h3> |
| <div class="content has-text-justified"> |
| <img src="./pictures/planbench_text_show.png" alt="PlanBench-Text Architecture" width="95%" style="margin:0 auto; display:block;"> |
| <br> |
| <figcaption> |
| <p style="text-align: center;"><b>Figure 1:</b> PlanBench Text Benchmark Architecture.</p> |
| </figcaption> |
| </div> |
| <div class="has-text-right mt-4"> |
| <p class="is-size-6 has-text-grey is-italic">📅 Release Date: May 19, 2025</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </section> |
|
|
|
|
| |
| <section class="hero is-light is-small"> |
| <div class="hero-body has-text-centered"> |
| <h1 class="title is-1" id="planbench-v"> |
| <span style="vertical-align: middle">PlanBench-V: Planning Visual Recognition Benchmark</span> |
| </h1> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container" style="margin-bottom: 2vh;"> |
| <div class="columns is-centered"> |
| <div class="column is-four-fifths has-text-centered"> |
| <h2 class="subtitle is-3 publication-subtitle" style="color: black; font-weight: bold;"> |
| Multimodal Multi-image Understanding for Evaluating Multimodal Large Language Models |
| </h2> |
| <div class="column has-text-centered"> |
| <div class="publication-links"> |
| <span class="link-block"> |
| <a href="https://behavioral-spatial-ai-lab.github.io/planvlm-bench/" class="external-link button is-normal is-rounded is-dark"> |
| <span>🔗Homepage</span> |
| </a> |
| </span> |
| </div> |
| </div> |
| <h3 class="title is-4">Abstract</h3> |
| <div class="content has-text-justified" style="line-height: 2;"> |
| <p> |
| National spatial planning maps visually present the concepts, goals, strategies, and specific measures of spatial planning, serving as a guide for coordinating various spatial development, protection, and utilization activities. They are not only crucial for planning decisions but also important tools for public participation and oversight of planning implementation. Planning is a highly interdisciplinary and specialized task; understanding planning maps requires grasping detailed elements (symbols, legends, geographic features) and the ability to conduct comprehensive analysis and judgment in conjunction with policies. This complexity makes understanding planning maps challenging. With the rapid development of multimodal large language models (MLLMs), we have established a benchmark for national spatial planning maps to evaluate MLLMs' capabilities in understanding these maps. Our contributions are as follows:<br><br> |
| <b>(1) Data:</b> We constructed the Spatial Planning Map Database (SPMD), featuring diverse image content and high-quality annotations provided by experts in the field of planning.<br> |
| <b>(2) Framework:</b> We proposed a comprehensive framework based on planning disciplines, measuring MLLMs' understanding of planning maps from four perspectives: perception, reasoning, association, and application, including eight subcategories.<br> |
| <b>(3) Experiments:</b> By constructing question-answer tasks based on authoritative question banks (China's Registered Urban Planner Qualification Examination), we significantly reduced the proportion of "hallucination-style normative citations" by models.<br> |
| <b>(4) Results:</b> All models performed worst in the application dimension, with Qwen2.5-VL-32B-Instruct achieving the highest overall score across all four dimensions. |
| </p> |
| </div> |
| <h3 class="title is-4">Architecture</h3> |
| <div class="content has-text-justified"> |
| <img src="./pictures/VLMbench250512.png" alt="PlanBench-V Architecture" width="95%" style="margin:0 auto; display:block;"> |
| <br> |
| <figcaption> |
| <p style="text-align: center;"><b>Figure 2:</b> PlanBench-V Architecture.</p> |
| </figcaption> |
| </div> |
| <div class="has-text-right mt-4"> |
| <p class="is-size-6 has-text-grey is-italic">📅 Release Date: May 19, 2025</p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </section> |
|
|
|
|
| |
| <section class="hero is-light is-small"> |
| <div class="hero-body has-text-centered"> |
| <h1 class="title is-1" id="results"> |
| <span style="vertical-align: middle">Benchmark Results</span> |
| </h1> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container" style="margin-bottom: 2vh;"> |
| <div class="columns is-centered"> |
| <div class="column is-four-fifths"> |
|
|
| <h3 class="title is-4">PlanBench-V Results (Vision, Judge: gpt-4o-mini, 300 items)</h3> |
| <div class="table-container"> |
| <table class="table is-bordered is-striped is-hoverable is-fullwidth"> |
| <thead> |
| <tr> |
| <th>Rank</th><th>Model</th><th>Overall</th> |
| <th>描述</th><th>类型</th><th>评价</th><th>决策</th> |
| <th>专业推理</th><th>关联</th><th>空间关系</th><th>要素</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr><td>🥇</td><td><b>gemini-2.5-pro</b></td><td><b>1.472/2 (73.6%)</b></td><td>1.775</td><td>1.656</td><td>1.439</td><td>1.525</td><td>1.425</td><td>1.468</td><td>1.444</td><td>1.408</td></tr> |
| <tr><td>🥈</td><td><b>gpt-5.4</b></td><td><b>1.431/2 (71.6%)</b></td><td>1.900</td><td>1.562</td><td>1.586</td><td>1.508</td><td>1.486</td><td>1.438</td><td>1.383</td><td>1.233</td></tr> |
| <tr><td>🥉</td><td><b>claude-opus-4.7</b></td><td><b>1.384/2 (69.2%)</b></td><td>1.825</td><td>1.320</td><td>1.434</td><td>1.321</td><td>1.558</td><td>1.493</td><td>1.295</td><td>1.186</td></tr> |
| <tr><td>4</td><td>gpt-4o-mini</td><td>1.084/2 (54.2%)</td><td>1.244</td><td>1.342</td><td>0.901</td><td>1.155</td><td>1.079</td><td>1.110</td><td>1.151</td><td>0.918</td></tr> |
| </tbody> |
| </table> |
| </div> |
|
|
| <br> |
|
|
| <h3 class="title is-4">PlanBench Results (Text, Judge: gpt-4o-mini, 405 items)</h3> |
| <div class="table-container"> |
| <table class="table is-bordered is-striped is-hoverable is-fullwidth"> |
| <thead> |
| <tr> |
| <th>Rank</th><th>Model</th><th>Score</th> |
| <th>Remember</th><th>Understand</th><th>Apply</th><th>Analyze</th><th>Evaluate</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr><td>1</td><td><b>Qwen3-32B</b></td><td><b>80.9%</b></td><td>97.5</td><td>86.4</td><td>95.1</td><td>86.1</td><td>39.5</td></tr> |
| <tr><td>2</td><td><b>Qwen3-14B</b></td><td><b>80.6%</b></td><td>97.5</td><td>77.8</td><td>92.6</td><td>86.8</td><td>48.1</td></tr> |
| <tr><td>3</td><td><b>QwQ-32B</b></td><td><b>80.4%</b></td><td>95.1</td><td>85.2</td><td>91.4</td><td>91.9</td><td>38.3</td></tr> |
| <tr><td>4</td><td>Qwen3-8B</td><td>80.0%</td><td>93.8</td><td>80.2</td><td>90.1</td><td>90.4</td><td>45.7</td></tr> |
| <tr><td>5</td><td>Qwen3-4B</td><td>78.8%</td><td>95.1</td><td>72.8</td><td>90.1</td><td>89.3</td><td>46.9</td></tr> |
| <tr><td>6</td><td>Qwen3-30B-A3B</td><td>78.4%</td><td>97.5</td><td>79.0</td><td>88.9</td><td>89.5</td><td>37.0</td></tr> |
| <tr><td>7</td><td>Qwen3-1.7B</td><td>74.1%</td><td>95.1</td><td>79.0</td><td>76.5</td><td>85.1</td><td>34.6</td></tr> |
| <tr><td>8</td><td>glm-4-9b-chat</td><td>73.3%</td><td>91.4</td><td>72.8</td><td>84.0</td><td>79.9</td><td>38.3</td></tr> |
| <tr><td>9</td><td>Meta-Llama-3-8B-Instruct</td><td>70.6%</td><td>95.1</td><td>58.0</td><td>72.8</td><td>78.8</td><td>48.1</td></tr> |
| <tr><td>10</td><td>Qwen2.5-3B-Instruct</td><td>70.3%</td><td>98.8</td><td>66.7</td><td>92.6</td><td>64.0</td><td>29.6</td></tr> |
| <tr><td>11</td><td>Qwen2.5-7B-Instruct</td><td>69.5%</td><td>98.8</td><td>70.4</td><td>81.5</td><td>65.9</td><td>30.9</td></tr> |
| <tr><td>12</td><td>Qwen2-VL-7B-Instruct</td><td>68.2%</td><td>93.8</td><td>65.4</td><td>76.5</td><td>65.7</td><td>39.5</td></tr> |
| <tr><td>13</td><td>DeepSeek-R1-Distill-Llama-8B</td><td>68.1%</td><td>93.8</td><td>64.2</td><td>75.3</td><td>78.8</td><td>28.4</td></tr> |
| <tr><td>14</td><td>DeepSeek-R1-Distill-Qwen-7B</td><td>68.0%</td><td>96.3</td><td>69.1</td><td>77.8</td><td>73.4</td><td>23.5</td></tr> |
| <tr><td>15</td><td>Qwen3-0.6B</td><td>55.9%</td><td>90.1</td><td>55.6</td><td>46.9</td><td>74.8</td><td>12.3</td></tr> |
| <tr><td>16</td><td>Llama-3.1-Tulu-3-8B</td><td>49.0%</td><td>60.5</td><td>56.8</td><td>30.9</td><td>80.8</td><td>16.0</td></tr> |
| <tr><td>17</td><td>chatglm3-6b</td><td>48.3%</td><td>80.2</td><td>37.5</td><td>44.4</td><td>58.3</td><td>21.0</td></tr> |
| <tr><td>18</td><td>Qwen2.5-0.5B-Instruct</td><td>39.3%</td><td>65.4</td><td>21.0</td><td>25.9</td><td>69.4</td><td>14.8</td></tr> |
| </tbody> |
| </table> |
| </div> |
|
|
| </div> |
| </div> |
| </div> |
| </section> |
|
|
|
|
| |
| <section class="section" id="BibTeX"> |
| <div class="container is-max-desktop content"> |
| <h2 class="title is-3 has-text-centered">Citation</h2> |
| <pre><code>@misc{zhu2024plangptenhancingurbanplanning, |
| title={PlanGPT: Enhancing Urban Planning with Tailored Language Model and Efficient Retrieval}, |
| author={He Zhu and Wenjia Zhang and Nuoxian Huang and Boyang Li and Luyao Niu and Zipei Fan and Tianle Lun and Yicheng Tao and Junyou Su and Zhaoya Gong and Chenyu Fang and Xing Liu}, |
| year={2024}, |
| eprint={2402.19273}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2402.19273}, |
| } |
|
|
| @misc{deng2025urban, |
| title={Urban Planning Bench: A Comprehensive Benchmark for Evaluating Urban Planning Capabilities in Large Language Models}, |
| author={Yijie Deng and He Zhu and Wen Wang and Minxin Chen and Junyou Su and Wenjia Zhang}, |
| year={2025}, |
| institution={Behavioral and Spatial AI Lab, Tongji University}, |
| }</code></pre> |
| </div> |
| </section> |
|
|
|
|
| |
| <div class="container mt-6"> |
| <div class="content"> |
| <h3 class="title is-4">Members</h3> |
| <div class="thanks-content" style="width: 100%;"> |
| <div class="thanks-text" style="line-height: 3;"> |
| <p style="font-size: 16px;"> |
| He Zhu, Minxin Chen, Yijie Deng, Junyou Su, Wen Wang, Yurun Wang, Yulin Wu, Caicheng Niu, Tianhua Lu, Chengcheng Liu, Boyang Li, Nuoxian Huang, Ying'er Cai, Yue Wei, Sizheng Yang, Luyao Niu, Jiayu Gu, Yuhan Zou, Fenghong An, Siqi Cha, Chuang Deng, Hanying Li, Hongzhou Zheng and Qi Wang. |
| </p> |
| </div> |
| </div> |
| <div class="image-container" style="display: flex; flex-wrap: wrap; justify-content: center; gap: 30px; align-items: center;"> |
| <div class="thanks-image" style="flex: 1 1 200px; max-width: 300px;"> |
| <img src="./pictures/BSAI.png" alt="BSAI" style="width: 100%; height: auto; object-fit: contain;"> |
| </div> |
| <div class="thanks-image" style="flex: 1 1 200px; max-width: 300px;"> |
| <img src="./pictures/caup.png" alt="CAUP" style="width: 100%; height: auto; object-fit: contain;"> |
| </div> |
| </div> |
| </div> |
| </div> |
|
|
| </div> |
| </div> |
| </div> |
| </section> |
| </body> |
| </html> |
|
|
