Upload folder using huggingface_hub

agent.md

@@ -0,0 +1,60 @@
+# Agent Notes
+
+- Use the `edit` conda environment for ChartPipeline commands:
+  `/data/wangzheng/miniconda3/envs/edit/bin/python`.
+- When invoking Python commands directly, prefer:
+  `PYTHONPATH=. /data/wangzheng/miniconda3/envs/edit/bin/python ...`.
+- For rendering, set:
+  `PUPPETEER_EXECUTABLE_PATH=/usr/bin/google-chrome` and `RENDER_LONGEST_SIDE=1600`.
+- For full pipeline runs in a non-login shell, include both runtime dirs in PATH:
+  `export PATH=/data/wangzheng/miniconda3/envs/edit/bin:/data/wangzheng/miniconda3/envs/edit/lib/python3.12/site-packages/playwright/driver:$PATH`.
+- The post-generation icon/image module is `modules.asset_regenerator.asset_regenerator`.
+  It runs after `infographics_generator`, builds one contact sheet containing the full infographic
+  plus all selected SVG `<image>` assets, calls `gpt-image-2`, chroma-keys the generated cells,
+  and replaces only the original SVG `<image href>` / `xlink:href` values with data URI PNGs.
+- During variation repair, do not call `gpt-image-2` while iterating on template/data fixes.
+  First regenerate with `--modules infographics_generator` only, or run asset regeneration with
+  `--dry-run` if candidate discovery needs checking. Run the real `asset_regenerator` only after
+  the template-side repair passes structural/rendered validation.
+- Keep before/after evidence for each repaired variation. Preserve the original sample SVG/PNG or
+  render a baseline from the unmodified template, then store side-by-side comparison artifacts
+  under an `output/chart_repair_comparison_*` directory.
+- Keep complete code backups for every full variation-repair or style-reference iteration. Before
+  editing a variation, create a timestamped backup directory under `backups/` and include every
+  variation template plus every support file that may be changed in the iteration, such as docs,
+  agent notes, pipeline modules, generator utilities, asset-regenerator code, and helper scripts.
+- Each complete iteration backup must keep three separate trees:
+  - `before_head`: the original repair-before version from `HEAD:<path>` for tracked files, or a
+    missing marker for files added during repair work.
+  - `before`: the exact workspace state immediately before the iteration edits begin.
+  - `after`: the exact workspace state after patching, regenerating, verifying, and documenting the
+    iteration.
+- Never overwrite an older backup tree. If a file was already modified before the next iteration,
+  preserve that pre-edit copy in `before` so the new iteration can be separated from earlier repairs.
+  For untracked support files, write a missing marker in `before_head`, copy the pre-edit file into
+  `before`, and copy the final file into `after`.
+- Every backup directory must include `files.tsv`, `manifest.md`, pre/post `git status --short`,
+  relevant `git diff --stat`, generated output paths, comparison evidence paths, verification
+  evidence paths, and whether `gpt-image-2` or `asset_regenerator` was called. The manifest must make
+  clear which tree is the repair-before code and which tree is the current/final code.
+- Run it in the pipeline with:
+  `--modules infographics_generator asset_regenerator`.
+- To test without calling the image model:
+  `PYTHONPATH=. /data/wangzheng/miniconda3/envs/edit/bin/python -m modules.asset_regenerator.asset_regenerator --svg <final.svg> --png <final.png> --dry-run`.
+- The image edit call defaults to `--model gpt-image-2 --input-fidelity high` so the model sees
+  the rendered infographic as context while only regenerating the green contact-sheet asset cells.
+- Keep this module scoped to decorative icon/image assets. Do not use it to raster-overwrite the
+  chart, title, labels, axes, legends, data values, chart marks, or background unless explicitly
+  passing `--include-background` for a controlled experiment.
+- For style-reference iterations, compare the current baseline for a variation against existing
+  PNGs under `../chart_template_samples_20260527_083945/gpt_image_2_improved/<variation>`. If the
+  variation was repaired in `docs/variation_repair_status.md`, use the repaired output as the
+  baseline; otherwise use `../data/all_variations_check/template_variations_20260527/<variation>`.
+  Choose one simple, readable, low-decoration GPT-improved reference as the style anchor, but do
+  not copy its chart geometry or numeric values when they conflict with source data.
+- During style-reference iterations, do not call `gpt-image-2`; existing GPT-improved PNGs are
+  references only. Keep title/icon/image generation on the existing pipeline path for now.
+- For every style-reference iteration, keep a timestamped code backup under `backups/`, regenerate
+  5 outputs, write a baseline/reference/after comparison under `output/style_ref_comparison_*`,
+  verify chart type and data mappings structurally, and record the iteration in
+  `docs/style_reference_repair_status.md`.

allowed_chart_types.json

@@ -0,0 +1,17 @@
+{
+  "chart_types": [
+    "donut chart",
+    "pie chart",
+    "multiple donut chart",
+    "multiple pie chart",
+    "line graph",
+    "multiple line graph",
+    "spline graph",
+    "multiple spline graph",
+    "step line graph",
+    "stepped line graph",
+    "multiple step line graph",
+    "sorted lines chart",
+    "slope chart"
+  ]
+}

allowed_chart_types.json.example

@@ -0,0 +1,19 @@
+{
+  "_comment": "Whitelist of chart_type names the infographics_generator is allowed to pick from.",
+  "_usage": [
+    "1) Copy this file to allowed_chart_types.json (same directory), OR",
+    "   set ALLOWED_CHART_TYPES_FILE=<path> to point at a custom location.",
+    "2) Edit the 'chart_types' list below to the chart_type values you care about.",
+    "3) Empty list / missing file means 'no filtering' (legacy behaviour).",
+    "Granularity is chart_type (NOT chart_name). All chart_name variations whose",
+    "directory lives under the listed chart_type folder will become selectable.",
+    "Run `awk -F' / ' '{print $1}' template_list.txt | sort -u` to see the full set."
+  ],
+  "chart_types": [
+    "bump chart",
+    "bar chart",
+    "line graph",
+    "bubble chart",
+    "scatterplot"
+  ]
+}

batch_run.py

@@ -0,0 +1,28 @@
+import json
+import subprocess
+from multiprocessing import Pool
+from tqdm import tqdm
+
+def run_command(item):
+    data_source = item["data_source"]
+    chart_variation = item["chart_variation"]
+    
+    cmd = [
+        "python", "pipeline.py",
+        "--modules", "infographics_generator",
+        "--input", data_source,
+        "--output", "./test_output_new_new",
+        "--chart-name", chart_variation
+    ]
+    
+    subprocess.run(cmd)
+    return item
+
+start_id = 0
+
+with open("extracted_info.json", "r") as f:
+    data = json.load(f)[start_id:start_id+1]
+
+with Pool(processes=8) as pool:
+    list(tqdm(pool.imap(run_command, data), total=len(data)))
+

passed_templates.txt

@@ -0,0 +1,49 @@
+bump_plain_chart_02
+bump_plain_chart_03
+circular_bar_chart_01
+circular_bar_chart_03
+circular_bar_chart_04
+circular_bar_chart_icons_01
+circular_bar_chart_icons_02
+horizontal_bar_chart_0
+horizontal_bar_chart_01
+horizontal_bar_chart_0_weilai
+horizontal_bar_chart_1
+horizontal_bar_chart_10
+horizontal_bar_chart_11
+horizontal_bar_chart_12
+horizontal_diverging_bar_chart_05
+horizontal_diverging_bar_chart_07
+horizontal_diverging_bar_chart_08
+horizontal_diverging_bar_chart_10
+horizontal_diverging_bar_chart_11
+horizontal_diverging_bar_chart_12
+line_graph_plain_chart_03
+lollipop_icon_chart_01
+lollipop_icon_chart_02
+lollipop_plain_chart_01
+multiple_gauge_chart_01
+multiple_gauge_chart_02
+multiple_gauge_chart_03
+multiple_gauge_chart_03_hand
+multiple_line_graph_01
+proportional_area_chart_hexagon_01
+proportional_area_chart_hexagon_02
+proportional_area_chart_other_01
+proportional_area_chart_pentagram_01
+range_area_chart_icons_01
+range_area_plain_chart_01
+rose_chart_01
+rose_chart_new_00
+rose_chart_new_00_hand
+rose_plain_chart_01
+stacked_area_chart_01
+stacked_area_chart_02
+stacked_area_chart_03
+stacked_area_chart_icons_01
+stacked_area_plain_chart_02
+vertical_group_bar_chart_12
+vertical_group_bar_chart_13
+vertical_group_bar_chart_14
+vertical_group_bar_chart_15
+vertical_group_bar_chart_16

passed_templates_audit.csv

@@ -0,0 +1,51 @@
+variation,final_status,final_stage,final_worker,final_line,pass_line,pass_stage,pass_worker,worker_verdict_is_provisional,controller_fresh_review_required,controller_review_required,review_report
+bump_plain_chart_02,pass,controller_fresh_review_round6,controller,51,51,controller_fresh_review_round6,controller,,False,,data/output/variation_skill_runs/bump_plain_chart_02/20260603_220532/variation_review_round6_controller_fresh_pass.md
+bump_plain_chart_03,pass,controller_fresh_review_round4,controller,124,124,controller_fresh_review_round4,controller,,False,,data/output/variation_skill_runs/bump_plain_chart_03/20260603_224758/variation_review_round4_controller_fresh_pass.md
+circular_bar_chart_01,pass,controller_fresh_pass_round14,,277,277,controller_fresh_pass_round14,,,False,False,data/output/variation_skill_runs/circular_bar_chart_01/20260603_231601/variation_review_round14_controller_fresh_pass.md
+circular_bar_chart_03,pass,controller_fresh_pass_round32,controller,542,542,controller_fresh_pass_round32,controller,,,,
+circular_bar_chart_04,pass,controller_fresh_pass_round8,controller,580,580,controller_fresh_pass_round8,controller,,,,
+circular_bar_chart_icons_01,pass,controller_fresh_pass_round5,controller,667,667,controller_fresh_pass_round5,controller,,,,
+circular_bar_chart_icons_02,pass,worker_final_round5_provisional,session_01,724,724,worker_final_round5_provisional,session_01,,True,,
+horizontal_bar_chart_0,pass,controller_fresh_review_round7,controller,60,60,controller_fresh_review_round7,controller,,False,,data/output/variation_skill_runs/horizontal_bar_chart_0/20260603_215905/variation_review_round7_controller_fresh_pass.md
+horizontal_bar_chart_01,pass,,,192,192,,,True,True,True,data/output/variation_skill_runs/horizontal_bar_chart_01/20260603_225157/variation_review_round6.md
+horizontal_bar_chart_0_weilai,pass,controller_fresh_pass_round2,,299,299,controller_fresh_pass_round2,,,False,False,data/output/variation_skill_runs/horizontal_bar_chart_0_weilai/20260603_235404/variation_review_round2_controller_fresh_pass.md
+horizontal_bar_chart_1,pass,controller_fresh_pass_round6,session_02,408,408,controller_fresh_pass_round6,session_02,,False,,
+horizontal_bar_chart_10,pass,controller_fresh_pass_round14,controller,534,534,controller_fresh_pass_round14,controller,,,,
+horizontal_bar_chart_11,pass,controller_fresh_pass_round7,controller,581,581,controller_fresh_pass_round7,controller,,,,
+horizontal_bar_chart_12,pass,worker_final_round12_provisional,session_02,688,688,worker_final_round12_provisional,session_02,,True,,data/output/variation_skill_runs/horizontal_bar_chart_12/20260604_051705/variation_review_round12.md
+horizontal_diverging_bar_chart_05,pass,worker_provisional_pass_round16_controller_review_required,session_03,218,218,worker_provisional_pass_round16_controller_review_required,session_03,,True,True,data/output/variation_skill_runs/horizontal_diverging_bar_chart_05/20260603_202638/variation_review_round16.md
+horizontal_diverging_bar_chart_07,pass,controller_fresh_pass_round12,session_03,388,388,controller_fresh_pass_round12,session_03,,False,,
+horizontal_diverging_bar_chart_08,pass,controller_fresh_pass_round12,controller,522,522,controller_fresh_pass_round12,controller,,,,
+horizontal_diverging_bar_chart_10,pass,worker_provisional_pass_round6_controller_required,session_03,576,576,worker_provisional_pass_round6_controller_required,session_03,True,True,,data/output/variation_skill_runs/horizontal_diverging_bar_chart_10/20260604_033039/variation_review_round6.md
+horizontal_diverging_bar_chart_11,pass,controller_fresh_pass_round8,controller,652,652,controller_fresh_pass_round8,controller,,,,
+horizontal_diverging_bar_chart_12,pass,worker_final_round5_provisional,session_03,705,705,worker_final_round5_provisional,session_03,,True,,data/output/variation_skill_runs/horizontal_diverging_bar_chart_12/20260604_062826/variation_review_round5.md
+line_graph_plain_chart_03,pass,controller_fresh_pass_round2,controller,546,546,controller_fresh_pass_round2,controller,,,,
+lollipop_icon_chart_01,pass,worker_provisional_pass_round11_controller_required_after_round8_fail,session_04,587,587,worker_provisional_pass_round11_controller_required_after_round8_fail,session_04,True,True,,data/output/variation_skill_runs/lollipop_icon_chart_01/20260604_040357/variation_review_round11.md
+lollipop_icon_chart_02,pass,controller_fresh_pass_round7,controller,638,638,controller_fresh_pass_round7,controller,,,,
+lollipop_plain_chart_01,pass,final_worker_record_round8_provisional,session_04,723,723,final_worker_record_round8_provisional,session_04,,True,,data/output/variation_skill_runs/lollipop_plain_chart_01/20260604_061905/variation_review_round8.md
+multiple_gauge_chart_01,pass,controller_fresh_review_round7,controller,37,37,controller_fresh_review_round7,controller,,False,,data/output/variation_skill_runs/multiple_gauge_chart_01/20260603_214437/variation_review_round7_controller_fresh_pass.md
+multiple_gauge_chart_02,pass,controller_update_endpoint_corner_modifier_notes_confirmed,session_04,161,161,controller_update_endpoint_corner_modifier_notes_confirmed,session_04,True,True,True,data/output/variation_skill_runs/multiple_gauge_chart_02/20260603_224205/variation_review_round12.md
+multiple_gauge_chart_03,pass,controller_fresh_pass_round10,session_04,392,392,controller_fresh_pass_round10,session_04,,False,,
+multiple_gauge_chart_03_hand,pass,controller_fresh_pass_round4,controller,438,438,controller_fresh_pass_round4,controller,,,,
+multiple_line_graph_01,pass,controller_fresh_pass_round7,controller,485,485,controller_fresh_pass_round7,controller,,,,
+proportional_area_chart_hexagon_01,pass,controller_fresh_pass_round16,,306,306,controller_fresh_pass_round16,,,False,False,data/output/variation_skill_runs/proportional_area_chart_hexagon_01/20260603_221214/variation_review_round16_controller_fresh_pass.md
+proportional_area_chart_hexagon_02,pass,controller_fresh_pass_round13,controller,448,448,controller_fresh_pass_round13,controller,,,,
+proportional_area_chart_other_01,pass,controller_fresh_pass_round34,controller,627,627,controller_fresh_pass_round34,controller,,,,
+proportional_area_chart_pentagram_01,pass,final_worker_record_round5,session_05,713,713,final_worker_record_round5,session_05,,True,,data/output/variation_skill_runs/proportional_area_chart_pentagram_01/20260604_060611/variation_review_round5.md
+range_area_chart_icons_01,pass,controller_fresh_review_round6,controller,129,129,controller_fresh_review_round6,controller,,False,,data/output/variation_skill_runs/range_area_chart_icons_01/20260603_222013/variation_review_round6_controller_fresh_pass.md
+range_area_plain_chart_01,pass,controller_fresh_pass_round10,,331,331,controller_fresh_pass_round10,,,False,False,data/output/variation_skill_runs/range_area_plain_chart_01/20260603_232034/variation_review_round10_controller_fresh_pass.md
+rose_chart_01,pass,controller_fresh_pass_round6_after_worker_provisional_pass,controller,428,428,controller_fresh_pass_round6_after_worker_provisional_pass,controller,,,,
+rose_chart_new_00,pass,controller_fresh_pass_round14,controller,544,544,controller_fresh_pass_round14,controller,,,,
+rose_chart_new_00_hand,pass,worker_provisional_pass_round10_controller_required,session_06,588,588,worker_provisional_pass_round10_controller_required,session_06,True,True,,data/output/variation_skill_runs/rose_chart_new_00_hand/20260604_035901/variation_review_round10.md
+rose_plain_chart_01,pass,final_worker_record_round9_provisional,session_06,686,686,final_worker_record_round9_provisional,session_06,,True,,data/output/variation_skill_runs/rose_plain_chart_01/20260604_052842/variation_review_round9.md
+stacked_area_chart_01,pass,controller_fresh_review_round8,controller,79,79,controller_fresh_review_round8,controller,,False,,data/output/variation_skill_runs/stacked_area_chart_01/20260603_213619/variation_review_round8_controller_fresh_pass.md
+stacked_area_chart_02,pass,completion_worker_provisional_pass_controller_required_round8,session_07,180,180,completion_worker_provisional_pass_controller_required_round8,session_07,True,True,True,data/output/variation_skill_runs/stacked_area_chart_02/20260603_225749/variation_review_round8.md
+stacked_area_chart_03,pass,controller_fresh_pass_round15,controller,454,454,controller_fresh_pass_round15,controller,,,,
+stacked_area_chart_icons_01,pass,variation_review_round7,,563,563,variation_review_round7,,,True,,data/output/variation_skill_runs/stacked_area_chart_icons_01/20260604_024630/variation_review_round7.md
+stacked_area_plain_chart_01,worker_provisional_pass_round13_controller_required,final_worker_record,session_07,687,680,controller_fresh_pass_round13,controller,,,,
+stacked_area_plain_chart_02,pass,,session_07,725,725,,session_07,,True,,data/output/variation_skill_runs/stacked_area_plain_chart_02/20260604_070936/variation_review_round3.md
+vertical_group_bar_chart_12,pass,worker_provisional_pass_round13_controller_fresh_review_required,session_08,236,236,worker_provisional_pass_round13_controller_fresh_review_required,session_08,,True,True,data/output/variation_skill_runs/vertical_group_bar_chart_12/20260603_215309/variation_review_round13.md
+vertical_group_bar_chart_13,pass,controller_fresh_pass_round10_after_worker_provisional_pass,controller,424,424,controller_fresh_pass_round10_after_worker_provisional_pass,controller,,,,
+vertical_group_bar_chart_14,pass,controller_fresh_pass_round4,controller,518,518,controller_fresh_pass_round4,controller,,,,
+vertical_group_bar_chart_15,pass,controller_fresh_pass_round5,controller,606,606,controller_fresh_pass_round5,controller,,,,
+vertical_group_bar_chart_16,pass,final_worker_record_round10_provisional,session_08,696,696,final_worker_record_round10_provisional,session_08,,True,,data/output/variation_skill_runs/vertical_group_bar_chart_16/20260604_053941/variation_review_round10.md

passed_templates_ever_pass.txt

@@ -0,0 +1,50 @@
+bump_plain_chart_02
+bump_plain_chart_03
+circular_bar_chart_01
+circular_bar_chart_03
+circular_bar_chart_04
+circular_bar_chart_icons_01
+circular_bar_chart_icons_02
+horizontal_bar_chart_0
+horizontal_bar_chart_01
+horizontal_bar_chart_0_weilai
+horizontal_bar_chart_1
+horizontal_bar_chart_10
+horizontal_bar_chart_11
+horizontal_bar_chart_12
+horizontal_diverging_bar_chart_05
+horizontal_diverging_bar_chart_07
+horizontal_diverging_bar_chart_08
+horizontal_diverging_bar_chart_10
+horizontal_diverging_bar_chart_11
+horizontal_diverging_bar_chart_12
+line_graph_plain_chart_03
+lollipop_icon_chart_01
+lollipop_icon_chart_02
+lollipop_plain_chart_01
+multiple_gauge_chart_01
+multiple_gauge_chart_02
+multiple_gauge_chart_03
+multiple_gauge_chart_03_hand
+multiple_line_graph_01
+proportional_area_chart_hexagon_01
+proportional_area_chart_hexagon_02
+proportional_area_chart_other_01
+proportional_area_chart_pentagram_01
+range_area_chart_icons_01
+range_area_plain_chart_01
+rose_chart_01
+rose_chart_new_00
+rose_chart_new_00_hand
+rose_plain_chart_01
+stacked_area_chart_01
+stacked_area_chart_02
+stacked_area_chart_03
+stacked_area_chart_icons_01
+stacked_area_plain_chart_01
+stacked_area_plain_chart_02
+vertical_group_bar_chart_12
+vertical_group_bar_chart_13
+vertical_group_bar_chart_14
+vertical_group_bar_chart_15
+vertical_group_bar_chart_16

passed_templates_final_status_contains_pass.txt

@@ -0,0 +1,50 @@
+bump_plain_chart_02
+bump_plain_chart_03
+circular_bar_chart_01
+circular_bar_chart_03
+circular_bar_chart_04
+circular_bar_chart_icons_01
+circular_bar_chart_icons_02
+horizontal_bar_chart_0
+horizontal_bar_chart_01
+horizontal_bar_chart_0_weilai
+horizontal_bar_chart_1
+horizontal_bar_chart_10
+horizontal_bar_chart_11
+horizontal_bar_chart_12
+horizontal_diverging_bar_chart_05
+horizontal_diverging_bar_chart_07
+horizontal_diverging_bar_chart_08
+horizontal_diverging_bar_chart_10
+horizontal_diverging_bar_chart_11
+horizontal_diverging_bar_chart_12
+line_graph_plain_chart_03
+lollipop_icon_chart_01
+lollipop_icon_chart_02
+lollipop_plain_chart_01
+multiple_gauge_chart_01
+multiple_gauge_chart_02
+multiple_gauge_chart_03
+multiple_gauge_chart_03_hand
+multiple_line_graph_01
+proportional_area_chart_hexagon_01
+proportional_area_chart_hexagon_02
+proportional_area_chart_other_01
+proportional_area_chart_pentagram_01
+range_area_chart_icons_01
+range_area_plain_chart_01
+rose_chart_01
+rose_chart_new_00
+rose_chart_new_00_hand
+rose_plain_chart_01
+stacked_area_chart_01
+stacked_area_chart_02
+stacked_area_chart_03
+stacked_area_chart_icons_01
+stacked_area_plain_chart_01
+stacked_area_plain_chart_02
+vertical_group_bar_chart_12
+vertical_group_bar_chart_13
+vertical_group_bar_chart_14
+vertical_group_bar_chart_15
+vertical_group_bar_chart_16

pipeline.py

@@ -0,0 +1,630 @@
+# -*- coding: utf-8 -*-
+
+"""
+ChartPipeline: 完整可视化生成管道
+此脚本执行完整的数据可视化生成流程，依次调用各模块实现从数据到最终图表的转换
+"""
+
+import os
+import json
+import argparse
+import logging
+import time
+from datetime import datetime
+from importlib import import_module
+from pathlib import Path
+from config import (
+    base_url,
+    api_key,
+    embed_model_path,
+    data_resource_path,
+    topk,
+    text_data_path,
+    text_index_path,
+    color_data_path,
+    color_index_path,
+    image_data_path,
+    image_index_path,
+    image_list_path,
+    image_resource_path
+)
+import random
+from concurrent.futures import ProcessPoolExecutor
+
+# 配置日志
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+)
+logger = logging.getLogger("ChartPipeline")
+
+# 模块配置
+MODULES = [
+    {
+        "name": "create_index",
+        "description": "索引创建模块",
+        "input_type": "none",
+        "output_type": "none"
+    },
+    {
+        "name": "preprocess",
+        "description": "数据预处理模块",
+        "input_type": "json",
+        "output_type": "json"
+    },
+    {
+        "name": "chart_type_recommender",
+        "description": "图表类型推荐模块",
+        "input_type": "json",
+        "output_type": "json"
+    },
+    {
+        "name": "datafact_generator",
+        "description": "数据洞察模块",
+        "input_type": "json",
+        "output_type": "json"
+    },
+    {
+        "name": "title_generator",
+        "description": "标题生成模块",
+        "input_type": "json",
+        "output_type": "json"
+    },
+    {
+        "name": "color_recommender",
+        "description": "色彩推荐模块",
+        "input_type": "json",
+        "output_type": "json"
+    },
+    {
+        "name": "image_recommender",
+        "description": "图像推荐模块",
+        "input_type": "json",
+        "output_type": "json"
+    },
+    {
+        "name": "all",
+        "description": "综合模块(datafact+title+color+image)",
+        "input_type": "json",
+        "output_type": "json"
+    },
+    {
+        "name": "infographics_generator",
+        "description": "信息图表生成模块",
+        "input_type": "json",
+        "output_type": "json"
+    },
+    {
+        "name": "asset_regenerator",
+        "description": "图像/图标统一生图后处理模块",
+        "input_type": "svg",
+        "output_type": "svg"
+    },
+    {
+        "name": "full_image_polisher",
+        "description": "整图GPT Image润色后处理模块",
+        "input_type": "png",
+        "output_type": "png"
+    },
+    {
+        "name": "chart_engine",
+        "description": "图表模板实现引擎",
+        "input_type": "json",
+        "output_type": "svg"
+    },
+    {
+        "name": "title_styler",
+        "description": "标题元素生成模块",
+        "input_type": "json",
+        "output_type": "svg"
+    }
+]
+
+
+def run_pipeline(input_path, output_path=None, temp_dir=None, modules_to_run=None, threads=None, chart_name=None, chart_only=False, output_png=False):
+    """
+    执行完整的图表生成管道
+
+    Args:
+        input_path (str): 输入数据文件路径(可以是文件或目录)
+        output_path (str, optional): 输出文件路径(可以是文件或目录)，如果为None则原地修改
+        temp_dir (str, optional): 临时文件目录，默认使用./tmp
+        modules_to_run (list, optional): 要运行的模块列表，默认运行所有模块
+        threads (int, optional): 处理目录时的并发线程数，仅在input_path为目录时生效
+        chart_name (str, optional): 指定图表名称，仅对infographics_generator模块有效
+        chart_only (bool, optional): 只输出 chart SVG（跳过 title/image/layout），仅对 infographics_generator 有效
+        output_png (bool, optional): 同时输出 PNG，仅对 infographics_generator 有效
+    """
+    # try:
+    if chart_name is not None:
+        output_path = os.path.join(output_path, chart_name)
+        import shutil
+        shutil.rmtree(output_path, ignore_errors=True)
+        os.makedirs(output_path, exist_ok=True)
+    print("modules_to_run: ", modules_to_run)
+    # 如果是create_index模块，单独处理
+    if modules_to_run and 'create_index' in modules_to_run:
+        return run_single_file(
+            input_path=input_path,
+            output_path=output_path,
+            temp_dir=temp_dir,
+            modules_to_run=modules_to_run,
+            chart_name=chart_name,
+            chart_only=chart_only,
+            output_png=output_png,
+        )
+
+    input_path = Path(input_path)
+    output_path = Path(output_path) if output_path else input_path
+    temp_dir = Path(temp_dir) if temp_dir else Path("./tmp")
+
+    if input_path.is_dir():
+        # 如果指定了输出目录且不同于输入目录，创建输出目录
+        if output_path != input_path:
+            output_path.mkdir(parents=True, exist_ok=True)
+
+        # 获取输入目录下所有JSON文件
+        input_files = list(input_path.glob('*.json'))
+        random.shuffle(input_files)  # 随机打乱文件顺序
+
+        if threads and threads > 1:
+            # 使用线程池并行处理文件
+            with ProcessPoolExecutor(max_workers=threads) as executor:
+                futures = []
+                for input_file in input_files:
+                    # 如果是inplace处理，输出路径就是输入路径
+                    if output_path == input_path:
+                        output_file = input_file
+                    else:
+                        # 确保输出文件保持相同的文件名
+                        output_file = output_path / input_file.name
+
+                    future = executor.submit(
+                        run_single_file,
+                        input_path=input_file,
+                        output_path=output_file,
+                        temp_dir=temp_dir,
+                        modules_to_run=modules_to_run,
+                        chart_name=chart_name,
+                        chart_only=chart_only,
+                        output_png=output_png,
+                    )
+                    futures.append(future)
+
+                # 等待所有任务完成并检查结果
+                results = [future.result() for future in futures]
+                return all(results)
+        else:
+            print("单线程顺序处理")
+            # 单线程顺序处理
+            success = True
+            for input_file in input_files:
+                if output_path == input_path:
+                    output_file = input_file
+                else:
+                    output_file = output_path / input_file.name
+
+                success &= run_single_file(
+                    input_path=input_file,
+                    output_path=output_file,
+                    temp_dir=temp_dir,
+                    modules_to_run=modules_to_run,
+                    chart_name=chart_name,
+                    chart_only=chart_only,
+                    output_png=output_png,
+                )
+            return success
+    else:
+        # 单文件处理
+        if output_path == input_path:
+            output_file = input_path
+        else:
+            # 如果指定了不同的输出路径，确保它的父目录存在
+            output_path.parent.mkdir(parents=True, exist_ok=True)
+            output_file = output_path
+
+        return run_single_file(
+            input_path=input_path,
+            output_path=output_file,
+            temp_dir=temp_dir,
+            modules_to_run=modules_to_run,
+            chart_name=chart_name,
+            chart_only=chart_only,
+            output_png=output_png,
+        )
+
+    # except Exception as e:
+    #     logger.error(f"管道执行失败: {str(e)}")
+    #     return False
+
+def run_single_file(input_path, output_path, temp_dir=None, modules_to_run=None, chart_name=None, chart_only=False, output_png=False, return_timing=False):
+    """
+    处理单个文件的管道逻辑
+
+    Args:
+        input_path (Path): 输入JSON文件路径
+        output_path (Path): 输出路径
+        temp_dir (Path): 临时文件目录，默认为./tmp
+        modules_to_run (list): 要运行的模块列表
+        chart_name (str, optional): 指定图表名称，仅对infographics_generator模块有效
+        chart_only (bool, optional): 只输出 chart SVG，仅对 infographics_generator 有效
+        output_png (bool, optional): 同时输出 PNG，仅对 infographics_generator 有效
+    """
+    # try:
+    run_start_time = time.time()
+    module_timings = {}
+
+    def finish(success: bool):
+        total_seconds = time.time() - run_start_time
+        logger.info(
+            "run_single_file timing: total=%.3fs modules=%s",
+            total_seconds,
+            {k: round(v, 3) for k, v in module_timings.items()},
+        )
+        if return_timing:
+            return {
+                "success": success,
+                "total_seconds": total_seconds,
+                "module_seconds": module_timings,
+            }
+        return success
+
+    def fail_module(module_name: str):
+        logger.error("模块执行失败: %s", module_name)
+        return finish(False)
+
+    # 确定要运行的模块
+    if not modules_to_run:
+        modules_to_run = [m["name"] for m in MODULES]
+
+    # 如果要运行create_index，单独处理并直接返回
+    if 'create_index' in modules_to_run:
+        logger.info("执行create_index模块")
+        # 依次调用三个模块的create_index
+        for module_type in ['title', 'color', 'image']:
+            if module_type == 'title':
+                print("Running modules.title_generator.create_index")
+                module = import_module('modules.title_generator.create_index')
+                if not Path(text_index_path).exists():
+                    module.process(
+                        data=text_data_path,
+                        index_path=text_index_path,
+                        data_path=text_data_path,
+                        embed_model_path=embed_model_path
+                    )
+            elif module_type == 'color':
+                print("Running modules.color_recommender.create_index")
+                module = import_module('modules.color_recommender.create_index')
+                if not Path(color_index_path).exists():
+                    module.main(
+                        input=color_data_path,
+                        output=color_index_path,
+                        embed_model_path=embed_model_path
+                    )
+            else:  # image
+                print("Running modules.image_recommender.create_index")
+                module = import_module('modules.image_recommender.create_index')
+                if not Path(image_index_path).exists():
+                    module.main(
+                        image_list_path=image_list_path,
+                        image_resource_path=image_resource_path,
+                        index_path=image_index_path,
+                        data_path=image_data_path,
+                        embed_model_path=embed_model_path
+                    )
+        total_seconds = time.time() - run_start_time
+        if return_timing:
+            return {
+                "success": True,
+                "total_seconds": total_seconds,
+                "module_seconds": module_timings,
+            }
+        return True
+
+    # 创建临时目录
+    if temp_dir is None:
+        temp_dir = Path("./tmp")
+    temp_dir.mkdir(parents=True, exist_ok=True)
+
+    # 确保输出目录存在
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+
+    # 当前输入文件
+    current_input = input_path
+
+    # 获取最后一个模块的配置
+    last_module = [m for m in MODULES if m["name"] in modules_to_run][-1]
+
+    # 根据最后一个模块的输出类型决定最终输出文件的扩展名
+    if last_module["name"] in ["chart_engine", "title_styler", "asset_regenerator"]:
+        final_output = output_path.with_suffix('.svg')
+    elif last_module["name"] == "full_image_polisher":
+        final_output = output_path.with_suffix('.png')
+    elif last_module["name"] == "infographics_generator":
+        final_output = output_path.parent / f"{output_path.stem}_final.svg"
+    else:
+        final_output = output_path.with_suffix('.json')
+
+    # 记录执行过程
+    # 依次执行各模块
+    for i, module_config in enumerate([m for m in MODULES if m["name"] in modules_to_run]):
+        module_name = module_config["name"]
+        module_desc = module_config["description"]
+        module_start_time = time.time()
+        # logger.info(f"执行模块 {i+1}/{len(modules_to_run)}: {module_name} - {module_desc}")
+
+        # 特殊处理all模块，依次执行datafact_generator、title_generator、color_recommender和image_recommender
+        if module_name == "all":
+            # 数据洞察模块
+            preprocess_module = import_module(f"modules.preprocess.preprocess")
+            if not should_skip_module(module_name, output_path):
+                step_start_time = time.time()
+                preprocess_module.process(input=str(current_input), output=str(output_path))
+                module_timings["all.preprocess"] = time.time() - step_start_time
+                current_input = output_path
+
+            datafact_module = import_module("modules.datafact_generator.datafact_generator")
+            if not should_skip_module("datafact_generator", output_path):
+                step_start_time = time.time()
+                datafact_module.process(input=str(current_input), output=str(output_path))
+                module_timings["all.datafact_generator"] = time.time() - step_start_time
+                current_input = output_path
+
+            # 标题生成模块
+            title_module = import_module("modules.title_generator.title_generator")
+            if not should_skip_module("title_generator", output_path):
+                step_start_time = time.time()
+                title_module.process(
+                    input=str(current_input),
+                    output=str(output_path),
+                    base_url=base_url,
+                    api_key=api_key,
+                    embed_model_path=embed_model_path,
+                    topk=topk,
+                    data_path=text_data_path,
+                    index_path=text_index_path
+                )
+                module_timings["all.title_generator"] = time.time() - step_start_time
+                current_input = output_path
+
+            # 色彩推荐模块
+            color_module = import_module("modules.color_recommender.color_recommender")
+            if not should_skip_module("color_recommender", output_path):
+                step_start_time = time.time()
+                color_module.process(
+                    input=str(current_input),
+                    output=str(output_path),
+                    base_url=base_url,
+                    api_key=api_key,
+                    embed_model_path=embed_model_path,
+                    data_path=color_data_path,
+                    index_path=color_index_path
+                )
+                module_timings["all.color_recommender"] = time.time() - step_start_time
+                current_input = output_path
+
+            # 图像推荐模块
+            image_module = import_module("modules.image_recommender.image_recommender")
+            if not should_skip_module("image_recommender", output_path):
+                step_start_time = time.time()
+                image_module.process(
+                    input=str(current_input),
+                    output=str(output_path),
+                    base_url=base_url,
+                    api_key=api_key,
+                    embed_model_path=embed_model_path,
+                    data_path=image_data_path,
+                    index_path=image_index_path,
+                    resource_path=image_resource_path
+                )
+                module_timings["all.image_recommender"] = time.time() - step_start_time
+                current_input = output_path
+
+        # 特殊处理title_generator模块，传入配置参数
+        elif module_name == "title_generator":
+            module = import_module(f"modules.{module_name}.{module_name}")
+            if not should_skip_module(module_name, output_path):
+                step_start_time = time.time()
+                module.process(
+                    input=str(current_input),
+                    output=str(output_path),
+                    base_url=base_url,
+                    api_key=api_key,
+                    embed_model_path=embed_model_path,
+                    topk=topk,
+                    data_path=text_data_path,
+                    index_path=text_index_path
+                )
+                module_timings["title_generator"] = time.time() - step_start_time
+                current_input = output_path
+        elif module_name == "color_recommender":
+            module = import_module(f"modules.{module_name}.{module_name}")
+            if not should_skip_module(module_name, output_path):
+                step_start_time = time.time()
+                module.process(
+                    input=str(current_input),
+                    output=str(output_path),
+                    base_url=base_url,
+                    api_key=api_key,
+                    embed_model_path=embed_model_path,
+                    data_path=color_data_path,
+                    index_path=color_index_path
+                )
+                module_timings["color_recommender"] = time.time() - step_start_time
+                current_input = output_path
+        elif module_name == "image_recommender":
+            module = import_module(f"modules.{module_name}.{module_name}")
+            print("image_recommender")
+            if not should_skip_module(module_name, output_path):
+                print("process")
+                step_start_time = time.time()
+                module.process(
+                    input=str(current_input),
+                    output=str(output_path),
+                    base_url=base_url,
+                    api_key=api_key,
+                    embed_model_path=embed_model_path,
+                    data_path=image_data_path,
+                    index_path=image_index_path,
+                    resource_path=image_resource_path
+                )
+                module_timings["image_recommender"] = time.time() - step_start_time
+                current_input = output_path
+        elif module_name == "infographics_generator":
+            module = import_module(f"modules.{module_name}.{module_name}")
+            if not should_skip_module(module_name, output_path):
+                step_start_time = time.time()
+                result = module.process(
+                    input=str(current_input),
+                    output=str(output_path),
+                    base_url=base_url,
+                    api_key=api_key,
+                    chart_name=chart_name,
+                    chart_only=chart_only,
+                    output_png=output_png,
+                )
+                module_timings["infographics_generator"] = time.time() - step_start_time
+                if result is False:
+                    return fail_module(module_name)
+                current_input = output_path
+        elif module_name == "asset_regenerator":
+            module = import_module(f"modules.{module_name}.{module_name}")
+            if not should_skip_module(module_name, output_path):
+                ok = module.process(
+                    input=str(current_input),
+                    output=str(output_path),
+                    base_url=base_url,
+                    api_key=api_key,
+                )
+                if not ok:
+                    return False
+                current_input = output_path
+        elif module_name == "full_image_polisher":
+            module = import_module(f"modules.{module_name}.{module_name}")
+            if not should_skip_module(module_name, output_path):
+                ok = module.process(
+                    input=str(current_input),
+                    output=str(output_path),
+                    base_url=base_url,
+                    api_key=api_key,
+                )
+                if not ok:
+                    return False
+                current_input = output_path
+        elif module_name == "chart_engine":
+            # 输入是JSON，输出是SVG
+            module = import_module(f"modules.{module_name}.{module_name}")
+            svg_output = output_path.with_suffix('.svg')
+            if True: # not svg_output.exists():
+                step_start_time = time.time()
+                result = module.process(input=str(current_input), output=str(svg_output))
+                module_timings["chart_engine"] = time.time() - step_start_time
+                if result is False:
+                    return fail_module(module_name)
+            current_input = svg_output  # 更新为SVG文件作为下一个模块的输入
+
+        elif module_name == "title_styler":
+            # 输入是JSON，输出是SVG
+            module = import_module(f"modules.{module_name}.{module_name}")
+            title_svg = output_path.parent / f"{output_path.stem}_title.svg"
+            if not title_svg.exists():
+                step_start_time = time.time()
+                result = module.process(input=str(current_input), output=str(title_svg))
+                module_timings["title_styler"] = time.time() - step_start_time
+                if result is False:
+                    return fail_module(module_name)
+            current_input = title_svg  # 更新为标题SVG作为下一个模块的输入
+
+        else:
+            # 普通模块：输入JSON，输出JSON
+            module = import_module(f"modules.{module_name}.{module_name}")
+            if not should_skip_module(module_name, output_path):
+                step_start_time = time.time()
+                result = module.process(input=str(current_input), output=str(output_path))
+                module_timings[module_name] = time.time() - step_start_time
+                if result is False:
+                    return fail_module(module_name)
+                current_input = output_path
+
+        module_timings[f"{module_name}.total"] = time.time() - module_start_time
+
+    return finish(True)
+
+    # except Exception as e:
+    #     logger.error(f"文件处理失败 {input_path}: {str(e)}")
+    #     return False
+
+def should_skip_module(module_name: str, output_path: Path) -> bool:
+    """检查是否需要跳过模块执行"""
+    try:
+        if not output_path.exists():
+            return False
+
+        # 检查JSON文件中的特定字段
+        with open(output_path) as f:
+            data = json.load(f)
+
+        skip_conditions = {
+            "preprocess": lambda d: "metadata" in d and "data" in d and "variables" in d and "processed" in d,
+            "chart_type_recommender": lambda d: "chart_type" in d,
+            "datafact_generator": lambda d: "datafacts" in d,
+            "title_generator": lambda d: False,#"titles" in d,
+            "color_recommender": lambda d: "colors" in d,
+            "image_recommender": lambda d: "images" in d
+        }
+
+        if module_name in skip_conditions:
+            return skip_conditions[module_name](data)
+
+        return False
+
+    except Exception as e:
+        logger.warning(f"检查跳过条件时出错: {str(e)}")
+        return False
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--input', type=str,
+                        help='Input json file path or directory (default: config.data_resource_path = data_resource_dirs[0])',
+                        default=data_resource_path)
+    parser.add_argument('--output', type=str, help='Output json file path', default='output')
+    parser.add_argument('--temp-dir', type=str, default='tmp')
+    parser.add_argument('--modules', type=str, nargs='+', help='Modules to run', default=['infographics_generator'])
+    parser.add_argument('--threads', type=int, help='Number of threads for directory processing', default=1)
+    parser.add_argument('--chart-name', type=str, help='Specific chart name to use for infographics_generator')
+    parser.add_argument('--chart-only', action='store_true', help='Only output the chart SVG; skip title/image/layout (infographics_generator only)')
+    parser.add_argument('--output-png', action='store_true', help='Also write a PNG next to the final SVG (uses rsvg-convert)')
+
+    args = parser.parse_args()
+    # 如果没有指定input，从data_resource_path随机选择
+    if args.input is None and 'create_index' not in args.modules:
+        json_files = [f for f in os.listdir(data_resource_path) if f.endswith('.json')]
+        if not json_files:
+            raise ValueError(f"在 {data_resource_path} 目录下没有找到json文件")
+        args.input = os.path.join(data_resource_path, random.choice(json_files))
+        print(f"随机��择输入文件: {args.input}")
+        args.output = "tmp.json"
+        print(f"使用默认输出文件: {args.output}")
+
+    return args
+
+def main():
+    args = parse_args()
+    modules_to_run = None
+    if args.modules:
+        modules_to_run = [m.strip() for m in args.modules]
+
+    ok = run_pipeline(
+        input_path=args.input,
+        output_path=args.output,
+        temp_dir=args.temp_dir,
+        modules_to_run=modules_to_run,
+        threads=args.threads,
+        chart_name=args.chart_name,
+        chart_only=args.chart_only,
+        output_png=args.output_png,
+    )
+    raise SystemExit(0 if ok else 1)
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,15 @@
+pandas>=1.3.0
+numpy>=1.20.0
+opencv-python-headless>=4.8.0
+IPython
+pyecharts
+snapshot_selenium
+imgkit
+networkx
+matplotlib
+openai
+serpapi
+
+clip
+torch
+faiss

setup.sh

@@ -0,0 +1,100 @@
+#!/usr/bin/env bash
+# Setup script for ChartPipeline on a fresh machine.
+# Run from the project root (the directory this script is in).
+#
+# Prereqs you have to handle BEFORE running this script:
+#   1. Python 3.10+ installed (conda or system).
+#   2. Node.js 18+ installed (`node -v`).
+#   3. Google Chrome or Chromium installed at /usr/bin/google-chrome
+#      (or set PUPPETEER_EXECUTABLE_PATH to its real path).
+#   4. librsvg2-bin installed (provides rsvg-convert).  On Ubuntu:
+#        sudo apt install librsvg2-bin tmux
+#      If you have no sudo, install via conda:
+#        conda install -c conda-forge librsvg tmux
+#   5. Input data pools available at the paths config.py points to,
+#      e.g. /data/liduan/resources/claude_data_v2 and claude_new.
+#      Edit config.data_resource_dirs to match this machine if needed.
+
+set -u
+cd "$(dirname "$0")"
+ROOT="$(pwd)"
+echo "[setup] cwd=$ROOT"
+
+# ---- 1. Python deps ----
+echo
+echo "============================================================"
+echo "[setup] installing python deps from requirements.txt"
+echo "============================================================"
+pip install -r requirements.txt
+# Things requirements.txt was missing on the source machine; install
+# defensively so chart_engine + selenium-based screenshot both work.
+pip install pyecharts snapshot_selenium selenium
+
+# ---- 2. Node deps ----
+echo
+echo "============================================================"
+echo "[setup] installing node deps (puppeteer / echarts ...)"
+echo "============================================================"
+if [[ -f package.json ]]; then
+  # IMPORTANT: ChartPipeline's puppeteer was patched to *re-use* an existing
+  # Chrome via PUPPETEER_EXECUTABLE_PATH; skip the bundled chromium download
+  # to save ~300 MB and avoid network friction.
+  PUPPETEER_SKIP_DOWNLOAD=1 npm install
+else
+  echo "[setup] package.json missing, skipping npm install"
+fi
+
+# ---- 3. System binaries sanity check ----
+echo
+echo "============================================================"
+echo "[setup] checking system binaries"
+echo "============================================================"
+for bin in google-chrome rsvg-convert tmux node npm python; do
+  loc="$(command -v "$bin" 2>/dev/null || true)"
+  if [[ -n "$loc" ]]; then
+    printf '  %-14s  OK   %s\n' "$bin" "$loc"
+  else
+    printf '  %-14s  MISSING  -- you need to install this before running pipeline.py\n' "$bin"
+  fi
+done
+
+# ---- 4. PUPPETEER_EXECUTABLE_PATH ----
+echo
+echo "============================================================"
+echo "[setup] puppeteer chrome path"
+echo "============================================================"
+CHROME_BIN="${PUPPETEER_EXECUTABLE_PATH:-$(command -v google-chrome || command -v chromium || true)}"
+if [[ -n "$CHROME_BIN" ]]; then
+  echo "  PUPPETEER_EXECUTABLE_PATH = $CHROME_BIN  (export this before running pipeline.py)"
+else
+  echo "  WARNING: no chrome/chromium found; puppeteer will fail at runtime"
+fi
+
+# ---- 5. Quick import test ----
+echo
+echo "============================================================"
+echo "[setup] verifying config + key imports"
+echo "============================================================"
+python - <<'PY'
+import importlib
+import config
+print(f"  config.resource_path        = {config.resource_path}")
+print(f"  config.data_resource_dirs[0]= {config.data_resource_dirs[0]}")
+for mod in (
+    "modules.chart_engine.template.template_registry",
+    "modules.infographics_generator.template_utils",
+    "modules.infographics_generator.infographics_generator",
+):
+    importlib.import_module(mod)
+    print(f"  import {mod} OK")
+PY
+
+echo
+echo "[setup] DONE."
+echo "Next steps:"
+echo "  1. Edit config.py 's data_resource_dirs if data pools live elsewhere on this host."
+echo "  2. (optional) Edit allowed_chart_types.json to restrict which chart_type are eligible."
+echo "  3. Launch a chart-only run, e.g.:"
+echo "       export PUPPETEER_EXECUTABLE_PATH=$CHROME_BIN"
+echo "       tmux new-session -d -s chart_10h \\"
+echo "         \"PRIOR_SESSION=__none__ THREADS=48 DEADLINE_SEC=36000 bash scripts/run_10h_loop.sh\""