docs/shared_output.md

Shared Output Directory

This project keeps generated charts, regression outputs, and quality-check artifacts out of Git. For multi-user development, use one shared output root in this repository. Each workspace links its output/ directory to that user's personal output folder under the shared root.

Shared Root

Default shared root:

/data/liduan/ChartPipeline/ChartPipeline/shared_output

Each workspace should have:

output -> /data/liduan/ChartPipeline/ChartPipeline/shared_output/personal/<user>

Run the setup script from the repository root:

bash scripts/use_shared_output.sh

To use a different shared root:

CHARTPIPE_SHARED_OUTPUT=/path/to/shared_output bash scripts/use_shared_output.sh

or:

bash scripts/use_shared_output.sh /path/to/shared_output

To use a different user folder name:

CHARTPIPE_SHARED_USER=alice bash scripts/use_shared_output.sh

The script is conservative:

• If output/ is already the expected symlink, it exits successfully.
• If output/ does not exist, it creates the symlink.
• If output/ is an empty real directory, it replaces it with the symlink.
• If output/ is a non-empty real directory, it refuses to modify it. Archive or migrate those results first, then rerun the script.
• The shared top-level folders are created before the script checks output/, so it is safe to run the script even before migrating an existing local output/ directory.

Shared Directory Layout

The shared root has four top-level folders:

shared_output/
  archive/
    <commit_sha_or_initial_run>/
  beautify/
    <beautify_run>/
  manual/
  personal/
    <user>/

Folder purpose:

• archive/: release-grade archives. Store the complete test results for each Git commit here. It can also hold named initial baselines that predate the current commit archive scheme.
• beautify/: outputs beautified by LLMs or other post-processing tools.
• manual/: shared scratch space for user-managed manual operations.
• personal/<user>/: each user's default output root. The workspace output/ symlink should point here.

Current seeded shared results:

shared_output/archive/initial_chart_template_samples_20260527_083945/
shared_output/beautify/initial_chart_template_outer_gpt_image_2_improved/

Personal Output Layout

Inside personal/<user>/, use versioned subdirectories so every run is preserved:

personal/<user>/
  quality_check/
    <branch>/
      <YYYYMMDD_HHMM>_<purpose>/
  chart_template_samples/
    <branch>/
      <YYYYMMDD_HHMM>_<purpose>/
  d3_regression/
    <branch>/
      <YYYYMMDD_HHMM>_<purpose>/
  regression/
    <branch>/
      <YYYYMMDD_HHMM>_<purpose>/
  manual/

Recommended run name format:

<YYYYMMDD_HHMM>_<short-purpose>

Examples:

20260602_1430_dev_liduan_merge_acceptance
20260602_1600_d3_template_smoke
20260603_1015_title_width_regression

Common Commands

These commands write through output/, which should point to shared_output/personal/<user>/.

Quality check:

PYTHONPATH=. python scripts/run_quality_check.py \
  --plan scripts/_smoke_templates.json \
  --output-dir output/quality_check/dev-liduan/20260602_1430_template_smoke \
  --threads 8

Chart template samples:

PYTHONPATH=. python scripts/generate_template_samples.py \
  --output output/chart_template_samples/dev-liduan \
  --samples-per-template 10 \
  --output-png

D3 regression from a baseline:

PYTHONPATH=. python scripts/run_d3_regression_from_baseline.py \
  --baseline-root output/chart_template_samples/dev/20260527_083945 \
  --output-root output/d3_regression/dev-liduan/20260602_1430_merge_acceptance

D3 template audit:

PYTHONPATH=. python scripts/audit_d3_template_standardization.py \
  --node-check \
  --output-json output/manual/dev-liduan/20260602_1430_d3_audit.json \
  --output-csv output/manual/dev-liduan/20260602_1430_d3_audit.csv

Cleanup Safety

It is safe for output/ itself to be a symlink to shared_output/personal/<user>/. Avoid making individual run directories symlinks when a command may clean that exact target.

Known cleanup-sensitive cases:

• scripts/generate_template_samples.py --clean removes the selected run directory before regenerating it. Use a normal directory under shared output, not a symlinked run directory.
• tests/title_styler_regression/run.py --run <name> clears tests/title_styler_regression/results/<name> before writing a new run. Do not make that specific run directory a symlink.
• tests/infographic_regression/run.py clears the entire tests/infographic_regression/results/ directory at startup. Keep that path as a local real directory and archive the completed results into output/regression/... afterward.

Regression Results Under tests/

These scripts currently write to fixed in-repo result directories:

• tests/title_styler_regression/run.py
• tests/infographic_regression/run.py

Their result directories are ignored by Git:

tests/*_regression/results/

For day-to-day preservation, copy or move each completed run into the user's personal output:

mkdir -p output/regression/dev-liduan/20260602_1430_title_styler
cp -a tests/title_styler_regression/results/. \
  output/regression/dev-liduan/20260602_1430_title_styler/

mkdir -p output/regression/dev-liduan/20260602_1430_infographic
cp -a tests/infographic_regression/results/. \
  output/regression/dev-liduan/20260602_1430_infographic/

If these regression scripts become part of the regular release flow, the next step should be adding an environment variable such as CHARTPIPE_OUTPUT_ROOT so they can write directly under shared output.

Commit Archives

For version acceptance, store complete results by commit under top-level archive/:

shared_output/archive/<commit_sha>/
  summary.md
  quality_check/
  chart_template_samples/
  d3_regression/
  regression/
  beautify_manifest.txt

Use the full Git SHA when possible. If a run starts before the commit exists, use a temporary branch/run name first, then move or copy it into archive/<commit_sha>/ after the commit is created.

When a commit archive has a corresponding beautified result, keep the heavy beautified output under shared_output/beautify/<beautify_run>/ and reference it from the commit archive's summary.md or beautify_manifest.txt. This avoids duplicating large image outputs under both archive/ and beautify/.

Git Policy

output/ stays ignored and should not be committed. The shared output root is the source of truth for generated artifacts, while Git tracks only source code, scripts, and documentation.

Before committing, check:

git status --short --branch --untracked-files=all

Generated files under output/ should not appear in the status output.