docs/shared_output.md

# Shared Output Directory

This project keeps generated charts, regression outputs, and quality-check
artifacts out of Git. For multi-user development, use one shared output root in
this repository. Each workspace links its `output/` directory to that user's
personal output folder under the shared root.

## Shared Root

Default shared root:

```bash
/data/liduan/ChartPipeline/ChartPipeline/shared_output
```

Each workspace should have:

```bash
output -> /data/liduan/ChartPipeline/ChartPipeline/shared_output/personal/<user>
```

Run the setup script from the repository root:

```bash
bash scripts/use_shared_output.sh
```

To use a different shared root:

```bash
CHARTPIPE_SHARED_OUTPUT=/path/to/shared_output bash scripts/use_shared_output.sh
```

or:

```bash
bash scripts/use_shared_output.sh /path/to/shared_output
```

To use a different user folder name:

```bash
CHARTPIPE_SHARED_USER=alice bash scripts/use_shared_output.sh
```

The script is conservative:

- If `output/` is already the expected symlink, it exits successfully.
- If `output/` does not exist, it creates the symlink.
- If `output/` is an empty real directory, it replaces it with the symlink.
- If `output/` is a non-empty real directory, it refuses to modify it. Archive
  or migrate those results first, then rerun the script.
- The shared top-level folders are created before the script checks `output/`,
  so it is safe to run the script even before migrating an existing local
  `output/` directory.

## Shared Directory Layout

The shared root has four top-level folders:

```text
shared_output/
  archive/
    <commit_sha_or_initial_run>/
  beautify/
    <beautify_run>/
  manual/
  personal/
    <user>/
```

Folder purpose:

- `archive/`: release-grade archives. Store the complete test results for each
  Git commit here. It can also hold named initial baselines that predate the
  current commit archive scheme.
- `beautify/`: outputs beautified by LLMs or other post-processing tools.
- `manual/`: shared scratch space for user-managed manual operations.
- `personal/<user>/`: each user's default output root. The workspace `output/`
  symlink should point here.

Current seeded shared results:

```text
shared_output/archive/initial_chart_template_samples_20260527_083945/
shared_output/beautify/initial_chart_template_outer_gpt_image_2_improved/
```

## Personal Output Layout

Inside `personal/<user>/`, use versioned subdirectories so every run is
preserved:

```text
personal/<user>/
  quality_check/
    <branch>/
      <YYYYMMDD_HHMM>_<purpose>/
  chart_template_samples/
    <branch>/
      <YYYYMMDD_HHMM>_<purpose>/
  d3_regression/
    <branch>/
      <YYYYMMDD_HHMM>_<purpose>/
  regression/
    <branch>/
      <YYYYMMDD_HHMM>_<purpose>/
  manual/
```

Recommended run name format:

```text
<YYYYMMDD_HHMM>_<short-purpose>
```

Examples:

```text
20260602_1430_dev_liduan_merge_acceptance
20260602_1600_d3_template_smoke
20260603_1015_title_width_regression
```

## Common Commands

These commands write through `output/`, which should point to
`shared_output/personal/<user>/`.

Quality check:

```bash
PYTHONPATH=. python scripts/run_quality_check.py \
  --plan scripts/_smoke_templates.json \
  --output-dir output/quality_check/dev-liduan/20260602_1430_template_smoke \
  --threads 8
```

Chart template samples:

```bash
PYTHONPATH=. python scripts/generate_template_samples.py \
  --output output/chart_template_samples/dev-liduan \
  --samples-per-template 10 \
  --output-png
```

D3 regression from a baseline:

```bash
PYTHONPATH=. python scripts/run_d3_regression_from_baseline.py \
  --baseline-root output/chart_template_samples/dev/20260527_083945 \
  --output-root output/d3_regression/dev-liduan/20260602_1430_merge_acceptance
```

D3 template audit:

```bash
PYTHONPATH=. python scripts/audit_d3_template_standardization.py \
  --node-check \
  --output-json output/manual/dev-liduan/20260602_1430_d3_audit.json \
  --output-csv output/manual/dev-liduan/20260602_1430_d3_audit.csv
```

## Cleanup Safety

It is safe for `output/` itself to be a symlink to
`shared_output/personal/<user>/`. Avoid making individual run directories
symlinks when a command may clean that exact target.

Known cleanup-sensitive cases:

- `scripts/generate_template_samples.py --clean` removes the selected run
  directory before regenerating it. Use a normal directory under shared output,
  not a symlinked run directory.
- `tests/title_styler_regression/run.py --run <name>` clears
  `tests/title_styler_regression/results/<name>` before writing a new run. Do
  not make that specific run directory a symlink.
- `tests/infographic_regression/run.py` clears the entire
  `tests/infographic_regression/results/` directory at startup. Keep that path
  as a local real directory and archive the completed results into
  `output/regression/...` afterward.

## Regression Results Under `tests/`

These scripts currently write to fixed in-repo result directories:

- `tests/title_styler_regression/run.py`
- `tests/infographic_regression/run.py`

Their result directories are ignored by Git:

```text
tests/*_regression/results/
```

For day-to-day preservation, copy or move each completed run into the user's
personal output:

```bash
mkdir -p output/regression/dev-liduan/20260602_1430_title_styler
cp -a tests/title_styler_regression/results/. \
  output/regression/dev-liduan/20260602_1430_title_styler/

mkdir -p output/regression/dev-liduan/20260602_1430_infographic
cp -a tests/infographic_regression/results/. \
  output/regression/dev-liduan/20260602_1430_infographic/
```

If these regression scripts become part of the regular release flow, the next
step should be adding an environment variable such as `CHARTPIPE_OUTPUT_ROOT` so
they can write directly under shared output.

## Commit Archives

For version acceptance, store complete results by commit under top-level
`archive/`:

```text
shared_output/archive/<commit_sha>/
  summary.md
  quality_check/
  chart_template_samples/
  d3_regression/
  regression/
  beautify_manifest.txt
```

Use the full Git SHA when possible. If a run starts before the commit exists,
use a temporary branch/run name first, then move or copy it into
`archive/<commit_sha>/` after the commit is created.

When a commit archive has a corresponding beautified result, keep the heavy
beautified output under `shared_output/beautify/<beautify_run>/` and reference it
from the commit archive's `summary.md` or `beautify_manifest.txt`. This avoids
duplicating large image outputs under both `archive/` and `beautify/`.

## Git Policy

`output/` stays ignored and should not be committed. The shared output root is
the source of truth for generated artifacts, while Git tracks only source code,
scripts, and documentation.

Before committing, check:

```bash
git status --short --branch --untracked-files=all
```

Generated files under `output/` should not appear in the status output.