social-data-attribution-w2/docs/VISUALIZATION_INVENTORY.md

Visualization Inventory

Generated 2026-05-09. Catalogues every notebook, script, module, CLI entrypoint, and output directory in this repo that produces a visualization (chart, plot, heatmap, figure, HTML report, or LaTeX figure file). Intended as a handoff for an external agent that needs to re-generate or edit these visualizations.

Excludes: vendored deps (bergson/, olmes/), tests, build artifacts, virtualenv.

Quick orientation

• Primary library: Plotly (interactive HTML + static PNG/PDF via Kaleido). Matplotlib is used in vendored Bergson benchmarks only — repo code is Plotly-first.
• Central paper-figure engine: src/dolma/distribution_report/. One module per figure type (heatmap, composite, radar, marginal, etc.) plus a shared style.py.
• Style canon: src/dolma/distribution_report/style.py — defines colorscales (RdBu diverging, Purples sequential), font sizes, figure dimensions, DPI scale. Edit there to change paper-wide look.
• Output convention: artifacts/ holds rendered figures; runs/ holds pipeline outputs (not figures).

---

1. Notebooks (.ipynb)

Path	Visualization purpose
notebooks/attribution_analysis.ipynb	Attribution score distributions, top-k influence inspection.
notebooks/label_distribution.ipynb	Label frequency plots (matplotlib bar/histogram).
notebooks/olmes_evaluation_analysis.ipynb	OLMES eval result charts (per-task accuracy, comparisons).
notebooks/weborganizer_topic_distribution.ipynb	WebOrganizer topic frequency visualizations.
notebooks/working_sample_playground.ipynb	Interactive exploration of the working sample (sampling, bin distributions).
artifacts/influence_bin_scores/test.ipynb	Sandbox notebook for bin-level influence scoring.

---

2. Visualization Python modules

2.1 src/dolma/distribution_report/ — paper-figure engine (Plotly)

The central package. Each module emits one family of figures and is invoked by either runner.py (sampling/marginal report) or influence_runner.py (influence/composite report).

Module	Produces
style.py	Shared layout: colorscales (RdBu, Purples), font sizes (12/22), figure dimensions, DPI scale. Edit here to change global look.
heatmap_figures.py	24×24 topic×format heatmaps for doc count, token count.
marginal_figures.py	Topic-only and format-only marginal bar charts.
sampling_figures.py	Sampling-strategy comparisons (representative vs stratified).
influence_figures.py	Per-benchmark influence heatmaps (signed).
influence_composite.py	Multi-panel signed influence composite.
influence_composite_abs.py	Absolute-value influence composite.
influence_composite_diffs.py	Difference between two conditions composite.
influence_composite_6way.py	6-way comparison across benchmarks.
influence_composite_signed_topic.py	Topic-rolled-up signed influence panels.
influence_composite_signed_6panel.py	6-panel signed-influence composite.
influence_composite_socialiqa.py	SocialIQA-specific composite.
influence_comparison.py	Side-by-side benchmark comparisons.
influence_histograms.py	Influence-score distribution histograms (per benchmark + overlay).
influence_marginals.py	Influence rolled to topic-only / format-only margins.
influence_radar.py	Topic radar chart of influence profiles.
influence_facets.py	Faceted influence small-multiples.

Helper modules in the same package (NOT visualization — data loading, tables, orchestration):

• cli.py — entrypoint for data-attribution-weborganizer-report / -paper-figures.
• runner.py — orchestrates sampling/marginal report.
• influence_runner.py — orchestrates influence report.
• data_loader.py, influence_loader.py, sampling_loader.py, sampling_data.py — data ingest.
• concentration_table.py, table_c1.py, influence_tables.py — LaTeX/CSV table emit.
• manifest_bridge.py — converts manifest parquet → EDA CSVs (used by SOC-135).
• metrics.py, report_metadata.py, report_summary.py, output_note.py, dummy_sampling.py — analysis/reporting glue.

2.2 src/dolma/eda/ — EDA HTML report

Module	Produces
charts.py	Plotly bar charts, heatmaps for topic/format distributions on EDA inputs.
report.py	HTML EDA report (artifacts/dolma_eda/report.html).

CLI: dolma-eda (entry: dolma.eda.core:main).

2.3 src/dolma/corpus_stats/ — corpus statistics report

Module	Produces
charts.py	Plotly histograms for document length statistics.
report.py	Wraps the chart pipeline.

CLI: dolma-corpus-stats (entry: dolma.corpus_stats.cli:main).

2.4 src/dolma/bin_analysis/ — bin-stratification report

Module	Produces
report.py	Plotly visualization of bin-level statistics.

CLI: data-attribution-bin-analysis (entry: dolma.bin_analysis.cli:main).

---

3. Standalone visualization scripts

Script	Output
scripts/build_corpus_explorer.py	artifacts/dolma3_corpus_explorer.html and dolma3_corpus_findings.html — interactive corpus explorer.
scripts/analysis/unlearn_report.py	artifacts/unlearn_report/unlearn_single_topic_report.html — Plotly heatmap report of single-topic unlearning gammas.
scripts/analysis/rq4_figures.py	RQ4 paper figures (lexical-feature heatmaps, profile comparison). PNG + PDF + HTML.
scripts/generate_6t_figures.py	Orchestrator: downloads 6T sample manifest from HF, runs WebOrganizer report, writes artifacts/paper_figures_6t/ and Table C1 LaTeX.

scripts/analysis/rq4_bin_characterization.py, rq4_characterize_top_bins.py, topic_missing_audit.py are characterization/data scripts, not figure generators.

---

4. CLI entrypoints (from pyproject.toml)

These are the user-facing commands that produce visualizations. All defined under [project.scripts].

Command	Module	Output dir
dolma-eda	dolma.eda.core:main	artifacts/dolma_eda/
dolma-corpus-stats	dolma.corpus_stats.cli:main	(configurable)
data-attribution-bin-analysis	dolma.bin_analysis.cli:main	(configurable)
data-attribution-weborganizer-report	dolma.distribution_report.cli:main	artifacts/paper_figures/
data-attribution-paper-figures	dolma.distribution_report.cli:main	(alias of above)

The other 43 CLI entries do data work (precache, ingest, manifest build, training-index build, scoring) and do not emit figures.

---

5. Output / artifact directories

Directory	What lives here
artifacts/paper_figures/	Main paper figures: ~150 PNG/PDF/HTML files (heatmaps, composites, comparisons, histograms, radar, format breakdowns) + table_concentration.tex, table_contrastive_socialiqa_vs_gsm8k.tex.
artifacts/paper_figures_6t/	6T working-sample regenerated figures (SOC-135): topic/format heatmaps PNG/PDF/HTML, comparison note, EDA dump.
artifacts/dolma_eda/	EDA HTML report (report.html) + plot subdir.
artifacts/influence_bin_scores/	Per-bin influence CSVs + sandbox notebook.
artifacts/influence_bin_scores_split/	Correctness-split per-bin influence CSVs.
artifacts/influence_bbh_base/, _split/, _figures/	BBH attribution scores (base model) — perquery CSVs and 51 PNG/HTML figures across the 3 BBH subtasks (snarks, causal_judgement, sports_understanding).
artifacts/influence_bbh_instruct/, _split/, _figures/	Same as above for instruct model.
artifacts/evaluation/	Evaluation result tables / charts.
artifacts/unlearn_report/	Unlearning HTML report + supporting data.
artifacts/proponent_examples/	Top-proponent example dumps.
artifacts/dolma3_corpus_explorer.html, dolma3_corpus_findings.html	Standalone interactive HTMLs from scripts/build_corpus_explorer.py.

---

6. LaTeX figure-bearing files

File	Contents
artifacts/results.tex	Results section with \includegraphics references into artifacts/paper_figures/.
artifacts/methodology.tex	Methodology section.
artifacts/paper_figures/table_concentration.tex	Concentration table (auto-generated by concentration_table.py).
artifacts/paper_figures/table_contrastive_socialiqa_vs_gsm8k.tex	Contrastive comparison table.
artifacts/paper_figures_6t/	Additional .tex fragments (Table C1) emitted by table_c1.py.

---

7. Datasets and benchmarks visualized

For figure context — what each visualization shows.

Query benchmarks (rows in influence figures):

• SocialIQA — main social-reasoning target
• GSM8K — math reference (correct vs incorrect splits)
• MMLU Social Sciences, MMLU STEM
• ARC Easy, ARC Challenge
• BBH Snarks, BBH Causal Judgement, BBH Sports Understanding (added in SOC-170)

Corpus / training-document axis:

• WebOrganizer 24 topics × 24 formats = 576 bins
• 6T deduplicated working sample (10K docs/bin sample published as HCAI-Lab/dolma3-6t-sample-10000-docs)
• Sampling strategies: representative vs stratified (visualized by sampling_figures.py)

Conditions compared:

• Base model vs Instruct model (separate _base / _instruct artifact dirs)
• Canonical vs non-canonical query variants (*_canonical.png)
• Signed vs absolute influence
• Pre vs post unlearning (in unlearn_report/)

---

8. Style and conventions (read this first when editing figures)

src/dolma/distribution_report/style.py:

• Diverging colorscale: RdBu (red = negative/degradation, blue = positive/improvement).
• Sequential colorscale: Purples (used for PPL panels and absolute-value heatmaps).
• Comparison colorscale (defined inline): green → white → purple, for paired-condition deltas.
• Font: 12 pt body, 22 pt title.
• Figure dimensions: 900×650 (bar default), 1100×900 (heatmap), 1600×900 (side-by-side).
• DPI scale factor: 3 (Plotly→Kaleido PNG export at 3× pixel density).
• Empty-cell color: #f0f0f0.

To regenerate any figure with a tweaked palette, change values in style.py and re-run the relevant CLI (data-attribution-paper-figures for the influence/sampling figures, dolma-eda for the EDA report). Each figure module reads the constants — no per-module color overrides.

---

9. Regeneration recipes (high level)

To regenerate	Run
All paper figures	uv run data-attribution-paper-figures --eda-dir artifacts/dolma_eda --output-dir artifacts/paper_figures --run-label <label> --format all
6T sample figures	uv run python scripts/generate_6t_figures.py
EDA report	uv run dolma-eda
Bin analysis	uv run data-attribution-bin-analysis
Corpus explorer HTML	uv run python scripts/build_corpus_explorer.py
Unlearn report	uv run python scripts/analysis/unlearn_report.py
RQ4 figures	uv run python scripts/analysis/rq4_figures.py
BBH influence figures	Triggered by the SOC-170 BBH scoring SLURM pipeline; figure step calls modules in src/dolma/distribution_report/.

See README.md § "WebOrganizer report" and docs/WEBORGANIZER_REPORT.md for the full report workflow with input flags.