main.tex

\documentclass[11pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{float}
\usepackage[hidelinks]{hyperref}
\usepackage{caption}
\captionsetup{font=small}
\title{Cross-Model TrackStar Attribution Comparison\\
\large Comma v0.1 1T, Comma v0.1 2T, and OLMo-3 7B base}
\date{2026-07-01}
\begin{document}
\maketitle

\section*{Executive summary}
This report compares training-influence profiles for three models across four evaluation probes (SocialIQA, MMLU Social Sciences, MMLU STEM, ARC-Challenge). All models share the 24$\times$24 WebOrganizer topic$\times$format taxonomy but attribute over different corpora (Comma: Common Pile; OLMo-3 7B: Dolma), so the comparison is of influence profiles over a shared category taxonomy, not shared documents. Signed influence is z-scored within each probe; the analysis is descriptive, not causal.

\paragraph{Cross-model alignment (Spearman of z-scored bin profiles, pooled over probes).}
\begin{center}
\begin{tabular}{lrrr}\toprule
 & Comma v0.1 1T & Comma v0.1 2T & OLMo-3 7B base \\\midrule
Comma v0.1 1T & 1.00 & 0.39 & 0.02 \\
Comma v0.1 2T & 0.39 & 1.00 & 0.01 \\
OLMo-3 7B base & 0.02 & 0.01 & 1.00 \\
\bottomrule\end{tabular}
\end{center}

The two Comma checkpoints are moderately aligned with each other and essentially uncorrelated with OLMo-3 7B, indicating the models rely on different content categories for these probes.

\section{Methodology and framing}
Influence is TrackStar gradient-dot attribution, aggregated to the 576 WebOrganizer bins (mean per bin), fused during scoring. Per-probe signed influence is z-scored within each model so cross-model comparison reflects relative structure; raw magnitudes differ by corpus and scale and are not compared directly. Rankings report the most helpful (positive) and most harmful (negative) topics and formats. Alignment is the Spearman correlation of the 576-bin z-scored profiles.

\section{SocialIQA deep dive: are the sources of capability shared?}
SocialIQA is the primary social-reasoning target. This section asks whether the models draw their SocialIQA capability from similar training content. The table reports Pearson and Spearman correlation of the models' z-scored SocialIQA influence at three levels: the 576 topic$\times$format bins, the 24 topic marginals, and the 24 format marginals.

\begin{center}
\begin{tabular}{llrr}\toprule
Model pair & Level & Pearson & Spearman \\\midrule
Comma v0.1 1T vs Comma v0.1 2T & bins & +0.09 & +0.12 \\
Comma v0.1 1T vs Comma v0.1 2T & topics & -0.01 & +0.02 \\
Comma v0.1 1T vs Comma v0.1 2T & formats & -0.20 & -0.10 \\
Comma v0.1 1T vs OLMo-3 7B base & bins & -0.05 & -0.12 \\
Comma v0.1 1T vs OLMo-3 7B base & topics & -0.05 & -0.06 \\
Comma v0.1 1T vs OLMo-3 7B base & formats & -0.36 & -0.30 \\
Comma v0.1 2T vs OLMo-3 7B base & bins & +0.00 & +0.02 \\
Comma v0.1 2T vs OLMo-3 7B base & topics & +0.01 & +0.06 \\
Comma v0.1 2T vs OLMo-3 7B base & formats & +0.29 & +0.36 \\
\bottomrule\end{tabular}
\end{center}
All pairwise correlations are near zero: the models reach SocialIQA capability from largely different content distributions, and even the two Comma checkpoints diverge on this probe. The strongest signal is Comma 2T vs.\ OLMo-3 7B at the format level. At the topic level OLMo-3 7B is driven by Literature and Social Life; Comma 2T shifts toward the same topics (Social Life ranks near the top), while Comma 1T does not (its top topics are Sports and Transportation). So the models partly agree on which social topics matter but not on the overall source distribution.

\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/comparison/figures/overlaid_topic_socialiqa.png}
\caption{SocialIQA topic influence by model.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/comparison/figures/overlaid_format_socialiqa.png}
\caption{SocialIQA format influence by model.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.55\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/comparison/figures/alignment_socialiqa.png}
\caption{SocialIQA cross-model bin alignment (Spearman).}
\end{figure}

\paragraph{Comma v0.1 1T --- SocialIQA topics.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & Sports And Fitness & +0.53 & Adult Content & -0.56 \\
2 & Transportation & +0.40 & Games & -0.38 \\
3 & Software & +0.21 & Entertainment & -0.36 \\
4 & Travel And Tourism & +0.20 & Health & -0.32 \\
5 & Fashion And Beauty & +0.19 & Food And Dining & -0.30 \\
6 & Literature & +0.18 & Religion & -0.22 \\
\bottomrule\end{tabular}
\end{center}
\paragraph{Comma v0.1 1T --- SocialIQA formats.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & Truncated & +1.39 & Q\&A Forum & -0.90 \\
2 & Legal Notices & +1.30 & FAQ & -0.87 \\
3 & Spam Ads & +0.98 & Knowledge Article & -0.54 \\
4 & Audio Transcript & +0.62 & User Review & -0.43 \\
5 & Creative Writing & +0.28 & News Article & -0.40 \\
6 & Nonfiction Writing & +0.21 & News Org & -0.40 \\
\bottomrule\end{tabular}
\end{center}
\paragraph{Comma v0.1 2T --- SocialIQA topics.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & Games & +0.92 & Industrial & -0.60 \\
2 & Social Life & +0.62 & Software Development & -0.55 \\
3 & History And Geography & +0.46 & Food And Dining & -0.52 \\
4 & Entertainment & +0.44 & Finance And Business & -0.52 \\
5 & Literature & +0.39 & Electronics And Hardware & -0.43 \\
6 & Fashion And Beauty & +0.39 & Health & -0.40 \\
\bottomrule\end{tabular}
\end{center}
\paragraph{Comma v0.1 2T --- SocialIQA formats.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & About Person & +1.08 & Audio Transcript & -1.01 \\
2 & User Review & +1.01 & Documentation & -0.64 \\
3 & FAQ & +0.42 & Personal Blog & -0.55 \\
4 & Comment Section & +0.32 & News Org & -0.38 \\
5 & Customer Support & +0.32 & Knowledge Article & -0.36 \\
6 & Creative Writing & +0.31 & Truncated & -0.34 \\
\bottomrule\end{tabular}
\end{center}
\paragraph{OLMo-3 7B base --- SocialIQA topics.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & Literature & +0.67 & Entertainment & -0.50 \\
2 & Industrial & +0.66 & Software Development & -0.36 \\
3 & Health & +0.53 & Travel And Tourism & -0.36 \\
4 & Social Life & +0.42 & Software & -0.36 \\
5 & Home And Hobbies & +0.28 & Electronics And Hardware & -0.35 \\
6 & Science Math And Technology & +0.16 & Games & -0.20 \\
\bottomrule\end{tabular}
\end{center}
\paragraph{OLMo-3 7B base --- SocialIQA formats.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & Customer Support & +1.13 & News Article & -0.56 \\
2 & FAQ & +0.42 & Knowledge Article & -0.52 \\
3 & Q\&A Forum & +0.38 & Legal Notices & -0.45 \\
4 & Comment Section & +0.27 & Nonfiction Writing & -0.34 \\
5 & About Person & +0.23 & Truncated & -0.33 \\
6 & Documentation & +0.21 & Spam Ads & -0.23 \\
\bottomrule\end{tabular}
\end{center}
\section{Comma v0.1 1T}
\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_1t/fig_influence_signed_2x2.png}
\caption{Comma v0.1 1T: signed influence, all four probes (z-scored).}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_1t/fig_influence_diff_correct_2x2.png}
\caption{Comma v0.1 1T: correct minus incorrect influence, all four probes.}
\end{figure}

\paragraph{Top topics.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & Crime And Law & +0.45 & Home And Hobbies & -0.53 \\
2 & Literature & +0.39 & Software Development & -0.44 \\
3 & History And Geography & +0.29 & Food And Dining & -0.32 \\
4 & Science Math And Technology & +0.27 & Industrial & -0.27 \\
5 & Social Life & +0.24 & Adult Content & -0.23 \\
\bottomrule\end{tabular}
\end{center}
\paragraph{Top formats.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & Legal Notices & +0.81 & Personal Blog & -0.83 \\
2 & About Person & +0.75 & FAQ & -0.52 \\
3 & Customer Support & +0.37 & Listicle & -0.36 \\
4 & Nonfiction Writing & +0.30 & Knowledge Article & -0.32 \\
5 & Q\&A Forum & +0.21 & News Org & -0.32 \\
\bottomrule\end{tabular}
\end{center}
\section{Comma v0.1 2T}
\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_2t/fig_influence_signed_2x2.png}
\caption{Comma v0.1 2T: signed influence, all four probes (z-scored).}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_2t/fig_influence_diff_correct_2x2.png}
\caption{Comma v0.1 2T: correct minus incorrect influence, all four probes.}
\end{figure}

\paragraph{Top topics.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & Games & +0.72 & Adult Content & -0.53 \\
2 & History And Geography & +0.53 & Electronics And Hardware & -0.53 \\
3 & Crime And Law & +0.46 & Food And Dining & -0.44 \\
4 & Literature & +0.41 & Finance And Business & -0.42 \\
5 & Social Life & +0.33 & Sports And Fitness & -0.40 \\
\bottomrule\end{tabular}
\end{center}
\paragraph{Top formats.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & About Person & +1.10 & Audio Transcript & -0.81 \\
2 & User Review & +0.96 & Documentation & -0.62 \\
3 & Comment Section & +0.71 & Personal Blog & -0.55 \\
4 & FAQ & +0.40 & Truncated & -0.53 \\
5 & Structured Data & +0.32 & Knowledge Article & -0.33 \\
\bottomrule\end{tabular}
\end{center}
\section{OLMo-3 7B base}
\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/report_olmo7b_base/fig_influence_signed_2x2.png}
\caption{OLMo-3 7B base: signed influence, all four probes (z-scored).}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/report_olmo7b_base/fig_influence_diff_correct_2x2.png}
\caption{OLMo-3 7B base: correct minus incorrect influence, all four probes.}
\end{figure}

\paragraph{Top topics.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & Industrial & +0.94 & Entertainment & -0.39 \\
2 & Science Math And Technology & +0.62 & Sports And Fitness & -0.33 \\
3 & Health & +0.46 & Religion & -0.28 \\
4 & Politics & +0.27 & History And Geography & -0.22 \\
5 & Software Development & +0.15 & Games & -0.21 \\
\bottomrule\end{tabular}
\end{center}
\paragraph{Top formats.}\begin{center}
\begin{tabular}{rlr@{\hskip 2em}lr}\toprule
\# & \multicolumn{2}{c}{Most helpful} & \multicolumn{2}{c}{Most harmful} \\\midrule
1 & Documentation & +1.71 & News Article & -0.72 \\
2 & Legal Notices & +0.58 & Listicle & -0.39 \\
3 & Academic Writing & +0.54 & News Org & -0.37 \\
4 & Q\&A Forum & +0.18 & Nonfiction Writing & -0.36 \\
5 & Structured Data & +0.18 & Comment Section & -0.30 \\
\bottomrule\end{tabular}
\end{center}
\section{Cross-model comparison}
\begin{figure}[htbp]\centering
\includegraphics[width=0.6\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/comparison/figures/alignment_pooled.png}
\caption{Cross-model alignment (Spearman, pooled over probes).}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/comparison/figures/overlaid_topic_socialiqa.png}
\caption{Topic influence by model: SocialIQA.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/comparison/figures/overlaid_topic_mmlu_social_science.png}
\caption{Topic influence by model: MMLU Social Sciences.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/comparison/figures/overlaid_topic_mmlu_stem.png}
\caption{Topic influence by model: MMLU STEM.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.92\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/comparison/figures/overlaid_topic_arc_challenge.png}
\caption{Topic influence by model: ARC-Challenge.}
\end{figure}

\paragraph{Shared vs.\ divergent top helpful bins (SocialIQA).}\begin{center}
\begin{tabular}{llll}\toprule
Topic & Format & \# models & Status \\\midrule
Fashion And Beauty & Spam Ads & 2 & partial \\
Crime And Law & Truncated & 2 & partial \\
Industrial & Legal Notices & 2 & partial \\
Crime And Law & Customer Support & 2 & partial \\
Adult Content & FAQ & 2 & partial \\
Social Life & About Person & 2 & partial \\
Travel And Tourism & Legal Notices & 1 & divergent \\
Sports And Fitness & Truncated & 1 & divergent \\
Adult Content & About Person & 1 & divergent \\
Health & Legal Notices & 1 & divergent \\
Science Math And Technology & Legal Notices & 1 & divergent \\
Transportation & Legal Notices & 1 & divergent \\
\bottomrule\end{tabular}
\end{center}
\section*{Appendix: per-probe signed heatmaps}
\subsection*{Comma v0.1 1T}
\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_1t/fig_influence_signed_socialiqa_canonical.png}
\caption{Comma v0.1 1T: SocialIQA signed influence.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_1t/fig_influence_signed_mmlu_social_science_canonical.png}
\caption{Comma v0.1 1T: MMLU Social Sciences signed influence.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_1t/fig_influence_signed_mmlu_stem_canonical.png}
\caption{Comma v0.1 1T: MMLU STEM signed influence.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_1t/fig_influence_signed_arc_challenge_canonical.png}
\caption{Comma v0.1 1T: ARC-Challenge signed influence.}
\end{figure}

\subsection*{Comma v0.1 2T}
\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_2t/fig_influence_signed_socialiqa_canonical.png}
\caption{Comma v0.1 2T: SocialIQA signed influence.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_2t/fig_influence_signed_mmlu_social_science_canonical.png}
\caption{Comma v0.1 2T: MMLU Social Sciences signed influence.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_2t/fig_influence_signed_mmlu_stem_canonical.png}
\caption{Comma v0.1 2T: MMLU STEM signed influence.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/comma_attribution/report_comma_2t/fig_influence_signed_arc_challenge_canonical.png}
\caption{Comma v0.1 2T: ARC-Challenge signed influence.}
\end{figure}

\subsection*{OLMo-3 7B base}
\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/report_olmo7b_base/fig_influence_signed_socialiqa_canonical.png}
\caption{OLMo-3 7B base: SocialIQA signed influence.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/report_olmo7b_base/fig_influence_signed_mmlu_social_science_canonical.png}
\caption{OLMo-3 7B base: MMLU Social Sciences signed influence.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/report_olmo7b_base/fig_influence_signed_mmlu_stem_canonical.png}
\caption{OLMo-3 7B base: MMLU STEM signed influence.}
\end{figure}

\begin{figure}[htbp]\centering
\includegraphics[width=0.8\textwidth]{/Users/glenn/dev/social-data-attribution/.claude/worktrees/crazy-carson-b9bb77/artifacts/model_comparison/report_olmo7b_base/fig_influence_signed_arc_challenge_canonical.png}
\caption{OLMo-3 7B base: ARC-Challenge signed influence.}
\end{figure}

\end{document}