add quality score analysis

app/src/content/chapters/analyses.mdx

@@ -134,4 +134,25 @@ GPU time across our 65 experiments varies by two orders of magnitude: the cheape
 />
 </Wide>
 
+
+
+
+### How Do Quality Scores Shift Through Rephrasing?
+
+The correlation analysis above tells us that quality scores are weak predictors of performance, but it does not show *how* scores change through rephrasing. <FigRef target="score-shift" /> visualizes this as a slope chart: each experiment is a line connecting its input score (left), output score (middle), and downstream `agg_score_macro` (right). Toggle between DCLM and edu-score views to see both perspectives.
+
+**DCLM scores almost universally increase through rephrasing.** Nearly every experiment shows an upward slope from input to output DCLM score, regardless of prompt type or model. This makes sense: the rephrasing models produce cleaner, more structured text that the DCLM classifier rewards. But the slope from output DCLM score to downstream performance is much flatter and noisier, confirming that a high DCLM score does not guarantee good training data.
+
+**Edu-scores tell the opposite story.** Most experiments *decrease* the edu-score through rephrasing, particularly those starting from high-quality sources (FineWeb-Edu-HQ has high baseline edu-scores). The edu-score classifier penalizes format changes like tables, FAQs, and math notation that our best prompts produce. This is a case where the proxy metric actively misleads: the "quality degradation" measured by edu-score corresponds to format transformations that *improve* downstream performance.
+
+{/* Seven early runs have incorrect input quality scores due to a scoring pipeline bug and
+    are excluded in the chart JS via BROKEN_INPUT_SCORES rather than patched in the JSON:
+    article/commentary/discussion/tutorial-1b-hq, tutorial-12b-hq, faq-1b-lq, faq-12b-lq */}
+<Wide>
+<HtmlEmbed
+  id="score-shift"
+  src="score-shift.html"
+  data="rephrasing_metadata.json"
+  desc="Slope chart showing how quality scores shift through rephrasing. Each line connects an experiment's input score, output score, and downstream performance. Toggle between DCLM and edu-score views."
+/>
 </Wide>

app/src/content/chapters/experiments.mdx

@@ -7,7 +7,13 @@ import FigRef from "../../components/FigRef.astro";
 {/* TODO: mention the currently running finephrase rephrasing with smollm2 */}
 {/* TODO: shorten the vllm inference benchmark or put stuff into the appendix */}
 {/* TODO: potentially make a widget for data exploration: look at the same few samples generated by different models or transformed with different prompts */}
-{/* TODO: Check if we have more information in the rephrasing_metadata that we can use to do analyses */}
+{/* TODO: Standardize colors across charts in the blog post more */}
+{/* TODO: Check all the charts again in dark mode */}
+{/* TODO: In analyses make transitions better */}
+{/* TODO: Combine verbosity and compression analysis under one title */}
+{/* TODO: Combine quality score and edu/dclm score analysis under one title */}
+{/* TODO: Move conclusions to after the experiments into conclusions subsection */}
+{/* TODO: put the new analyses into context and update the intro paragraphs of the experiments and analyses accordingly */}
 
 ## Experiments
 

app/src/content/embeds/score-shift.html

@@ -0,0 +1,376 @@
+<div class="d3-score-shift" style="width:100%;margin:10px 0;min-height:420px;"></div>
+<style>
+  .d3-score-shift { font-family: system-ui, -apple-system, sans-serif; position: relative; }
+  .d3-score-shift .d3-tooltip {
+    position: absolute; top: 0; left: 0;
+    transform: translate(-9999px, -9999px);
+    pointer-events: none;
+    padding: 10px 14px; border-radius: 10px;
+    font-size: 13px; line-height: 1.4;
+    border: 1px solid var(--border-color);
+    background: var(--surface-bg); color: var(--text-color);
+    box-shadow: 0 6px 24px rgba(0,0,0,.22);
+    opacity: 0; transition: opacity .12s ease;
+    z-index: 20; max-width: 360px;
+  }
+  .d3-score-shift .controls {
+    display: flex; gap: 16px; align-items: center; justify-content: flex-end; flex-wrap: wrap;
+    margin-top: 8px;
+  }
+  .d3-score-shift .control-group {
+    display: flex; flex-direction: column; align-items: flex-start; gap: 4px;
+  }
+  .d3-score-shift .controls label {
+    font-size: 13px; font-weight: 700; color: var(--text-color);
+  }
+  .d3-score-shift .controls select {
+    font-size: 13px; padding: 6px 28px 6px 10px; border: 1px solid var(--border-color);
+    border-radius: 8px; background: var(--surface-bg); color: var(--text-color);
+    appearance: none; cursor: pointer;
+    background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='12' height='12' viewBox='0 0 12 12'%3E%3Cpath d='M3 5l3 3 3-3' stroke='%23888' stroke-width='1.5' fill='none'/%3E%3C/svg%3E");
+    background-repeat: no-repeat; background-position: right 8px center;
+  }
+  .d3-score-shift .legend {
+    display: flex; flex-direction: column; align-items: flex-start; gap: 6px; margin-top: 8px;
+  }
+  .d3-score-shift .legend-title { font-size: 13px; font-weight: 700; color: var(--text-color); }
+  .d3-score-shift .legend .items { display: flex; flex-wrap: wrap; gap: 6px 14px; }
+  .d3-score-shift .legend .item {
+    display: inline-flex; align-items: center; gap: 6px; white-space: nowrap;
+    font-size: 13px; color: var(--text-color); cursor: pointer;
+  }
+  .d3-score-shift .legend .swatch {
+    width: 14px; height: 14px; border-radius: 3px; border: 1px solid var(--border-color);
+  }
+</style>
+<script>
+(() => {
+  const ensureD3 = (cb) => {
+    if (window.d3 && typeof window.d3.select === 'function') return cb();
+    let s = document.getElementById('d3-cdn-script');
+    if (!s) { s = document.createElement('script'); s.id = 'd3-cdn-script'; s.src = 'https://cdn.jsdelivr.net/npm/d3@7/dist/d3.min.js'; document.head.appendChild(s); }
+    const onReady = () => { if (window.d3 && typeof window.d3.select === 'function') cb(); };
+    s.addEventListener('load', onReady, { once: true });
+    if (window.d3) onReady();
+  };
+
+  const bootstrap = () => {
+    const scriptEl = document.currentScript;
+    let container = scriptEl ? scriptEl.previousElementSibling : null;
+    while (container && !(container.classList && container.classList.contains('d3-score-shift'))) {
+      container = container.previousElementSibling;
+    }
+    if (!container) {
+      const cs = Array.from(document.querySelectorAll('.d3-score-shift'))
+        .filter(el => !(el.dataset && el.dataset.mounted === 'true'));
+      container = cs[cs.length - 1] || null;
+    }
+    if (!container) return;
+    if (container.dataset.mounted === 'true') return;
+    container.dataset.mounted = 'true';
+
+    let mountEl = container;
+    while (mountEl && !mountEl.getAttribute?.('data-datafiles')) mountEl = mountEl.parentElement;
+    const dataAttr = mountEl?.getAttribute?.('data-datafiles');
+    const dataPaths = dataAttr
+      ? [dataAttr.includes('/') ? dataAttr : `/data/${dataAttr}`]
+      : ['/data/rephrasing_metadata.json', './assets/data/rephrasing_metadata.json'];
+
+    const fetchFirst = async (paths) => {
+      for (const p of paths) {
+        try { const r = await fetch(p, { cache: 'no-cache' }); if (r.ok) return r.json(); } catch(_) {}
+      }
+      throw new Error('Data not found');
+    };
+
+    fetchFirst(dataPaths).then(data => buildChart(data)).catch(err => {
+      container.innerHTML = `<pre style="color:red;padding:12px;">Error loading data: ${err.message}</pre>`;
+    });
+
+    function buildChart(rawData) {
+      const SOURCE_MAP = {
+        'fineweb-edu-hq-20BT': 'FW-Edu HQ', 'fineweb-edu-lq-20BT': 'FW-Edu LQ',
+        'dclm-37BT': 'DCLM', 'cosmopedia-25BT': 'Cosmopedia'
+      };
+      const PROMPT_LABELS = {
+        'article': 'Article', 'commentary': 'Commentary', 'discussion': 'Discussion',
+        'faq': 'FAQ', 'math': 'Math', 'table': 'Table', 'tutorial': 'Tutorial',
+        'distill': 'Distill', 'diverse_qa_pairs': 'Diverse QA',
+        'extract_knowledge': 'Extract Knowledge', 'knowledge_list': 'Knowledge List',
+        'wikipedia_style_rephrasing': 'Wikipedia Style',
+        'guided_rewrite_improved': 'Guided Rewrite+', 'guided_rewrite_original': 'Guided Rewrite'
+      };
+
+      const getFamily = (m) => {
+        const ml = m.toLowerCase();
+        if (ml.includes('smollm')) return 'SmolLM2';
+        if (ml.includes('gemma')) return 'Gemma';
+        if (ml.includes('qwen')) return 'Qwen';
+        if (ml.includes('falcon')) return 'Falcon';
+        if (ml.includes('granite')) return 'Granite';
+        if (ml.includes('llama')) return 'Llama';
+        return 'Other';
+      };
+
+      const allPromptKeys = [...new Set(rawData.map(d => d.prompt.split('/')[1].replace('.md', '')))].sort();
+      const promptColors = {};
+      const cat = window.ColorPalettes ? window.ColorPalettes.getColors('categorical', allPromptKeys.length) : d3.schemeTableau10.concat(d3.schemePastel1);
+      allPromptKeys.forEach((k, i) => { promptColors[PROMPT_LABELS[k] || k] = cat[i % cat.length]; });
+
+      const SCORE_MODES = [
+        { key: 'dclm', label: 'DCLM Score', inputKey: 'input_dclm_score', outputKey: 'output_dclm_score' },
+        { key: 'edu', label: 'Edu Score', inputKey: 'input_edu_score', outputKey: 'output_edu_score' }
+      ];
+
+      // These early runs have incorrect input quality scores (pipeline bug)
+      const BROKEN_INPUT_SCORES = new Set([
+        'format/article-1b-hq', 'format/commentary-1b-hq',
+        'format/discussion-1b-hq', 'format/tutorial-1b-hq',
+        'format/tutorial-12b-hq',
+        'format/faq-1b-lq', 'format/faq-12b-lq'
+      ]);
+
+      const experiments = rawData
+        .filter(d => !BROKEN_INPUT_SCORES.has(d.run))
+        .map(d => {
+          const [, promptFile] = d.prompt.split('/');
+          const promptKey = promptFile.replace('.md', '');
+          return {
+            run: d.run,
+            prompt: PROMPT_LABELS[promptKey] || promptKey,
+            model: d.model.split('/').pop(),
+            source: SOURCE_MAP[d.source_dataset] || d.source_dataset,
+            family: getFamily(d.model),
+            inputEdu: d.input_edu_score,
+            outputEdu: d.output_edu_score,
+            inputDclm: d.input_dclm_score,
+            outputDclm: d.output_dclm_score,
+            eduDiff: d.edu_score_difference,
+            dclmDiff: d.dclm_score_difference,
+            aggMacro: d.results.agg_score_macro
+          };
+        });
+
+      let currentMode = SCORE_MODES[0].key;
+
+      const svg = d3.select(container).append('svg').attr('width', '100%').style('display', 'block');
+      const gGrid = svg.append('g');
+      const gLines = svg.append('g');
+      const gDots = svg.append('g');
+      const gAxes = svg.append('g');
+
+      let tip = container.querySelector('.d3-tooltip');
+      let tipInner;
+      if (!tip) {
+        tip = document.createElement('div'); tip.className = 'd3-tooltip';
+        tipInner = document.createElement('div'); tipInner.className = 'd3-tooltip__inner';
+        tipInner.style.textAlign = 'left';
+        tip.appendChild(tipInner); container.appendChild(tip);
+      } else { tipInner = tip.querySelector('.d3-tooltip__inner') || tip; }
+
+      const margin = { top: 36, right: 16, bottom: 16, left: 16 };
+
+      function render() {
+        const width = container.clientWidth || 800;
+        const height = Math.max(420, Math.round(width / 1.8));
+        svg.attr('width', width).attr('height', height);
+        const iw = width - margin.left - margin.right;
+        const ih = height - margin.top - margin.bottom;
+
+        const mode = SCORE_MODES.find(m => m.key === currentMode);
+        const inputKey = currentMode === 'dclm' ? 'inputDclm' : 'inputEdu';
+        const outputKey = currentMode === 'dclm' ? 'outputDclm' : 'outputEdu';
+        const diffKey = currentMode === 'dclm' ? 'dclmDiff' : 'eduDiff';
+
+        // Three column positions
+        const colPad = Math.max(50, iw * 0.08);
+        const colX = [
+          margin.left + colPad,
+          margin.left + iw / 2,
+          width - margin.right - colPad
+        ];
+        const colLabels = [
+          `Input ${mode.label}`,
+          `Output ${mode.label}`,
+          'agg_score_macro'
+        ];
+
+        // Scales for each column (all vertical, higher = better at top)
+        const inputVals = experiments.map(d => d[inputKey]);
+        const outputVals = experiments.map(d => d[outputKey]);
+        const allScoreVals = inputVals.concat(outputVals);
+        const scorePad = (d3.max(allScoreVals) - d3.min(allScoreVals)) * 0.06;
+        const scoreScale = d3.scaleLinear()
+          .domain([d3.min(allScoreVals) - scorePad, d3.max(allScoreVals) + scorePad])
+          .range([height - margin.bottom, margin.top]);
+
+        const macroVals = experiments.map(d => d.aggMacro);
+        const macroPad = (d3.max(macroVals) - d3.min(macroVals)) * 0.08;
+        const macroScale = d3.scaleLinear()
+          .domain([d3.min(macroVals) - macroPad, d3.max(macroVals) + macroPad])
+          .range([height - margin.bottom, margin.top]);
+
+        const scales = [scoreScale, scoreScale, macroScale];
+
+        const getY = (d, col) => {
+          if (col === 0) return scales[0](d[inputKey]);
+          if (col === 1) return scales[1](d[outputKey]);
+          return scales[2](d.aggMacro);
+        };
+
+        // Grid / axis lines
+        gGrid.selectAll('*').remove();
+        colX.forEach((cx, ci) => {
+          gGrid.append('line')
+            .attr('x1', cx).attr('x2', cx)
+            .attr('y1', margin.top).attr('y2', height - margin.bottom)
+            .attr('stroke', 'var(--axis-color)').attr('stroke-width', 1).attr('opacity', 0.3);
+
+          // Ticks
+          const scale = scales[ci];
+          const ticks = scale.ticks(6);
+          const fmt = ci === 2 ? d3.format('.3f') : d3.format('.2f');
+          ticks.forEach(t => {
+            const y = scale(t);
+            gGrid.append('line')
+              .attr('x1', cx - 4).attr('x2', cx + 4)
+              .attr('y1', y).attr('y2', y)
+              .attr('stroke', 'var(--tick-color)').attr('stroke-width', 0.8);
+            gGrid.append('text')
+              .attr('x', cx - 8).attr('y', y)
+              .attr('text-anchor', 'end').attr('dominant-baseline', 'central')
+              .attr('fill', 'var(--tick-color)').attr('font-size', '12px')
+              .text(fmt(t));
+          });
+
+          // Column header
+          gGrid.append('text')
+            .attr('x', cx).attr('y', margin.top - 12)
+            .attr('text-anchor', 'middle').attr('fill', 'var(--text-color)')
+            .attr('font-size', '14px').attr('font-weight', '700')
+            .text(colLabels[ci]);
+        });
+
+        // Lines connecting the three points per experiment
+        const lineGen = (d) => {
+          return `M${colX[0]},${getY(d, 0)} L${colX[1]},${getY(d, 1)} L${colX[2]},${getY(d, 2)}`;
+        };
+
+        gLines.selectAll('path').data(experiments, d => d.run).join('path')
+          .attr('d', lineGen)
+          .attr('fill', 'none')
+          .attr('stroke', d => promptColors[d.prompt] || '#999')
+          .attr('stroke-width', 1.5)
+          .attr('stroke-opacity', 0.35)
+          .attr('pointer-events', 'none');
+
+        // Dots at each column
+        const dotData = [];
+        experiments.forEach(d => {
+          [0, 1, 2].forEach(col => {
+            dotData.push({ exp: d, col, x: colX[col], y: getY(d, col) });
+          });
+        });
+
+        const rBase = Math.max(4, Math.min(7, width * 0.006));
+
+        gDots.selectAll('circle').data(dotData, d => d.exp.run + '-' + d.col).join('circle')
+          .attr('cx', d => d.x).attr('cy', d => d.y)
+          .attr('r', rBase)
+          .attr('fill', d => promptColors[d.exp.prompt] || '#999')
+          .attr('fill-opacity', 0.7)
+          .attr('stroke', d => promptColors[d.exp.prompt] || '#999')
+          .attr('stroke-width', 1)
+          .attr('stroke-opacity', 0.2)
+          .attr('cursor', 'pointer')
+          .on('mouseenter', function(ev, d) {
+            const exp = d.exp;
+            // Highlight this experiment's line and dots
+            gLines.selectAll('path')
+              .attr('stroke-opacity', p => p.run === exp.run ? 0.9 : 0.06)
+              .attr('stroke-width', p => p.run === exp.run ? 3 : 1);
+            gDots.selectAll('circle')
+              .attr('fill-opacity', dd => dd.exp.run === exp.run ? 1 : 0.08)
+              .attr('stroke-opacity', dd => dd.exp.run === exp.run ? 0.8 : 0.04)
+              .attr('r', dd => dd.exp.run === exp.run ? rBase * 1.5 : rBase);
+
+            const fmtSign = (v, p) => (v >= 0 ? '+' : '') + v.toFixed(p || 3);
+            const dCol = (v) => v >= 0 ? '#5BC0A4' : '#E889AB';
+            tipInner.innerHTML =
+              `<div style="font-weight:700;font-size:14px;margin-bottom:4px;">${exp.prompt}</div>` +
+              `<div style="font-size:12px;color:var(--muted-color);margin-bottom:6px;">` +
+              `${exp.model} · ${exp.source}</div>` +
+              `<div style="display:grid;grid-template-columns:auto 1fr;gap:2px 10px;font-size:13px;">` +
+              `<span style="color:var(--muted-color);">DCLM</span>` +
+              `<span>${exp.inputDclm.toFixed(3)} → ${exp.outputDclm.toFixed(3)} <b style="color:${dCol(exp.dclmDiff)};">${fmtSign(exp.dclmDiff)}</b></span>` +
+              `<span style="color:var(--muted-color);">Edu</span>` +
+              `<span>${exp.inputEdu.toFixed(2)} → ${exp.outputEdu.toFixed(2)} <b style="color:${dCol(exp.eduDiff)};">${fmtSign(exp.eduDiff, 2)}</b></span>` +
+              `<span style="color:var(--muted-color);">agg_score_macro</span>` +
+              `<span style="font-weight:700;">${exp.aggMacro.toFixed(4)}</span>` +
+              `</div>`;
+            tip.style.opacity = '1';
+          })
+          .on('mousemove', (ev) => {
+            const [mx, my] = d3.pointer(ev, container);
+            const bw = tip.offsetWidth || 300;
+            const bh = tip.offsetHeight || 160;
+            const ox = (mx + bw + 20 > width) ? -(bw + 12) : 14;
+            const oy = (my + bh + 20 > (height + 60)) ? -(bh + 12) : 14;
+            tip.style.transform = `translate(${Math.round(mx + ox)}px,${Math.round(my + oy)}px)`;
+          })
+          .on('mouseleave', function() {
+            gLines.selectAll('path').attr('stroke-opacity', 0.35).attr('stroke-width', 1.5);
+            gDots.selectAll('circle')
+              .attr('fill-opacity', 0.7).attr('stroke-opacity', 0.2).attr('r', rBase);
+            tip.style.opacity = '0';
+            tip.style.transform = 'translate(-9999px,-9999px)';
+          });
+      }
+
+      // Controls
+      const controls = document.createElement('div'); controls.className = 'controls';
+      const cg = document.createElement('div'); cg.className = 'control-group';
+      const lbl = document.createElement('label'); lbl.textContent = 'Score Type'; lbl.setAttribute('for', 'ss-score-select');
+      const sel = document.createElement('select'); sel.id = 'ss-score-select';
+      SCORE_MODES.forEach(m => {
+        const opt = document.createElement('option'); opt.value = m.key; opt.textContent = m.label; sel.appendChild(opt);
+      });
+      sel.value = currentMode;
+      sel.addEventListener('change', () => { currentMode = sel.value; render(); });
+      cg.appendChild(lbl); cg.appendChild(sel); controls.appendChild(cg); container.appendChild(controls);
+
+      // Legend
+      const legend = document.createElement('div'); legend.className = 'legend';
+      const ltitle = document.createElement('div'); ltitle.className = 'legend-title'; ltitle.textContent = 'Legend';
+      const items = document.createElement('div'); items.className = 'items';
+      const usedPrompts = [...new Set(experiments.map(d => d.prompt))].sort();
+      usedPrompts.forEach(p => {
+        const el = document.createElement('span'); el.className = 'item';
+        const sw = document.createElement('span'); sw.className = 'swatch'; sw.style.background = promptColors[p];
+        const txt = document.createElement('span'); txt.textContent = p;
+        el.appendChild(sw); el.appendChild(txt); items.appendChild(el);
+        el.addEventListener('mouseenter', () => {
+          gLines.selectAll('path')
+            .attr('stroke-opacity', d => d.prompt === p ? 0.8 : 0.04)
+            .attr('stroke-width', d => d.prompt === p ? 2.5 : 1);
+          gDots.selectAll('circle')
+            .attr('fill-opacity', d => d.exp.prompt === p ? 0.9 : 0.06)
+            .attr('stroke-opacity', d => d.exp.prompt === p ? 0.6 : 0.03);
+        });
+        el.addEventListener('mouseleave', () => {
+          gLines.selectAll('path').attr('stroke-opacity', 0.35).attr('stroke-width', 1.5);
+          gDots.selectAll('circle').attr('fill-opacity', 0.7).attr('stroke-opacity', 0.2);
+        });
+      });
+      legend.appendChild(ltitle); legend.appendChild(items); container.appendChild(legend);
+
+      render();
+      if (window.ResizeObserver) new ResizeObserver(() => render()).observe(container);
+      else window.addEventListener('resize', render);
+    }
+  };
+
+  if (document.readyState === 'loading') document.addEventListener('DOMContentLoaded', () => ensureD3(bootstrap), { once: true });
+  else ensureD3(bootstrap);
+})();
+</script>