removed figrefs and changed the setup for charts: first set the scene, then show the plot and finally analyze the resutls and do the transition.

app/src/content/chapters/1-introduction.mdx

@@ -4,14 +4,13 @@ import HtmlEmbed from "../../components/HtmlEmbed.astro";
 import Wide from "../../components/Wide.astro";
 import Sidenote from "../../components/Sidenote.astro";
 import Note from "../../components/Note.astro";
-import FigRef from "../../components/FigRef.astro";
 import syntheticDataScaleImg from "../assets/image/synthetic-data-scale.jpg";
 
 
 ## Introduction
 
 
-We ran 90 experiments, generated over 1 trillion tokens, and spent 12.7 GPU years to find the best recipe for synthetic pretraining data. The result is **FinePhrase**, a 486B token dataset that clearly outperforms all existing synthetic data baselines (<FigRef target="finephrase-vs-baselines" />). It's [available on the Hub](https://huggingface.co/datasets/HuggingFaceFW/finephrase), and this post walks you through everything we learned along the way.
+We ran 90 experiments, generated over 1 trillion tokens, and spent 12.7 GPU years to find the best recipe for synthetic pretraining data. The result is **FinePhrase**, a 486B token dataset that clearly outperforms all existing synthetic data baselines. It's [available on the Hub](https://huggingface.co/datasets/HuggingFaceFW/finephrase), and this post walks you through everything we learned along the way.
 <Sidenote>
 Reading time: One weekend
 </Sidenote>
@@ -40,7 +39,7 @@ If you read some of the latest LLM papers (e.g., Nemotron 3 [@nemotron3], Qwen3
 
 We are seeing a radical shift in compute allocation for model training: while the model training dominated the compute budget early on, we see more and more compute allocated to curate and improve the training datasets, both in pretraining and post-training.
 
-The scale is staggering: NVIDIA used LLMs to rephrase around 2 trillion tokens of web text for their [Nemotron-CC dataset](https://huggingface.co/datasets/nvidia/Nemotron-CC-v2) [@nemotroncc], while Z.ai generated 500 billion reasoning tokens to mid-train the GLM-4.5 series [@glm45]. <FigRef target="synthetic-data-scale" /> shows just how much synthetic data recent models are using.
+The scale is staggering: NVIDIA used LLMs to rephrase around 2 trillion tokens of web text for their [Nemotron-CC dataset](https://huggingface.co/datasets/nvidia/Nemotron-CC-v2) [@nemotroncc], while Z.ai generated 500 billion reasoning tokens to mid-train the GLM-4.5 series [@glm45]. Here's how much synthetic data recent models are using:
 
 <figure id="synthetic-data-scale">
 <Image src={syntheticDataScaleImg} alt="Scale of synthetic data in recent LLM training runs" />

app/src/content/chapters/3-experiments.mdx

@@ -2,8 +2,6 @@ import HtmlEmbed from "../../components/HtmlEmbed.astro";
 import Note from "../../components/Note.astro";
 import Sidenote from "../../components/Sidenote.astro";
 import Glossary from "../../components/Glossary.astro";
-import FigRef from "../../components/FigRef.astro";
-
 
 {/* TODO: Integrate decay experiment as another analysis for proxy */}
 {/* TODO: share on a bunch of discords/slacks/hackernews/locallama */}
@@ -18,7 +16,7 @@ Notes:
 
 ## Experiments
 
-Time to put all of this to the test. We ran 90 experiments to systematically answer our questions, and the journey took some unexpected turns. <FigRef target="experiment-overview" /> shows the full landscape: source datasets flowing through prompt strategies to model families. We start by seeing how existing datasets stack up, then dissect what makes their prompts tick. From there we design our own prompts, explore how the rephrasing model affects quality, and investigate the interplay between synthetic and original data. Along the way, we stumble into some surprising findings about typos and template collapse.
+Time to put all of this to the test. We ran 90 experiments to systematically answer our questions, and the journey took some unexpected turns. Here's the full landscape of what we explored, with source datasets flowing through prompt strategies to model families:
 
 <HtmlEmbed
   id="experiment-overview"
@@ -27,9 +25,11 @@ Time to put all of this to the test. We ran 90 experiments to systematically ans
   desc="Flow of experiments from source datasets through prompt strategies to model families. Hover over nodes and links to see experiment counts."
 />
 
+We start by seeing how existing datasets stack up, then dissect what makes their prompts tick. From there we design our own prompts, explore how the rephrasing model affects quality, and investigate the interplay between synthetic and original data. Along the way, we stumble into some surprising findings about typos and template collapse.
+
 ### How Do Existing Datasets Compare?
 
-First things first: where does the bar sit? We train on eight datasets under identical conditions and compare their final evaluation performance. DCLM, Nemotron-HQ-Synth, and REWIRE come out on top by a clear margin (see <FigRef target="baselines-comparison" />). The remaining datasets, including Cosmopedia, FineWeb-Edu (both HQ and LQ), Ultra-FineWeb, SYNTH, and EssentialWeb, fall notably behind. DCLM is the strongest baseline and becomes our target to beat for everything that follows.
+First things first: where does the bar sit? We establish baselines and train on eight popular datasets under identical conditions and compare their evaluation performance:
 
 <HtmlEmbed
   id="baselines-comparison"
@@ -50,11 +50,13 @@ First things first: where does the bar sit? We train on eight datasets under ide
   }}
 />
 
+DCLM, Nemotron-HQ-Synth, and REWIRE come out on top by a clear margin. The remaining datasets, including Cosmopedia, FineWeb-Edu (both HQ and LQ), Ultra-FineWeb, SYNTH, and EssentialWeb, fall notably behind. DCLM is the strongest baseline and becomes our target to beat for everything that follows.
+
 Nemotron-HQ-Synth and REWIRE are both mixes of several prompts. So what's actually doing the heavy lifting inside them?
 
 #### Which Individual Prompts Match DCLM?
 
-We isolate each prompt from Nemotron-HQ-Synth ([diverse_qa_pairs](#diverse_qa_pairs), [extract_knowledge](#extract_knowledge), [distill](#distill), [wikipedia_style_rephrasing](#wikipedia_style_rephrasing), [knowledge_list](#knowledge_list)), the REWIRE [guided_rewrite](#guided_rewrite_original) prompt, and the two prompts from BeyondWeb [@beyondweb] ([continue](#continue), [summarize](#summarize)), all using Gemma-3-1B on FineWeb-Edu-HQ as source. Only [diverse_qa_pairs](#diverse_qa_pairs) (driven by very strong SQuAD performance) and REWIRE's [guided_rewrite](#guided_rewrite_original) match DCLM (see <FigRef target="dissecting-baselines" />). The BeyondWeb-inspired [continue](#continue) and [summarize](#summarize) prompts don't reach DCLM level. So out of all the prompts from prior work, only two actually match our baseline.
+We isolate each prompt from Nemotron-HQ-Synth ([diverse_qa_pairs](#diverse_qa_pairs), [extract_knowledge](#extract_knowledge), [distill](#distill), [wikipedia_style_rephrasing](#wikipedia_style_rephrasing), [knowledge_list](#knowledge_list)), the REWIRE [guided_rewrite](#guided_rewrite_original) prompt, and the two prompts from BeyondWeb [@beyondweb] ([continue](#continue), [summarize](#summarize)), all using Gemma-3-1B on FineWeb-Edu-HQ as source:
 
 <Sidenote>
 The BeyondWeb dataset was never released and the paper omits key details, yet claims strong performance. We tested their [continue](#continue) and [summarize](#summarize) prompts to verify those claims and make the knowledge publicly available.
@@ -81,11 +83,11 @@ The BeyondWeb dataset was never released and the paper omits key details, yet cl
   }}
 />
 
-That's a pretty underwhelming hit rate. Can we do better with our own prompts?
+Only [diverse_qa_pairs](#diverse_qa_pairs) (driven by very strong SQuAD performance) and REWIRE's [guided_rewrite](#guided_rewrite_original) match DCLM. The BeyondWeb-inspired [continue](#continue) and [summarize](#summarize) prompts don't reach DCLM level. So out of all the prompts from prior work, only two actually match our baseline. That's a pretty underwhelming hit rate. Can we do better with our own prompts?
 
 ### Can New Prompts Beat DCLM?
 
-Since most existing prompts fail to beat DCLM, we designed nine novel prompt formats targeting different skills ([article](#article), [commentary](#commentary), [discussion](#discussion), [explanation](#explanation), [faq](#faq), [math](#math), [narrative](#narrative), [table](#table), [tutorial](#tutorial)), all using Gemma-3-1B on FineWeb-Edu-HQ. Four of them ([faq](#faq), [math](#math), [table](#table), [tutorial](#tutorial)) clearly outperform DCLM, while the other five sit at or below DCLM level (see <FigRef target="new-prompts" />). The winning prompts share a common trait: they all restructure the source content into pedagogically rich formats rather than just paraphrasing it.
+Since most existing prompts fail to beat DCLM, we designed nine novel prompt formats targeting different skills ([article](#article), [commentary](#commentary), [discussion](#discussion), [explanation](#explanation), [faq](#faq), [math](#math), [narrative](#narrative), [table](#table), [tutorial](#tutorial)), all using Gemma-3-1B on FineWeb-Edu-HQ:
 
 <HtmlEmbed
   id="new-prompts"
@@ -107,6 +109,8 @@ Since most existing prompts fail to beat DCLM, we designed nine novel prompt for
   }}
 />
 
+Four of them ([faq](#faq), [math](#math), [table](#table), [tutorial](#tutorial)) clearly outperform DCLM, while the other five sit at or below DCLM level. The winning prompts share a common trait: they all restructure the source content into pedagogically rich formats rather than just paraphrasing it.
+
 So far we've been using Gemma-3-1B for everything. A natural question is: can we squeeze out more performance by throwing a bigger or better model at the problem?
 
 ### Impact of the Rephrasing Model
@@ -115,12 +119,7 @@ We look at this from three angles: model size, model family, and model generatio
 
 #### Does the model size matter?
 
-We compare all Gemma-3 sizes (270M, 1B, 4B, 12B, 27B) on the [math](#math), [tutorial](#tutorial), and REWIRE's [guided_rewrite](#guided_rewrite_original) prompts (use the Setup dropdown in <FigRef target="model-size" /> to switch between them).
-For [math](#math) and [tutorial](#tutorial), the 270M model underperforms, but 1B through 27B show no significant difference.
-SmolLM2 (135M, 360M, 1.7B) tells the same story on [tutorial](#tutorial): there is a clear performance gradient up to the 1B range.
-The one exception is [guided_rewrite](#guided_rewrite_original), where the 4B model edges ahead of the 1B, while 4B through 27B remain equivalent.
-This prompt is substantially more complex (detailed rewriting instructions, quality criteria, multi-step formatting requirements), which likely raises the minimum capability threshold.
-The takeaway: beyond a baseline capability (reached around 1B for simple prompts and 4B for complex ones), bigger models don't buy you better synthetic data. This is great news for cost: you can use cheap, fast models for most rephrasing tasks.
+We compare all Gemma-3 sizes (270M, 1B, 4B, 12B, 27B) on the [math](#math), [tutorial](#tutorial), and REWIRE's [guided_rewrite](#guided_rewrite_original) prompts. Use the Setup dropdown to switch between them:
 
 <Sidenote>
 It is possible that larger models produce richer or more nuanced rephrasings that our benchmark suite does not capture. Our evaluations measure a fixed set of skills, and subtler improvements in data quality could go undetected.
@@ -174,11 +173,17 @@ It is possible that larger models produce richer or more nuanced rephrasings tha
   }}
 />
 
+For [math](#math) and [tutorial](#tutorial), the 270M model underperforms, but 1B through 27B show no significant difference.
+SmolLM2 (135M, 360M, 1.7B) tells the same story on [tutorial](#tutorial): there is a clear performance gradient up to the 1B range.
+The one exception is [guided_rewrite](#guided_rewrite_original), where the 4B model edges ahead of the 1B, while 4B through 27B remain equivalent.
+This prompt is substantially more complex (detailed rewriting instructions, quality criteria, multi-step formatting requirements), which likely raises the minimum capability threshold.
+The takeaway: beyond a baseline capability (reached around 1B for simple prompts and 4B for complex ones), bigger models don't buy you better synthetic data. This is great news for cost: you can use cheap, fast models for most rephrasing tasks.
+
 That raises an interesting follow-up. REWIRE claims that you specifically need large models to salvage low-quality data. Does that hold up?
 
 #### Do we need better models for rephrasing low-quality data?
 
-REWIRE [@rewire] used Llama-3.3 70B and argued that upcycling low-quality data requires large models. We put this to the test by comparing 1B vs 12B models on HQ vs LQ source data across four prompts ([continue](#continue), [summarize](#summarize), [faq](#faq), [tutorial](#tutorial)). Use the Setup dropdown to switch between prompts. The results are mixed: for some prompts 12B helps slightly with LQ data, but for the [FAQ](#faq) prompt the 1B model actually wins (see <FigRef target="size-quality" />). We see no consistent advantage of using larger models for low-quality data.
+REWIRE [@rewire] used Llama-3.3 70B and argued that upcycling low-quality data requires large models. We put this to the test by comparing 1B vs 12B models on HQ vs LQ source data across four prompts ([continue](#continue), [summarize](#summarize), [faq](#faq), [tutorial](#tutorial)). Use the Setup dropdown to switch between prompts:
 
 <HtmlEmbed
   id="size-quality"
@@ -226,11 +231,13 @@ REWIRE [@rewire] used Llama-3.3 70B and argued that upcycling low-quality data r
   }}
 />
 
+The results are mixed: for some prompts 12B helps slightly with LQ data, but for the [FAQ](#faq) prompt the 1B model actually wins. We see no consistent advantage of using larger models for low-quality data.
+
 So model size doesn't matter much. But what if you're using the wrong model family entirely?
 
 #### Does the model family matter?
 
-We test six model families (SmolLM2, Falcon3 [@falcon3], Qwen3, Gemma-3, Granite3 [@granite3], Llama-3.2) at ~1B scale on eight prompts. Use the Setup dropdown to compare across prompts. The result here is striking: SmolLM2 consistently and clearly outperforms all others across every single prompt (see <FigRef target="model-family" />). 
+We test six model families (SmolLM2, Falcon3 [@falcon3], Qwen3, Gemma-3, Granite3 [@granite3], Llama-3.2) at ~1B scale on eight prompts. Use the Setup dropdown to compare across prompts:
 
 <Sidenote>
 We hypothesize that SmolLM2's consistently strong rephrasing performance originates from explicit [rewrite tasks](https://huggingface.co/datasets/HuggingFaceTB/smoltalk/viewer/smol-rewrite?row=0&views%5B%5D=smol_rewrite_train) in its instruction tuning data (smoltalk). This would mean the model already "knows" how to rewrite well before we even prompt it.
@@ -334,6 +341,8 @@ We hypothesize that SmolLM2's consistently strong rephrasing performance origina
   }}
 />
 
+The result is striking: SmolLM2 consistently and clearly outperforms all others across every single prompt.
+
 SmolLM2 is already over a year old at this point. If model quality matters, should we just wait for the next generation?
 <Sidenote>
 [SmolLM3](https://huggingface.co/HuggingFaceTB/SmolLM3-3B) was released during our experiments but is not compatible with the vLLM version we used for inference and dependency hell prohibited us from updating vLLM.
@@ -341,7 +350,7 @@ SmolLM2 is already over a year old at this point. If model quality matters, shou
 
 #### Does the model generation matter?
 
-We compare Qwen models from versions 1.5 [@qwen], 2 [@qwen2], 2.5 [@qwen25], and 3 on the [tutorial](#tutorial) prompt. The differences are small, but there is a consistent upward trend: newer versions lead to slightly higher evaluation performance (see <FigRef target="model-generation" />), especially cumulative from version 1.5 to 3.
+We compare Qwen models from versions 1.5 [@qwen], 2 [@qwen2], 2.5 [@qwen25], and 3 on the [tutorial](#tutorial) prompt:
 
 <HtmlEmbed
   id="model-generation"
@@ -358,6 +367,8 @@ We compare Qwen models from versions 1.5 [@qwen], 2 [@qwen2], 2.5 [@qwen25], and
   }}
 />
 
+The differences are small, but there is a consistent upward trend: newer versions lead to slightly higher evaluation performance, especially cumulative from version 1.5 to 3.
+
 <Note title="Summary: Impact of the Rephrasing Model" variant="info">
 **Model size**: 1B is sufficient. Larger models do not help.<br/>
 **Model family**: SmolLM2 dominates across all prompts.<br/>
@@ -373,7 +384,7 @@ So far we've always mixed synthetic data with a <Glossary term="source dataset"
 
 #### Is synthetic data enough?
 
-The dream scenario would be generating all your training data synthetically, no curation needed. We test this by comparing synthetic-only training vs mixed training (synthetic + source) across all our prompts on DCLM and FineWeb-Edu-HQ sources. Unfortunately, synthetic-only training falls short of both DCLM and mixed training (see <FigRef target="synthetic-only" />). Mixing consistently improves over both the synthetic-only and original-data-only baselines, regardless of prompt type.
+The dream scenario would be generating all your training data synthetically, no curation needed. We test this by comparing synthetic-only training vs mixed training (synthetic + source) across all our prompts on DCLM and FineWeb-Edu-HQ sources:
 
 <HtmlEmbed
   id="synthetic-only"
@@ -424,11 +435,13 @@ The dream scenario would be generating all your training data synthetically, no
   }}
 />
 
+Unfortunately, synthetic-only training falls short of both DCLM and mixed training. Mixing consistently improves over both the synthetic-only and original-data-only baselines, regardless of prompt type.
+
 OK, so we need to mix in original data. But how much does the specific choice of mix-in dataset affect performance?
 
 #### Does the mix-in dataset matter?
 
-We apply the [tutorial](#tutorial) prompt using Gemma-3-1B on FineWeb-Edu-HQ, then mix in one of four datasets: DCLM, Cosmopedia, FineWeb-Edu-HQ, or FineWeb-Edu-LQ. Use the Setup dropdown to also see results with LQ source data. DCLM outperforms other mix-in datasets across the board. Adding synthetic data improves performance for all mix-in datasets, with the effect especially pronounced for the weaker ones (see <FigRef target="mixin-dataset" />). This was one of our bigger surprises: the mix-in dataset is a major performance driver, sometimes more important than the synthetic data itself.
+We apply the [tutorial](#tutorial) prompt using Gemma-3-1B on FineWeb-Edu-HQ, then mix in one of four datasets: DCLM, Cosmopedia, FineWeb-Edu-HQ, or FineWeb-Edu-LQ. Use the Setup dropdown to also see results with LQ source data:
 
 <HtmlEmbed
   id="mixin-dataset"
@@ -464,11 +477,13 @@ We apply the [tutorial](#tutorial) prompt using Gemma-3-1B on FineWeb-Edu-HQ, th
   }}
 />
 
+DCLM outperforms other mix-in datasets across the board. Adding synthetic data improves performance for all mix-in datasets, with the effect especially pronounced for the weaker ones. This was one of our bigger surprises: the mix-in dataset is a major performance driver, sometimes more important than the synthetic data itself.
+
 If the mix-in dataset matters so much, what about the source dataset we're actually rephrasing?
 
 #### Does the source dataset matter?
 
-We rephrase four datasets (DCLM, Cosmopedia, FineWeb-Edu-HQ, FineWeb-Edu-LQ) with [faq](#faq) and [tutorial](#tutorial) prompts, testing two regimes: (a) mix-in equals source, and (b) fixed mix-in (FineWeb-Edu-HQ). When mix-in varies with source, source quality appears to matter: FineWeb-Edu-HQ and DCLM clearly outperform FineWeb-Edu-LQ and Cosmopedia (see <FigRef target="source-dataset-mixin-source" />). But when we fix the mix-in to FineWeb-Edu-HQ, the source effect nearly vanishes (see <FigRef target="source-dataset-fixed-mixin" />). This is exciting: it means you can rephrase even low-quality data and still get competitive results, as long as you pair it with a strong mix-in dataset.
+We rephrase four datasets (DCLM, Cosmopedia, FineWeb-Edu-HQ, FineWeb-Edu-LQ) with [faq](#faq) and [tutorial](#tutorial) prompts, testing two regimes: (a) mix-in equals source, and (b) fixed mix-in (FineWeb-Edu-HQ). First, here's what happens when mix-in varies with source:
 
 <HtmlEmbed
   id="source-dataset-mixin-source"
@@ -498,6 +513,8 @@ We rephrase four datasets (DCLM, Cosmopedia, FineWeb-Edu-HQ, FineWeb-Edu-LQ) wit
   }}
 />
 
+Source quality appears to matter here: FineWeb-Edu-HQ and DCLM clearly outperform FineWeb-Edu-LQ and Cosmopedia. But when we fix the mix-in to FineWeb-Edu-HQ, the source effect nearly vanishes:
+
 <HtmlEmbed
   id="source-dataset-fixed-mixin"
   src="d3-benchmark-comparison.html"
@@ -526,11 +543,11 @@ We rephrase four datasets (DCLM, Cosmopedia, FineWeb-Edu-HQ, FineWeb-Edu-LQ) wit
   }}
 />
 
-That opens up a much larger pool of source data to draw from. But can we squeeze out even more performance by increasing diversity in the synthetic portion?
+This is exciting: it means you can rephrase even low-quality data and still get competitive results, as long as you pair it with a strong mix-in dataset. That opens up a much larger pool of source data to draw from. But can we squeeze out even more performance by increasing diversity in the synthetic portion?
 
 #### Does increased diversity help?
 
-We test three diversity strategies: mixing prompts, mixing model families, and mixing both. Use the Setup dropdown to compare strategies. None of them show a significant improvement over the best individual configuration. Performance averages rather than compounds (see <FigRef target="diversity" />). This was a bit disappointing. That said, our ablations train on only 20B tokens, so diversity benefits may emerge at larger scales where the model can better exploit the varied signal.
+We test three diversity strategies: mixing prompts, mixing model families, and mixing both. Use the Setup dropdown to compare strategies:
 
 <Sidenote>
 Interestingly, when mixing enough different prompts together, we don't seem to need the source dataset for good performance. This could mean that diverse synthetic data can substitute for the original data, but a single synthetic dataset cannot.
@@ -578,6 +595,8 @@ Interestingly, when mixing enough different prompts together, we don't seem to n
   }}
 />
 
+None of them show a significant improvement over the best individual configuration. Performance averages rather than compounds. This was a bit disappointing. That said, our ablations train on only 20B tokens, so diversity benefits may emerge at larger scales where the model can better exploit the varied signal.
+
 <Note title="Summary: Impact of the Dataset Choices" variant="info">
 **Synthetic-only**: Not enough. Always mix with original data.<br/>
 **Mix-in dataset**: Major performance driver, sometimes more important than the synthetic data itself.<br/>
@@ -590,7 +609,7 @@ We've covered prompts, models, and datasets. One last fun question: how sensitiv
 
 ### Do Typos in the Prompt Hurt?
 
-While implementing the REWIRE prompt, we noticed it contained several typos and grammatical errors. So we cleaned it up and ran both versions. The result? Typos don't hurt at all. For the 1B model, the typo-laden [original](#guided_rewrite_original) actually performs slightly better than the [improved version](#guided_rewrite_improved) (see <FigRef target="typos-effect" />). So much for prompt polish.
+While implementing the REWIRE prompt, we noticed it contained several typos and grammatical errors. So we cleaned it up and ran both versions:
 
 <HtmlEmbed
   id="typos-effect"
@@ -607,6 +626,8 @@ While implementing the REWIRE prompt, we noticed it contained several typos and
   }}
 />
 
+Typos don't hurt at all. For the 1B model, the typo-laden [original](#guided_rewrite_original) actually performs slightly better than the [improved version](#guided_rewrite_improved). So much for prompt polish.
+
 With that final detail in hand, let's take stock of everything we've found.
 
 ### Takeaways

app/src/content/chapters/4-analyses.mdx

@@ -1,5 +1,4 @@
 import HtmlEmbed from "../../components/HtmlEmbed.astro";
-import FigRef from "../../components/FigRef.astro";
 import Note from "../../components/Note.astro";
 import Wide from "../../components/Wide.astro";
 
@@ -10,11 +9,7 @@ The experiments tell us *what* works. Now let's zoom out and ask *why*. We look
 
 ### Is More Compute Worth It?
 
-Running 90 experiments is not cheap. GPU time varies by two orders of magnitude: the cheapest run (Table with SmolLM2) took 8 days, while the most expensive (Guided Rewrite with Gemma-3 27B) consumed over 15 months of GPU time. <FigRef target="cost-efficiency" /> plots each experiment's downstream performance against its GPU cost on a log scale, with a Pareto frontier connecting the most efficient configurations.
-
-**The Pareto frontier is dominated by small models with simple prompts.** The best cost-performance tradeoffs come from 1B-class models (Gemma-3-1B, SmolLM2-1.7B) paired with format prompts like Math, Table, and FAQ. Scaling up to 12B or 27B models pushes GPU time by 5-10x while at the same time *decreasing* performance.
-
-**The message is clear: invest in prompt design, not model size.** A well-chosen prompt on a 1B model will outperform a generic prompt on a 27B model at a tiny fraction of the cost. The only scenario where larger models might be justified is for complex prompts (like Guided Rewrite) that require more capable instruction following, but even there the gains are marginal.
+Running 90 experiments is not cheap. GPU time varies by two orders of magnitude: the cheapest run (Table with SmolLM2) took 8 days, while the most expensive (Guided Rewrite with Gemma-3 27B) consumed over 15 months of GPU time. Here's each experiment's downstream performance plotted against its GPU cost on a log scale, with a Pareto frontier connecting the most efficient configurations:
 
 <Wide>
 <HtmlEmbed
@@ -25,24 +20,18 @@ Running 90 experiments is not cheap. GPU time varies by two orders of magnitude:
 />
 </Wide>
 
+**The Pareto frontier is dominated by small models with simple prompts.** The best cost-performance tradeoffs come from 1B-class models (Gemma-3-1B, SmolLM2-1.7B) paired with format prompts like Math, Table, and FAQ. Scaling up to 12B or 27B models pushes GPU time by 5-10x while at the same time *decreasing* performance.
+
+**The message is clear: invest in prompt design, not model size.** A well-chosen prompt on a 1B model will outperform a generic prompt on a 27B model at a tiny fraction of the cost. The only scenario where larger models might be justified is for complex prompts (like Guided Rewrite) that require more capable instruction following, but even there the gains are marginal.
+
 Even the cheapest configurations still take over a week of GPU time, and we only know which ones work *after* rephrasing 10B tokens and then training a model. Wouldn't it be nice if we could just score the rephrased outputs directly and skip the expensive train-then-evaluate loop?
 
 ### Can Quality Scores Predict Performance?
 
-FineWeb-Edu-score and DCLM-score are great quality filters for human-written web data. If they also work for synthetic data, we could score rephrased outputs directly and iterate on prompts without running the full pipeline each time. We computed Spearman rank correlations between various edu-score and DCLM-score metrics (input scores, output scores, score differences, and relative improvements) and all downstream benchmark results across our 90 experiments.[^broken-scores] <FigRef target="score-correlation" /> shows the full correlation matrix.
+FineWeb-Edu-score and DCLM-score are great quality filters for human-written web data. If they also work for synthetic data, we could score rephrased outputs directly and iterate on prompts without running the full pipeline each time. We computed Spearman rank correlations between various edu-score and DCLM-score metrics (input scores, output scores, score differences, and relative improvements) and all downstream benchmark results across our 90 experiments.[^broken-scores] Here's the full correlation matrix:
 
 [^broken-scores]: Seven early runs had incorrect input quality scores due to a scoring pipeline bug and are excluded from the quality score analyses: `article-1b-hq`, `commentary-1b-hq`, `discussion-1b-hq`, `tutorial-1b-hq`, `tutorial-12b-hq`, `faq-1b-lq`, and `faq-12b-lq`. Their downstream benchmark results are unaffected and included in all other analyses.
 
-**DCLM-score is a moderate predictor of aggregate performance.** The DCLM-score difference (output minus input) shows the strongest correlation with `agg_score_macro` (ρ = 0.61, p {'<'} 0.001), followed by the output DCLM-score (ρ = 0.56). These are moderate correlations at best. The DCLM-score variants are particularly predictive for table understanding (ρ = 0.47–0.54) and reading comprehension (ρ = 0.49–0.52).
-
-**Edu-score tells a more nuanced story.** The input edu-score (the score of the original data before rephrasing) correlates with aggregate performance (ρ = 0.27, p {'<'} 0.05), but the output edu-score (the score of the rephrased data) shows essentially no correlation (ρ = −0.08, not significant). Starting with higher-quality source data matters, but the edu-score of the synthetic output is not a reliable proxy at all.
-
-{/*
-**The HellaSwag/PIQA anomaly deserves a closer look.** Edu-score improvement shows strong *positive* correlations with HellaSwag (ρ = 0.60) and PIQA (ρ = 0.58), while being *negatively* correlated with math (ρ = −0.39) and reading comprehension (ρ = −0.30). We investigated whether this was a confound from prompt type (FAQ and tutorial prompts both increase edu-scores and might independently help NLU). The correlation survives partial correlation controlling for prompt type (ρ = 0.65 for HellaSwag, ρ = 0.56 for PIQA, both p {'<'} 0.001) and for model size within the Gemma family (ρ = 0.60 and 0.68). So the effect is real. However, the practical magnitude is tiny: HellaSwag scores range from 0.066 to 0.092 across all 90 experiments (CV = 5.8%), compared to `agg_score_macro` ranging from 0.096 to 0.172 (CV = 10.5%). The edu-score captures something about sentence-completion and physical-intuition quality, but the absolute differences are so small that optimizing for it would be chasing noise.
-*/}
-
-**Neither score is a reliable universal proxy.** WinoGrande shows essentially zero correlation with any predictor. The strongest individual correlations (ρ ≈ 0.56–0.61) are still only moderate, explaining roughly 30% of the variance at best. **The bottom line: for synthetic data, there is no shortcut. You have to train models and evaluate them.**
-
 {/* 
 Seven early runs have incorrect input quality scores due to a scoring pipeline bug and
 are excluded in both charts' JS via BROKEN_INPUT_SCORES rather than patched in the JSON:
@@ -58,11 +47,17 @@ article/commentary/discussion/tutorial-1b-hq, tutorial-12b-hq, faq-1b-lq, faq-12
 />
 </Wide>
 
-The correlation matrix tells us that quality scores are weak predictors, but not *how* scores change through rephrasing. <FigRef target="score-shift" /> visualizes this as a slope chart: each experiment is a line connecting its input score (left), output score (middle), and downstream `agg_score_macro` (right). Toggle between DCLM and edu-score views to see both perspectives.
+**DCLM-score is a moderate predictor of aggregate performance.** The DCLM-score difference (output minus input) shows the strongest correlation with `agg_score_macro` (ρ = 0.61, p {'<'} 0.001), followed by the output DCLM-score (ρ = 0.56). These are moderate correlations at best. The DCLM-score variants are particularly predictive for table understanding (ρ = 0.47–0.54) and reading comprehension (ρ = 0.49–0.52).
 
-**DCLM scores almost universally increase through rephrasing.** Nearly every experiment shows an upward slope from input to output DCLM score, regardless of prompt type or model. The rephrasing models produce cleaner, more structured text that the DCLM classifier rewards. But the slope from output DCLM score to downstream performance is much flatter and noisier, confirming that a high DCLM score does not guarantee good training data.
+**Edu-score tells a more nuanced story.** The input edu-score (the score of the original data before rephrasing) correlates with aggregate performance (ρ = 0.27, p {'<'} 0.05), but the output edu-score (the score of the rephrased data) shows essentially no correlation (ρ = −0.08, not significant). Starting with higher-quality source data matters, but the edu-score of the synthetic output is not a reliable proxy at all.
 
-**Edu-scores tell the opposite story.** Most experiments *decrease* the edu-score through rephrasing, particularly those starting from high-quality sources (FineWeb-Edu-HQ has high baseline edu-scores). The edu-score classifier penalizes format changes like tables, FAQs, and math notation that our best prompts produce. This is a case where the proxy metric actively misleads: the "quality degradation" measured by edu-score corresponds to format transformations that *improve* downstream performance.
+{/*
+**The HellaSwag/PIQA anomaly deserves a closer look.** Edu-score improvement shows strong *positive* correlations with HellaSwag (ρ = 0.60) and PIQA (ρ = 0.58), while being *negatively* correlated with math (ρ = −0.39) and reading comprehension (ρ = −0.30). We investigated whether this was a confound from prompt type (FAQ and tutorial prompts both increase edu-scores and might independently help NLU). The correlation survives partial correlation controlling for prompt type (ρ = 0.65 for HellaSwag, ρ = 0.56 for PIQA, both p {'<'} 0.001) and for model size within the Gemma family (ρ = 0.60 and 0.68). So the effect is real. However, the practical magnitude is tiny: HellaSwag scores range from 0.066 to 0.092 across all 90 experiments (CV = 5.8%), compared to `agg_score_macro` ranging from 0.096 to 0.172 (CV = 10.5%). The edu-score captures something about sentence-completion and physical-intuition quality, but the absolute differences are so small that optimizing for it would be chasing noise.
+*/}
+
+**Neither score is a reliable universal proxy.** WinoGrande shows essentially zero correlation with any predictor. The strongest individual correlations (ρ ≈ 0.56–0.61) are still only moderate, explaining roughly 30% of the variance at best. **The bottom line: for synthetic data, there is no shortcut. You have to train models and evaluate them.**
+
+The correlation matrix tells us that quality scores are weak predictors, but not *how* scores change through rephrasing. The slope chart below visualizes this: each experiment is a line connecting its input score (left), output score (middle), and downstream `agg_score_macro` (right). Toggle between DCLM and edu-score views to see both perspectives:
 
 <Wide>
 <HtmlEmbed
@@ -73,11 +68,15 @@ The correlation matrix tells us that quality scores are weak predictors, but not
 />
 </Wide>
 
+**DCLM scores almost universally increase through rephrasing.** Nearly every experiment shows an upward slope from input to output DCLM score, regardless of prompt type or model. The rephrasing models produce cleaner, more structured text that the DCLM classifier rewards. But the slope from output DCLM score to downstream performance is much flatter and noisier, confirming that a high DCLM score does not guarantee good training data.
+
+**Edu-scores tell the opposite story.** Most experiments *decrease* the edu-score through rephrasing, particularly those starting from high-quality sources (FineWeb-Edu-HQ has high baseline edu-scores). The edu-score classifier penalizes format changes like tables, FAQs, and math notation that our best prompts produce. This is a case where the proxy metric actively misleads: the "quality degradation" measured by edu-score corresponds to format transformations that *improve* downstream performance.
+
 So quality scores designed for filtering web data don't transfer to synthetic data. Maybe looking at the outputs more directly helps. For instance, does the length of the rephrased output tell us anything?
 
 ### Do Chatty Models Make Better Data?
 
-Different prompt formats produce wildly different output lengths. <FigRef target="verbosity" /> shows the output tokens per document across four prompt types, broken down by model family. Table and Math prompts tend to be concise, while FAQ and Tutorial prompts generate significantly more tokens per document. The spread within each prompt type varies across model families: some models are consistently verbose regardless of the prompt, while others adapt their output length to the task.
+Different prompt formats produce wildly different output lengths. Here are the output tokens per document across four prompt types, broken down by model family:
 
 <Wide>
 <HtmlEmbed
@@ -88,9 +87,9 @@ Different prompt formats produce wildly different output lengths. <FigRef target
 />
 </Wide>
 
-But does this variation actually affect downstream performance? Our prompts produce outputs ranging from 25% of the input length (Commentary) to 150% (Guided Rewrite at 12B). <FigRef target="compression-performance" /> plots each experiment's compression ratio against its benchmark score.
+Table and Math prompts tend to be concise, while FAQ and Tutorial prompts generate significantly more tokens per document. The spread within each prompt type varies across model families: some models are consistently verbose regardless of the prompt, while others adapt their output length to the task.
 
-**There is no meaningful relationship between compression ratio and performance.** Highly compressive prompts (Commentary at 0.26x, Table at 0.25x) and expansive ones (Guided Rewrite at 1.5x) both appear across the full range of performance scores. The best-performing experiments cluster around 0.3x–0.8x compression, but this likely reflects the distribution of prompt types rather than any causal effect of compression itself. FAQ and Tutorial prompts, which happen to compress moderately, also happen to be the strongest prompts for other reasons (pedagogical restructuring, diverse output formats). What matters is the content and structure of the output, not its length relative to the input.
+But does this variation actually affect downstream performance? Our prompts produce outputs ranging from 25% of the input length (Commentary) to 150% (Guided Rewrite at 12B). Here's each experiment's compression ratio plotted against its benchmark score:
 
 <Wide>
 <HtmlEmbed
@@ -101,6 +100,8 @@ But does this variation actually affect downstream performance? Our prompts prod
 />
 </Wide>
 
+**There is no meaningful relationship between compression ratio and performance.** Highly compressive prompts (Commentary at 0.26x, Table at 0.25x) and expansive ones (Guided Rewrite at 1.5x) both appear across the full range of performance scores. The best-performing experiments cluster around 0.3x–0.8x compression, but this likely reflects the distribution of prompt types rather than any causal effect of compression itself. FAQ and Tutorial prompts, which happen to compress moderately, also happen to be the strongest prompts for other reasons (pedagogical restructuring, diverse output formats). What matters is the content and structure of the output, not its length relative to the input.
+
 So output length doesn't predict quality either. But we stumbled onto something more interesting while looking at output *diversity*: a case where a model that follows instructions poorly actually produces better training data.
 
 ### Math Rephrasing: When "Worse" Outputs Win

app/src/content/chapters/5-infrastructure.mdx

@@ -1,6 +1,5 @@
 import HtmlEmbed from "../../components/HtmlEmbed.astro";
 import Sidenote from "../../components/Sidenote.astro";
-import FigRef from "../../components/FigRef.astro";
 import Accordion from "../../components/Accordion.astro";
 import Wide from "../../components/Wide.astro";
 
@@ -13,7 +12,7 @@ Thanks to fast inference engines like [vLLM](https://github.com/vllm-project/vll
 
 We made major extensions to [DataTrove](https://github.com/huggingface/datatrove) [@datatrove] to handle this. DataTrove supports both local generation and large-scale distributed runs on Slurm clusters, handling chunking, checkpointing, distributed queueing, and Hugging Face dataset management so you can focus on synthetic data design rather than operational glue. We used it for every experiment in this blog post, from 10k-example test runs to the full FinePhrase production pipeline.
 
-<FigRef target="datatrove-pipeline" /> gives an overview of the pipeline. Let's walk through it.
+Here's an overview of the pipeline:
 
 <HtmlEmbed
   id="datatrove-pipeline"
@@ -21,6 +20,8 @@ We made major extensions to [DataTrove](https://github.com/huggingface/datatrove
   caption="Overview of the DataTrove synthetic data generation pipeline. Documents flow through a three-stage pipeline (Read, Transform, Write), with the InferenceRunner dispatching rollout functions to vLLM/SGLang. The system supports local and Slurm-based execution with automatic upload and progress monitoring."
 />
 
+Let's walk through it.
+
 ### Generating synthetic data at scale
 
 At the core is `examples/inference/benchmark/generate_data.py`, a [Typer](https://typer.tiangolo.com/)-powered entry point that orchestrates the full synthetic data loop:
@@ -281,7 +282,7 @@ So what did all 801 configurations tell us?
 
 #### Results
 
-<FigRef target="optimization-sweep" /> shows the progression from baseline (vLLM defaults) through tier 0 and tier 1 optimization for all 18 models. Hover over any point to see the exact configuration and throughput.
+Here's the progression from baseline (vLLM defaults) through tier 0 and tier 1 optimization for all 18 models. Hover over any point to see the exact configuration and throughput:
 
 <HtmlEmbed
   id="optimization-sweep"
@@ -435,9 +436,7 @@ Further improvement ideas:
 - add a second model below so we can compare.
 */}
 
-To get a feel for what these throughput numbers actually mean, <FigRef target="inference-throughput" /> lets you pick a model and scale up the number of GPUs. Each page represents roughly 500 tokens of generated text. At high enough throughput, pages roll up into books (500 pages each), and books into shelves (500 books each).
-
-With all these infrastructure pieces in place, we have everything we need to build FinePhrase: the right prompts, the right model, and the machinery to run it all at scale.
+To get a feel for what these throughput numbers actually mean, pick a model below and scale up the number of GPUs. Each page represents roughly 500 tokens of generated text. At high enough throughput, pages roll up into books (500 pages each), and books into shelves (500 books each).
 
 <Wide>
 <HtmlEmbed
@@ -446,3 +445,5 @@ With all these infrastructure pieces in place, we have everything we need to bui
   caption="Interactive GPU throughput simulator. Select a model and adjust the number of H100 GPUs to see how fast text is generated. Scale mapping: 📄 1 page = 500 toks, 📖 1 book = 500 pages, 📚 1 shelf = 500 books."
 />
 </Wide>
+
+With all these infrastructure pieces in place, we have everything we need to build FinePhrase: the right prompts, the right model, and the machinery to run it all at scale.

app/src/content/chapters/6-finephrase.mdx

@@ -1,7 +1,6 @@
 import Image from "../../components/Image.astro";
 import HtmlEmbed from "../../components/HtmlEmbed.astro";
 import Sidenote from "../../components/Sidenote.astro";
-import FigRef from "../../components/FigRef.astro";
 import Wide from "../../components/Wide.astro";
 
 import datasetCardImg from "../assets/image/auto-dataset-card.png";
@@ -66,14 +65,14 @@ On Slurm, a single `generate_data` call orchestrates three coordinated jobs: the
 
 ### Automatic HF Upload and Progress Monitoring
 
-We want you to be able to just press a button, let the GPUs go brrrr, and check back in to the finished dataset. DataTrove continuously uploads data to your specified Hugging Face dataset repo whenever a chunk is finished, using `ParquetWriter` with `hf://` paths so data appears on the Hub within minutes of generation, not after the full run completes. At the end, the `InferenceDatasetCardGenerator` pipeline step checks the logs directory, collects information about the throughput, and uploads a dataset card to document your new synthetic dataset. <FigRef target="auto-dataset-card" /> shows an example of the auto-generated dataset card.
+We want you to be able to just press a button, let the GPUs go brrrr, and check back in to the finished dataset. DataTrove continuously uploads data to your specified Hugging Face dataset repo whenever a chunk is finished, using `ParquetWriter` with `hf://` paths so data appears on the Hub within minutes of generation, not after the full run completes. At the end, the `InferenceDatasetCardGenerator` pipeline step checks the logs directory, collects information about the throughput, and uploads a dataset card to document your new synthetic dataset. Here's an example of the auto-generated dataset card:
 
 <figure id="auto-dataset-card">
 <Image src={datasetCardImg} alt="Auto-generated dataset card on the Hugging Face Hub" />
 <figcaption>Example of an auto-generated dataset card with throughput metrics, uploaded to the Hugging Face Hub after inference completes.</figcaption>
 </figure>
 
-For long-running inference jobs like FinePhrase (which runs for about two weeks), the `InferenceProgressMonitor` runs as a separate Slurm job alongside the inference workers. It periodically scans the output directory, counts completed chunks across all 100 tasks, and updates the dataset card on the Hub with a progress bar and ETA for each prompt template. <FigRef target="finephrase-progress" /> shows the live progress dashboard during the FinePhrase generation run.
+For long-running inference jobs like FinePhrase (which runs for about two weeks), the `InferenceProgressMonitor` runs as a separate Slurm job alongside the inference workers. It periodically scans the output directory, counts completed chunks across all 100 tasks, and updates the dataset card on the Hub with a progress bar and ETA for each prompt template. Here's the live progress dashboard during the FinePhrase generation run:
 
 <figure id="finephrase-progress">
 <Image src={finephraseProgressImg} alt="Live progress monitoring of the FinePhrase generation run" />
@@ -143,7 +142,7 @@ mkdir -p "$HF_XET_CACHE"
 
 #### Multi-config dataset support
 
-FinePhrase runs four prompt templates that produce four independent dataset configs (faq, math, table, tutorial). Without config-awareness, all four templates would fight over a single dataset card and progress counters would exceed 100%. [PR #447](https://github.com/huggingface/datatrove/pull/447) adds first-class config support: outputs go to config-specific folders (`hf://datasets/HuggingFaceFW/finephrase/faq/`, `.../math/`, etc.), the dataset card merges information from all configs, and the progress monitor tracks each config independently so you see four separate progress bars (as in <FigRef target="finephrase-progress" />).
+FinePhrase runs four prompt templates that produce four independent dataset configs (faq, math, table, tutorial). Without config-awareness, all four templates would fight over a single dataset card and progress counters would exceed 100%. [PR #447](https://github.com/huggingface/datatrove/pull/447) adds first-class config support: outputs go to config-specific folders (`hf://datasets/HuggingFaceFW/finephrase/faq/`, `.../math/`, etc.), the dataset card merges information from all configs, and the progress monitor tracks each config independently so you see four separate progress bars (as in the [progress dashboard above](#finephrase-progress)).
 
 #### Configurable server startup
 
@@ -164,7 +163,7 @@ With all these fixes in place, the pipeline ran to completion. So what does the
 
 ### What's in the Dataset?
 
-<FigRef target="finephrase-explorer" /> lets you browse real examples from FinePhrase. Each sample shows the original FineWeb-Edu source document alongside all four rephrased versions. Navigate through samples to see how the same web document becomes a FAQ, a math problem, a structured table, and a step-by-step tutorial.
+Browse some real examples from FinePhrase below. Each sample shows the original FineWeb-Edu source document alongside all four rephrased versions. Navigate through samples to see how the same web document becomes a FAQ, a math problem, a structured table, and a step-by-step tutorial:
 
 <Wide>
 <HtmlEmbed