moved conclusions to experiments, polished transitions and filled in fancy numbers

app/src/content/chapters/conclusions.mdx

@@ -1,33 +1,5 @@
 ## Conclusions
 
-Here are the key takeaways from our experiments:
-
-- **Q: How do existing datasets compare?**<br/>
-  A: DCLM, Nemotron-HQ-Synth, and REWIRE lead. Most synthetic baselines fall behind.
-- **Q: Which individual prompts from the synthetic baselines match DCLM?**<br/>
-  A: Only Diverse QA Pairs and REWIRE's Guided Rewrite.
-- **Q: Can new prompts beat DCLM?**<br/>
-  A: Yes. Math, Table, FAQ, and Tutorial all outperform DCLM.
-- **Q: Does model size matter?**<br/>
-  A: Not much. 1B is sufficient for simple prompts, 4B for complex ones.
-- **Q: Do we need better models for low-quality data?**<br/>
-  A: No consistent advantage from larger models on low-quality sources.
-- **Q: Does the model family matter?**<br/>
-  A: Yes. SmolLM2 dominates across all prompts.
-- **Q: Does the model generation matter?**<br/>
-  A: Slightly. Newer Qwen versions trend better.
-- **Q: Is synthetic data enough?**<br/>
-  A: No. Always mix synthetic with original data.
-- **Q: Does the mix-in dataset matter?**<br/>
-  A: Yes, a major performance driver, sometimes more important than the synthetic data.
-- **Q: Does the source dataset matter?**<br/>
-  A: Not with a strong mix-in. Even low-quality sources produce competitive results.
-- **Q: Does increased diversity help?**<br/>
-  A: No, performance averages rather than compounds.
-- **Q: Do typos in the prompt hurt?**<br/>
-  A: No. Typos have no negative effect on downstream performance.
-
-The bottom line: the details of synthetic rephrasing matter a lot, and knowing which ones matter is the key to scaling it up. Prompt design is the single biggest lever, with structured formats like Math, Table, FAQ, and Tutorial consistently beating curated baselines. But equally important is knowing where you can cut corners without losing quality. You don't need a large rephrasing model (1B is enough for simple prompts, 4B for complex ones). You don't need pristine source data (even low-quality sources work with a strong mix-in). Smaller models generate faster, directly translating into higher throughput. And tolerating lower-quality sources opens up a much bigger and more diverse data pool to draw from. The practical recipe is straightforward: pick a strong structured prompt, use the smallest model that handles it, blend with high-quality original data, and spend your remaining compute on volume.
 
 ### Next Steps
 

app/src/content/chapters/experiments.mdx

@@ -9,11 +9,6 @@ import FigRef from "../../components/FigRef.astro";
 {/* TODO: potentially make a widget for data exploration: look at the same few samples generated by different models or transformed with different prompts */}
 {/* TODO: Standardize colors across charts in the blog post more */}
 {/* TODO: Check all the charts again in dark mode */}
-{/* TODO: In analyses make transitions better */}
-{/* TODO: Combine verbosity and compression analysis under one title */}
-{/* TODO: Combine quality score and edu/dclm score analysis under one title */}
-{/* TODO: Move conclusions to after the experiments into conclusions subsection */}
-{/* TODO: put the new analyses into context and update the intro paragraphs of the experiments and analyses accordingly */}
 
 ## Experiments
 
@@ -552,7 +547,7 @@ Interestingly, when mixing enough different prompts together, we don't seem to n
 **Practical takeaway**: Invest in a high-quality mix-in dataset. The source quality matters less.
 </Note>
 
-Let's turn to some unexpected findings from our experiments.
+We've covered prompts, models, and datasets. One last question: how sensitive is all of this to small details in the prompt itself?
 
 ### Do Typos in the Prompt Hurt?
 
@@ -574,3 +569,34 @@ We compare REWIRE's [original prompt](#guided_rewrite_original) (with typos) aga
   }}
 />
 
+### Takeaways
+
+Here are the key takeaways from our experiments:
+
+- **Q: How do existing datasets compare?**<br/>
+  A: DCLM, Nemotron-HQ-Synth, and REWIRE lead. Most synthetic baselines fall behind.
+- **Q: Which individual prompts from the synthetic baselines match DCLM?**<br/>
+  A: Only Diverse QA Pairs and REWIRE's Guided Rewrite.
+- **Q: Can new prompts beat DCLM?**<br/>
+  A: Yes. Math, Table, FAQ, and Tutorial all outperform DCLM.
+- **Q: Does model size matter?**<br/>
+  A: Not much. 1B is sufficient for simple prompts, 4B for complex ones.
+- **Q: Do we need better models for low-quality data?**<br/>
+  A: No consistent advantage from larger models on low-quality sources.
+- **Q: Does the model family matter?**<br/>
+  A: Yes. SmolLM2 dominates across all prompts.
+- **Q: Does the model generation matter?**<br/>
+  A: Slightly. Newer Qwen versions trend better.
+- **Q: Is synthetic data enough?**<br/>
+  A: No. Always mix synthetic with original data.
+- **Q: Does the mix-in dataset matter?**<br/>
+  A: Yes, a major performance driver, sometimes more important than the synthetic data.
+- **Q: Does the source dataset matter?**<br/>
+  A: Not with a strong mix-in. Even low-quality sources produce competitive results.
+- **Q: Does increased diversity help?**<br/>
+  A: No, performance averages rather than compounds.
+- **Q: Do typos in the prompt hurt?**<br/>
+  A: No. Typos have no negative effect on downstream performance.
+
+The bottom line: the details of synthetic rephrasing matter a lot, and knowing which ones matter is the key to scaling it up. Prompt design is the single biggest lever, with structured formats like Math, Table, FAQ, and Tutorial consistently beating curated baselines. But equally important is knowing where you can cut corners without losing quality. You don't need a large rephrasing model (1B is enough for simple prompts, 4B for complex ones). You don't need pristine source data (even low-quality sources work with a strong mix-in). Smaller models generate faster, directly translating into higher throughput. And tolerating lower-quality sources opens up a much bigger and more diverse data pool to draw from. The practical recipe is straightforward: pick a strong structured prompt, use the smallest model that handles it, blend with high-quality original data, and spend your remaining compute on volume.
+

app/src/content/chapters/introduction.mdx

@@ -31,7 +31,7 @@ Synthetic data also plays a central role in post-training via *distillation*, wh
 
 However, how to do synthetic data generation properly still resembles alchemy these days: Which model should you use? Which prompts work best and how many do you need? And how do you even scale this effectively?
 
-In this blog post we take a journey to answer all these questions systematically. We ran XXX experiments and generated YYY tokens in total to find the ideal settings for synthetic data.
+In this blog post we take a journey to answer all these questions systematically. We ran 65 experiments, generated over 750 billion tokens and spent {'>'}74,000 GPU hours (~8.5 GPU years) for rephrasing alone to find the ideal settings for synthetic data.
 
 Here's the plan:
 
@@ -39,7 +39,7 @@ We start with the [Infrastructure](#infrastructure) needed for synthetic data ge
 
 We continue with the [Setup](#setup), a walkthrough of the different approaches for synthetic data in pretraining, from explaining what prior work did to the prompts we are experimenting with.
 
-Finally we present the suite of XXX [Experiments](#experiments) we ran to figure out best practices regarding what models, prompts and settings work well.
+Finally we present the suite of 65 [Experiments](#experiments) we ran to figure out best practices regarding what models, prompts and settings work well.
 
 Here's a preview of where we end up: FinePhrase, our best configuration, clearly outperforms all existing synthetic data baselines (<FigRef target="finephrase-vs-baselines" />). The rest of this post explains what's needed to get there.