Spaces:
Running on CPU Upgrade
Running on CPU Upgrade
Commit ·
1ac525d
1
Parent(s): 7171fcc
add intro plot again and compare to synth baselines
Browse files
app/src/content/chapters/6-finephrase.mdx
CHANGED
|
@@ -173,4 +173,46 @@ Browse some real examples from FinePhrase below. Each sample shows the original
|
|
| 173 |
/>
|
| 174 |
</Wide>
|
| 175 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
That's the full picture: 90 experiments, a battle-tested infrastructure, and 486 billion tokens of public synthetic data. Let's wrap up with what we learned and where to go next.
|
|
|
|
| 173 |
/>
|
| 174 |
</Wide>
|
| 175 |
|
| 176 |
+
### How Does FinePhrase Compare?
|
| 177 |
+
|
| 178 |
+
In the introduction we showed a single FinePhrase prompt (table) against the baselines. Now that the full dataset is built, here's how all four FinePhrase prompts stack up against the strongest synthetic data baselines:
|
| 179 |
+
|
| 180 |
+
<HtmlEmbed
|
| 181 |
+
id="finephrase-all-prompts"
|
| 182 |
+
src="d3-benchmark-comparison.html"
|
| 183 |
+
desc="All four FinePhrase prompts compared against synthetic data baselines across evaluation metrics."
|
| 184 |
+
config={{
|
| 185 |
+
defaultView: "line",
|
| 186 |
+
datasets: {
|
| 187 |
+
"mix-fw_edu_hq-table_smollm2_1.7b_hq": { display: "FinePhrase (table)", color: "#EBA937" },
|
| 188 |
+
"mix-fw_edu_hq-math_smollm2_1.7b_hq": { display: "FinePhrase (math)", color: "#D4892A" },
|
| 189 |
+
"mix-fw_edu_hq-faq_smollm2_1.7b_hq": { display: "FinePhrase (faq)", color: "#BD6E1E" },
|
| 190 |
+
"mix-fw_edu_hq-tutorial_smollm2_1.7b_hq": { display: "FinePhrase (tutorial)", color: "#A65514" },
|
| 191 |
+
cosmopedia: "Cosmopedia",
|
| 192 |
+
nemotron_hq_synth: "Nemotron-HQ-Synth",
|
| 193 |
+
rewire: "REWIRE",
|
| 194 |
+
synth_query_reasoning_answer: "SYNTH"
|
| 195 |
+
}
|
| 196 |
+
}}
|
| 197 |
+
/>
|
| 198 |
+
|
| 199 |
+
All four FinePhrase prompts outperform every synthetic baseline by a clear margin. Table and math lead the pack, with FAQ and tutorial close behind. The per-benchmark breakdown (switch with the dropdown above) tells a familiar story. FinePhrase prompts dominate on ARC, SQuAD, and DROP (knowledge and reading comprehension), while the baselines hold a slight edge on HellaSwag and PIQA (commonsense). This is the same commonsense-vs-knowledge trade-off we observed throughout the experiments, and it's exactly why FinePhrase is designed to be mixed with original data rather than used alone. The aggregate wins because the knowledge gains far outweigh the commonsense losses.
|
| 200 |
+
|
| 201 |
+
What makes this result especially compelling is the cost efficiency. Here is how FinePhrase compares to other synthetic data projects:
|
| 202 |
+
|
| 203 |
+
<figure id="cost-efficiency">
|
| 204 |
+
|
| 205 |
+
| Dataset | Generator | Tokens | GPU Hours | Tokens/GPU-Hour |
|
| 206 |
+
|:-----------------|:-------------------|---------:|-----------:|----------------:|
|
| 207 |
+
| Cosmopedia | Mixtral 8x7B | 25B | {'>'} 10K | {'<'} 2.5M |
|
| 208 |
+
| SYNTH | custom fine-tuned | ~200B | ~20K\* | ~10M\* |
|
| 209 |
+
| REWIRE | Llama-3.3 70B | 400B | **~352K** | ~1.1M |
|
| 210 |
+
| Nemotron-CC | Mistral NeMo 12B | **1.9T** | n/a | n/a |
|
| 211 |
+
| **FinePhrase** | SmolLM2-1.7B | 486B | ~14.7K | **~33.1M**|
|
| 212 |
+
|
| 213 |
+
<figcaption>Compute cost comparison across synthetic data generation projects. All GPU hours are H100. REWIRE hours extrapolated from their reported 88K per 100B tokens. \*SYNTH's 20K hours include both generation and model training, making their per-token rate an upper bound. Nemotron-CC did not report generation cost.</figcaption>
|
| 214 |
+
</figure>
|
| 215 |
+
|
| 216 |
+
FinePhrase achieves **~33M tokens per GPU hour**, roughly 30x more efficient than REWIRE and over 13x more than Cosmopedia. It generates more tokens than REWIRE while using 24x less compute, thanks to the combined payoff of a 1.7B model (vs 70B), optimized inference settings, and speculative decoding. The takeaway: you do not need large models for high-quality synthetic data generation.
|
| 217 |
+
|
| 218 |
That's the full picture: 90 experiments, a battle-tested infrastructure, and 486 billion tokens of public synthetic data. Let's wrap up with what we learned and where to go next.
|