Upload folder using huggingface_hub

014b14c verified 4 days ago

38.3 kB

	# Brainlift: Training and Optimizing Small Language Models to Reliably Perform Specific Behaviors

	## TL;DR
	- For instilling one narrow, falsifiable behavior into a 0.6B–4B model, the dataset is the deliverable: LIMA (1,000 examples), s1 (1,000 traces), and QLoRA's own ablations converge on quality-and-diversity over quantity, and QLoRA makes single-GPU tuning routine. Capability is mostly elicited from a base model that already latently has it — not installed from scratch.
	- QLoRA is the right default precisely because it "forgets less" (Biderman et al.), and reliability comes from an engineering stack — distilling a teacher's process, constrained decoding for output form, and an adversarial base-vs-tuned eval — not from a bigger model or a cleverer prompt.
	- Build the evaluation before you generate a single training row, and make it adversarial, because small (and large) LMs fail in predictable, testable ways: compositional reasoning collapse (Faith and Fate), irrelevant-clause sensitivity (GSM-Symbolic, up to 65% drop), the reversal curse, lost-in-the-middle, and confident hallucination.

	## Purpose
	A portable research knowledge base for instilling ONE specific, falsifiable behavior into a small open base model (0.6B–4B) via supervised fine-tuning (QLoRA), using distillation from a frontier teacher to build the dataset, and evaluating with LLM-as-judge plus a base-vs-tuned comparison.

	In scope: data-centric generation/curation and distillation; PEFT (LoRA/QLoRA); SFT mechanics; preference optimization (DPO and variants); compression/quantization; small-model landscape; evaluation and LLM-as-judge; reliability and structured output; failure modes and limits of (small) LMs; practitioner playbooks.

	Out of scope: pretraining from scratch; large-scale RLHF infrastructure; multimodal training; serving/infra beyond quantization formats; agent orchestration.

	## Key Findings
	1. Small-data alignment works when the capability is latent. Two independent 1,000-example results (LIMA for general alignment, s1 for reasoning) plus QLoRA's finding that data quality beats size establish that a narrow behavior can be reached cheaply — but LoRA-Learns-Less shows this breaks down for genuinely new hard skills.
	2. Distill the process, not the style. Orca and DeepSeek-R1 show reasoning traces transfer real capability into small models; Orca explicitly warns that style-only imitation inflates apparent quality on shallow evals.
	3. PEFT trades peak capacity for forgetting-resistance, which is a feature for narrow-behavior work. Full fine-tuning still wins on hard new skills.
	4. LLM-as-judge is usable (>80% human agreement) but biased (position, verbosity, self-preference); design evals around the biases.
	5. Reliability = data + constrained decoding + calibration eval. Neither fine-tuning nor grammar-constrained decoding is individually sufficient.
	6. Small models fail in named, reproducible ways — build adversarial evals that target exactly those failure modes.

	## DOK 4 — Spiky Points of View

	1. For a narrow behavior, ~1,000–2,000 excellent, diverse examples is enough; past that, more data is usually wasted effort.

	- _Why:_ pretraining already installed the capability, so SFT only has to pin a consistent output policy over what the model already knows (Superficial Alignment Hypothesis). Rows past the first ~1k mostly re-teach that and add overfitting risk.
	- What still helps is coverage of the behavior's input variety, not raw count.
	- Two independent replications on very different tasks (LIMA on alignment, s1 on competition math) landed on the same ~1,000 figure, so it isn't a single-domain fluke.
	- _Evidence:_ LIMA (1,000 examples rival RLHF models trained on 52× more data), s1 (7 vs 394 GPU-hours for near-identical accuracy), QLoRA's quality>size ablation.
	- _When wrong:_ if the base genuinely lacks the capability (novel reasoning, unfamiliar code), LoRA-Learns-Less shows you need more data AND higher rank or full FT.
	- _Abandon when:_ the base-vs-tuned delta plateaus below target despite clean, diverse data; that plateau means the capability is absent, not under-sampled.

	2. Use QLoRA by default and treat "forgets less" as a feature, not a compromise.

	- _Why:_ full FT rewrites high-rank directions across all weights and injects "intruder dimensions" that overwrite unrelated pretrained abilities; LoRA's low-rank cap keeps the update from traveling that far.
	- For one-behavior work you don't want to disturb anything else, so LoRA's capacity ceiling is the guardrail you want, not a weakness.
	- The alternative fails concretely: Raschka saw a model "unlearn arithmetic" from narrow SFT; Luo et al. show forgetting worsens with scale under continual full tuning.
	- QLoRA is single-GPU with no measured quality loss on tested tasks, so the default costs nothing.
	- _Evidence:_ Biderman et al. (LoRA forgets less, retains out-of-domain capability), intruder-dimensions paper (the mechanism), QLoRA (single-GPU, no perf loss), Raschka.
	- _Counter:_ full FT wins peak in-domain accuracy on genuinely hard new skills.
	- _Wrong if:_ your ONLY metric is target-task accuracy and base-capability retention is irrelevant; then use full FT or high-rank (r≈256) LoRA.

	3. Build the eval before you generate a single training row, make it adversarial, and trust base-vs-tuned deltas over any single judge score.

	- Without a pre-committed eval, "we fine-tuned a model" is unfalsifiable; the eval is what turns a vibe into a measurement.
	- Make it adversarial: small models fail in named ways (irrelevant clauses, reversed relations, mid-context facts) a happy-path test never triggers, so a fine-tune that clears only easy cases has learned the surface.
	- Read the delta, not the absolute score: judge biases (position, verbosity, self-preference) distort raw numbers but largely cancel when one judge compares two outputs with randomized order.
	- Prefer binary pass/fail over 1–5: it forces a crisp definition and is far more stable across judge runs.
	- _Evidence:_ Husain (evals first, 60–80% of effort; binary > 1–5), Zheng et al. (quantified biases; >80% human agreement), GSM-Symbolic and Reversal Curse (fragilities to target).
	- _Spiky corollary:_ a fine-tune shipped with no robustness eval should not be trusted, however good the demos.
	- _Wrong if:_ the behavior is code-verifiable by exact-match/regex (skip the judge), or you've validated high judge-human agreement on your task.

	4. Reliability is an engineering stack (data + constrained decoding + a calibration/abstention eval), not a bigger model or a better prompt.

	- Three levers fix three failure sources: fine-tuning raises the odds of correct content but can't guarantee valid form; constrained decoding guarantees form but can push the model off-path and hurt content; a calibration eval surfaces the residual confident-guess errors that only rewarding "I don't know" suppresses.
	- The deeper point: reliability is a tail property (the 1-in-50 malformed output breaks the pipeline), and scaling or prompting shifts the average while leaving the tail. Engineer the tail.
	- _Evidence:_ constrained-decoding literature (Outlines/XGrammar guarantee form), Why-Models-Hallucinate (residual errors are calibration failures), plus fine-tuning raising correct-behavior probability.
	- _Counter:_ constrained decoding adds ~2–5× overhead and can degrade content; over-constraining hides genuine uncertainty.
	- _Wrong if:_ the base is already schema-reliable and latency can't absorb decoding overhead.

	## DOK 3 — Insights

	1. The dataset is the deliverable; capability is mostly elicited, not installed.

	- LIMA (1,000 examples), s1 (1,000 traces; 394 vs 7 GPU-hours), and QLoRA's quality>size finding converge: a small, diverse, high-quality set beats a large noisy one.
	- _Why:_ pretraining did the heavy lifting, so SFT mainly selects a consistent output policy over existing knowledge; beyond a point you pay to re-teach what the model knows, and coverage matters more than count.
	- _Tension:_ LoRA-Learns-Less shows that for genuinely NEW capabilities (code, math) low-rank small-data tuning underperforms full FT—elicitation works when the capability is latent, not when it's absent.

	2. Distill the process, not the style.

	- Orca and DeepSeek-R1 show that transferring reasoning traces/explanations (not just final answers) is what moves a real capability into a small model; Orca warns that style-only imitation inflates apparent quality.
	- _Why:_ final-answer-only data teaches outputs without the intermediate computation, so it copies tone and collapses when the problem shifts; traces hand over the steps the student can internalize. It's also why style-rewarding evals overrate imitation-trained models.
	- _Tension:_ DeepSeek-R1 distillation still leaves persistent gaps for ≤7B students; process distillation narrows but doesn't close the teacher-student gap.

	3. LoRA/QLoRA is the right default for behavior instillation precisely because it forgets less.

	- Biderman et al. and the intruder-dimension paper show LoRA sacrifices peak capacity but preserves base competence—ideal when you want ONE behavior without collateral regression.
	- _Why:_ the low-rank cap limits how far the update moves the weights, so unrelated abilities survive; full FT has no such cap. For single-behavior work, preserving everything else is the goal, so the capacity limit is the property you want.
	- Raschka's "unlearned arithmetic" and Luo et al.'s forgetting-scales-with-size results show full FT/narrow data risks broad capability loss.
	- _Steelman for full FT:_ if the behavior is a hard new skill, full FT (or high-rank LoRA, α=2r, all modules) wins on the target metric.

	4. Evaluation must precede and drive training.

	- Hamel Husain (evals first, 60–80% of effort), LIMA (perplexity ≠ quality), and practitioner consensus all treat the eval harness as the real deliverable.
	- _Why:_ loss and perplexity don't track the behavior you care about, so without a behavioral eval you optimize a proxy; the eval is also the only signal for whether the fix is more data, different data, or a different method.
	- A base-vs-tuned LLM-judge comparison on a held-out set is the minimum viable measurement.

	5. LLM-as-judge is usable but biased; design around it.

	- Zheng et al. quantify position, verbosity, and self-preference biases (position can swing win-rate 10–15 points; length bias ~+17%).
	- _Why it's still workable:_ the biases are systematic, not random, so they distort absolute scores but largely cancel in A/B comparisons under a fixed judge with randomized order.
	- Mitigations: randomize order, control for length, don't judge with the same model family, prefer binary pass/fail (Husain).
	- _Tension:_ judges agree with humans >80%, so the biases are manageable, not disqualifying.

	6. Reliability comes from data + decoding, not prompting.

	- Fine-tuning raises the probability of correct behavior; constrained decoding (Outlines/XGrammar) guarantees output FORM.
	- _Why both:_ they govern different things (content odds vs form validity), and reliability is a tail property, so closing it takes both at once rather than a prompt that only shifts the average.
	- Neither alone suffices: constrained decoding can force off-distribution tokens and hurt content; fine-tuning can't guarantee 100% valid structure.

	7. Small models fail in predictable, testable ways; build adversarial evals for exactly those.

	- Named fragilities: GSM-Symbolic (irrelevant-clause drops up to 65%), Faith-and-Fate (multiplication 59%→4%), Reversal Curse (near-0% on reversed facts), Lost-in-the-Middle (U-shaped context use), and the hallucination-calibration work.
	- _Why it helps:_ because the failures are named and reproducible, the papers' own perturbations become your robustness suite, turning known weaknesses into a pre-deployment checklist.
	- A robustness eval should perturb numbers, add distractor clauses, reverse relations, and move key info to the middle.

	8. "Emergence" is partly a measurement artifact.

	- Schaeffer et al. show apparent emergence tracks metric choice.
	- _Why it matters here:_ a behavior that looks absent at small scale under exact-match may just be the metric masking steady progress; continuous metrics (edit distance, token-level, Brier) expose the gradient and prevent premature abandonment.

	## DOK 2 — Knowledge Tree

	### A. Data-centric AI & distillation

	LIMA: Less Is More for Alignment — Zhou et al., Meta AI, NeurIPS 2023. https://arxiv.org/abs/2305.11206. Peer-reviewed / researcher.
	- Fine-tuned LLaMa-65B on only 1,000 curated prompt-response pairs with standard supervised loss, no RLHF.
	- In a controlled human study, LIMA responses were equivalent or preferred to GPT-4 in 43% of cases; it beat DaVinci003 (RLHF-trained) and Alpaca-65B (trained on 52× more data). Even GPT-4 preferred LIMA's output over its own 19% of the time.
	- Hyperparameters: 15 epochs, LR 1e-5→1e-6, batch 32; perplexity did NOT correlate with generation quality, so checkpoints were selected manually on a 50-example dev set.
	- DOK 2 Summary: Proposes the "Superficial Alignment Hypothesis" — nearly all knowledge is learned in pretraining and alignment mainly teaches format/style, so a tiny high-quality dataset can suffice. Intellectual foundation for behavior-focused small-data fine-tuning.

	Self-Instruct / Stanford Alpaca — Wang et al. 2022 (https://arxiv.org/abs/2212.10560); Taori et al. 2023. Peer-reviewed + technical / practitioner.
	- Self-Instruct bootstraps 52K instructions from 175 seed tasks using a base LLM; improved the SUPER-NATURALINSTRUCTIONS baseline by 33%.
	- Alpaca reused the pipeline with text-davinci-003, generated 52K samples, and fine-tuned LLaMA-7B cheaply; AlpaGasus later showed filtering Alpaca's low-quality rows improves the model.
	- DOK 2 Summary: Established the template for distilling a teacher's behavior into a smaller student via synthetic instruction data — the direct ancestor of the "generate data from a frontier teacher" workflow.

	Evol-Instruct / WizardLM — Xu et al. 2023. Peer-reviewed / researcher.
	- Iteratively rewrites simple instructions into more complex/diverse ones (in-depth and in-breadth evolution).
	- DOK 2 Summary: Complexity and diversity of instructions, not just count, drive instruction-following quality — a concrete lever for dataset design.

	Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — Mukherjee et al., Microsoft 2023. https://arxiv.org/abs/2306.02707. Peer-reviewed / researcher.
	- 13B model trained to imitate GPT-4 reasoning via explanation traces and step-by-step thought, with ChatGPT as intermediate teacher; ~5M samples.
	- Beat Vicuna-13B by >100% on Big-Bench Hard and 42% on AGIEval.
	- Explicitly warns: naive imitation makes students copy the STYLE but not the REASONING of teachers, and shallow evaluation overestimates capability.
	- DOK 2 Summary: To transfer a behavior (not just surface style), distill the process — reasoning traces and explanations — and evaluate rigorously, because style mimicry masks capability gaps.

	Textbooks Are All You Need (phi-1) / phi-1.5 — Gunasekar et al. 2023 (https://arxiv.org/abs/2306.11644); Li et al. 2023 (https://arxiv.org/abs/2309.05463). Peer-reviewed / researcher.
	- phi-1: 1.3B params, trained on 6B tokens of "textbook-quality" web data + 1B tokens of GPT-3.5 synthetic textbooks/exercises; 50.6% pass@1 on HumanEval, 55.5% on MBPP. phi-1-small (350M) still hit 45% on HumanEval.
	- phi-1.5: 1.3B, matches models 5× larger on reasoning; challenges the notion that capability is determined solely by scale.
	- DOK 2 Summary: Data quality can substitute for scale — carefully curated/synthetic "textbook" data lets tiny models punch far above their weight, reinforcing the data-as-deliverable thesis.

	FineWeb-Edu / Cosmopedia / SmolLM-Corpus — Hugging Face (Penedo et al.). Technical / practitioner.
	- Cosmopedia: 25B tokens, 30M synthetic samples (Mixtral-generated textbooks/blogs/stories) — largest open synthetic dataset at release. FineWeb-Edu is an educational-quality filtered subset of FineWeb; FineMath ~50B tokens.
	- DOK 2 Summary: Open replications confirm that quality-filtering and synthetic generation at scale produce better small models than raw web crawl.

	Collective note: The synthetic-data surveys ("Best Practices and Lessons Learned on Synthetic Data," https://arxiv.org/abs/2404.07503; "A Survey on Post-training of LLMs," https://arxiv.org/abs/2503.06072) document that the recurring quality levers are complexity, diversity, and scale, and that quality filtering beats raw quantity. Tooling: distilabel/Argilla for programmatic synthetic-data and preference-pair pipelines.

	### B. PEFT / LoRA

	LoRA: Low-Rank Adaptation — Hu et al., Microsoft, ICLR 2022. https://arxiv.org/abs/2106.09685. Peer-reviewed / researcher.
	- Freezes base weights, learns low-rank update matrices A,B; far fewer trainable params and no inference latency once merged.
	- DOK 2 Summary: The core PEFT method — adapt behavior by learning a small rank-r perturbation instead of all weights.

	QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al., UW, NeurIPS 2023. https://arxiv.org/abs/2305.14314. Peer-reviewed / researcher.
	- Finetunes a 65B model on a single 48GB GPU. Innovations: 4-bit NormalFloat (NF4), double quantization (~0.37 bits/param saved, ~3GB on a 65B model), paged optimizers. Storage in NF4, compute de-quantized to bf16.
	- Guanaco reached 99.3% of ChatGPT on the Vicuna benchmark in 24h on one GPU; the authors trained >1,000 models.
	- Found data quality > dataset size for instruction following, and that MMLU does not predict chatbot quality.
	- DOK 2 Summary: Makes single-GPU fine-tuning of small models routine, with no measured performance loss vs 16-bit for the tested tasks.

	LoRA Learns Less and Forgets Less — Biderman et al., Columbia/Databricks Mosaic, TMLR 2024. https://arxiv.org/abs/2405.09673. Peer-reviewed / researcher.
	- On code and math, standard low-rank LoRA substantially underperforms full fine-tuning; full FT learns perturbations 10–100× higher rank than typical LoRA configs.
	- But LoRA forgets less (better retains out-of-domain capability) and maintains more diverse generations; mitigates forgetting more than weight decay/dropout. Example: code IFT full-FT scored 0.414 vs LoRA r=64's 0.509 on retention.
	- Best practices: target all modules (attention + MLP), use α=2r, rank ~16 as a starting point, ≥4 epochs; LoRA is highly LR-sensitive.
	- DOK 2 Summary: LoRA trades peak in-domain capacity for regularization/forgetting-resistance — a favorable trade when instilling a narrow behavior without wrecking base competence.

	LoRA vs Full Fine-tuning: An Illusion of Equivalence — Shuttleworth et al. 2024. https://arxiv.org/abs/2410.21228. Peer-reviewed / researcher.
	- LoRA introduces "intruder dimensions" — high-ranking singular vectors dissimilar to the pretrained weights — that full FT does not; causal intervention on them shows they drive forgetting.
	- DOK 2 Summary: LoRA and full FT reach structurally different solutions; that difference explains both LoRA's forgetting-resistance and its capacity limits.

	Sebastian Raschka — Practical Tips for Finetuning LLMs Using LoRA (Ahead of AI, 2023). https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms. Technical blog / practitioner.
	- QLoRA: ~33% memory savings for ~33% runtime increase. Optimizer choice barely matters. In his sweep r=256/α=512 gave the best performance.
	- Apply LoRA to ALL layers, not just K/V. Multiple epochs on small instruction sets can overfit and hurt.
	- Observed a model "unlearned arithmetic" because Alpaca lacked arithmetic examples — a concrete instance of narrow capability regression.
	- DOK 2 Summary: Hard-won operational defaults for LoRA; the "unlearned arithmetic" anecdote grounds the forgetting risk from narrow data.

	### C. SFT mechanics
	- Loss masking / train-on-completions-only: standard TRL SFTTrainer practice, but "Instruction Tuning With Loss Over Instructions" (Shi et al. 2024) questions always masking the instruction.
	- Chat templates: model-specific special tokens (LIMA introduced an explicit EOT token distinct from EOS); mismatched templates silently degrade behavior.
	- Hyperparameters: small datasets overfit fast; LIMA found perplexity uncorrelated with quality, so gate on behavioral eval, not loss. Use packing for throughput; watch effective batch size and LR.
	- DOK 2 Summary (collective): For narrow-behavior SFT the choices that matter most are correct chat-template/token handling, masking the prompt so loss applies only to completions, and early stopping by behavioral eval rather than loss.

	### D. Preference optimization

	DPO: Direct Preference Optimization — Rafailov et al., Stanford, NeurIPS 2023. https://arxiv.org/abs/2305.18290. Peer-reviewed / researcher.
	- Reparameterizes the RLHF reward so the optimal policy has a closed form; trains directly on preference pairs with a simple classification loss — no separate reward model, no sampling during training.
	- Matches or beats PPO-based RLHF on sentiment control, summarization, and single-turn dialogue while being far simpler and more stable.
	- DOK 2 Summary: Makes preference tuning accessible on small hardware; the go-to "stretch" step after SFT when you have chosen/rejected pairs.

	ORPO, KTO, SimPO, IPO — Hong et al. 2024; Ethayarajh et al. 2024; Meng et al. 2024 (https://arxiv.org/abs/2405.14734); Azar et al. 2024. Peer-reviewed / researcher.
	- ORPO: reference-model-free, folds preference into SFT via an odds-ratio term (one model, one dataset), more efficient but hyperparameter-sensitive.
	- KTO: learns from unpaired binary (good/bad) signals — no preference pairs required.
	- SimPO: reference-free, length-normalized reward + target margin; reported to outperform DPO and ORPO in its own experiments while tolerating noisier labels.
	- IPO: addresses DPO overfitting with a theoretically grounded objective.
	- DOK 2 Summary: A menu trading off data format (paired vs unpaired), compute (reference model or not), and stability; choose by what feedback you can cheaply generate.

	Collective note: Argilla's RLHF overview (https://argilla.io/blog/mantisnlp-rlhf-part-9/) and post-training-stack write-ups stress that DPO alone optimizes generic preference; controllable/domain behavior often needs SFT first, then a preference pass, and that all these methods are β- and length-term–sensitive.

	### E. Compression / quantization & reasoning distillation

	Distilling the Knowledge in a Neural Network — Hinton, Vinyals, Dean, 2015. https://arxiv.org/abs/1503.02531. Peer-reviewed / researcher. Origin of soft-target knowledge distillation.

	DeepSeek-R1 (distilled models) — DeepSeek-AI, 2025. https://arxiv.org/abs/2501.12948. Technical / researcher-practitioner.
	- Fine-tuned six dense students (1.5B–70B, Qwen2.5 & Llama) on 800K reasoning traces from R1; SFT only, no RL stage.
	- Per the R1 technical report, DeepSeek-R1-Distill-Qwen-32B scores 72.6% on AIME 2024, 94.3% on MATH-500, and 57.2% on LiveCodeBench — results that significantly outperform previous open-source models and are comparable to o1-mini.
	- Key claim: distilling from a strong teacher beats running large-scale RL directly on a small model.
	- DOK 2 Summary: Landmark evidence that a hard capability (long-CoT reasoning) transfers into small models purely via SFT on teacher traces — though persistent gaps remain for ≤7B students.

	s1: Simple Test-Time Scaling — Muennighoff et al., Stanford, EMNLP 2025. https://arxiv.org/abs/2501.19393. Peer-reviewed / researcher.
	- Per the paper: "After supervised finetuning the Qwen2.5-32B-Instruct language model on s1K and equipping it with budget forcing, our model s1-32B exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24)"; budget forcing (appending "Wait") scaled AIME24 from 50% to 57%.
	- s1K = 1,000 traces selected for difficulty, diversity, quality. Training on the full 59K vs the 1K cost 394 vs 7 H100-hours for marginal gains — data selection dominates.
	- Explicitly invokes LIMA's Superficial Alignment Hypothesis.
	- DOK 2 Summary: A second independent confirmation that ~1,000 well-chosen examples can activate a latent capability; the behavior is elicited, not taught from scratch.

	Quantization formats — GPTQ, AWQ, bitsandbytes (NF4), GGUF/llama.cpp. Docs / practitioner.
	- DOK 2 Summary (collective): Post-training quantization (GGUF for llama.cpp/CPU, AWQ/GPTQ for GPU) lets a tuned small model deploy on edge; 4-bit is the practical sweet spot.

	### F. Small-model landscape

	SmolLM3-3B — Hugging Face, released July 8, 2025. https://huggingface.co/HuggingFaceTB/SmolLM3-3B. Technical / practitioner.
	- 3B, trained on 11.2T tokens; dual reasoning (think/no_think); 64k context (128k via YaRN); fully open (data mixture, configs, 100+ intermediate checkpoints). Trained on 384 H100s for 24 days.
	- Outperforms Llama-3.2-3B and Qwen2.5-3B; competitive with Qwen3-4B and Gemma3-4B. Instruct variant optimized for reasoning and tool use.
	- DOK 2 Summary: Best current fully-open small base for tuning, with a documented recipe and a strong instruct variant.

	Collective note: Qwen3 (0.6/1.7/4B), Llama 3.2 (1B/3B), Gemma 3 small, Phi family, SmolLM2 (135M/360M/1.7B) are the practical base pool. Instruct variants ship chat templates and instruction-following priors; base variants give a cleaner slate but need format teaching. Tokenizer/number-tokenization differences matter (Goat attributed arithmetic gains to LLaMA's consistent digit tokenization).

	### G. Evaluation & LLM-as-judge

	Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al. 2023. https://arxiv.org/abs/2306.05685. Peer-reviewed / researcher.
	- Per the paper: "strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans" (validated on ~3K controlled expert votes + ~3K crowdsourced votes).
	- Documents position bias (most judges favor the first answer; only GPT-4 stays consistent in >60% of swapped cases), verbosity bias, and self-enhancement bias (judges favor their own outputs). Independent studies measure length bias around +17% for LLM judges vs ~+13% for humans.
	- DOK 2 Summary: LLM-as-judge is scalable and roughly human-aligned but carries measurable biases — always randomize position, control length, and avoid judging with the same model family.

	Hamel Husain — Your AI Product Needs Evals / LLM-as-a-Judge / Evals FAQ (hamel.dev). https://hamel.dev/blog/posts/evals/. Technical blog / practitioner.
	- Error analysis first: read ~100 traces; stop when ~20 traces reveal no new failure mode ("theoretical saturation").
	- Use a single domain-expert "benevolent dictator"; build a custom, friction-free data-viewing tool. Prefer binary pass/fail over 1–5 scales ("Critique Shadowing"); pass rate is a product decision, not necessarily 100%.
	- Per the Evals FAQ: "In the projects we've worked on, we've spent 60-80% of our development time on error analysis and evaluation"; he notes a ~70% pass rate can indicate an eval that is actually stress-testing the app.
	- DOK 2 Summary: Build evals before/alongside the model; the eval harness IS the improvement flywheel and the debugging infrastructure.

	Collective note: G-Eval (Liu et al. 2023) uses CoT + form-filling for judge scoring; JSONSchemaBench (https://arxiv.org/abs/2501.10868) shows models still struggle with real-world schemas. Watch for contamination, overfitting to the judge, and benchmark gaming.

	### H. Reliability & structured output

	Constrained decoding — Outlines, XGrammar, Guidance, llama.cpp GBNF; JSONSchemaBench — Willard & Louf 2023; Dong et al. 2024 (XGrammar); https://arxiv.org/abs/2501.10868. Docs + peer-reviewed / practitioner.
	- FSM/token-masking sets invalid-token logits to −∞, guaranteeing schema-valid output; Outlines compiles JSON schemas for ~O(1) valid-token lookup per step.
	- Overhead ranges from minimal to ~2–5× for naive implementations; XGrammar is current SOTA for low overhead. Output quality can suffer when constraints force the model off its preferred tokens, but constrained decoding sometimes IMPROVES task performance.
	- DOK 2 Summary: For structured-output behaviors, constrained decoding guarantees form while fine-tuning improves the model's tendency to produce correct content within that form — complementary, not substitutes.

	### I. Failure modes & limits (dedicated strand)

	GSM-Symbolic — Mirzadeh et al., Apple, 2024 (ICLR 2025). https://arxiv.org/abs/2410.05229. Peer-reviewed / researcher.
	- All tested models drop accuracy when only numbers change in a template; performance degrades further as clause count rises.
	- Per the paper: "adding seemingly relevant but ultimately irrelevant information to problems, we demonstrate substantial performance drops (up to 65%) across all state-of-the-art models" — Phi-3-mini experienced over a 65% drop on the GSM-NoOp variant.
	- Interpretation: reasoning resembles sophisticated pattern matching, not robust logic. Contested: Ivanova and others critique the statistical rigor; treat as evidence of fragility, not proof of "no reasoning."

	Faith and Fate: Limits of Transformers on Compositionality — Dziri et al., NeurIPS 2023. https://arxiv.org/abs/2305.18654. Peer-reviewed / researcher.
	- GPT-4 zero-shot: 59% on 3×3-digit multiplication, dropping to 4% on 4×4; a few-shot scratchpad lifts 3×3 to 92% but stays near 0 on the hardest cases.
	- Transformers reduce compositional reasoning to "linearized subgraph matching"; under stated assumptions the probability of error converges toward 1 as problem size grows. Exhaustive fine-tuning (~1.8M multiplication pairs) generalized in-distribution but "utterly failed" out-of-distribution.

	Why Language Models Hallucinate — Kalai, Nachum, Vempala, Zhang (OpenAI/Georgia Tech), 2025. https://arxiv.org/abs/2509.04664. Researcher.
	- Frames hallucination as statistical error: generative error rate is lower-bounded by the "Is-It-Valid" binary misclassification rate — even with clean data, generating valid outputs is statistically harder than classifying validity.
	- Post-training persists/worsens hallucination because benchmarks reward confident guessing over "I don't know"; the proposed fix is reworking eval metrics to reward calibrated uncertainty/abstention.

	Lost in the Middle: How Language Models Use Long Contexts — Liu et al., TACL 2024. https://arxiv.org/abs/2307.03172. Peer-reviewed / researcher.
	- U-shaped accuracy: models use info at the start/end of context well and degrade sharply in the middle, even in explicitly long-context models.

	Are Emergent Abilities of Large Language Models a Mirage? — Schaeffer, Miranda, Koyejo, Stanford, NeurIPS 2023. https://arxiv.org/abs/2304.15004. Peer-reviewed / researcher.
	- Core claim: apparent "emergence" appears "due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale." Nonlinear/discontinuous metrics (exact match) manufacture sharp jumps; linear/continuous metrics (token edit distance, Brier score) show smooth, predictable scaling. They even induce apparent emergence in vision models by choosing metrics.

	The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" — Berglund et al., ICLR 2024. https://arxiv.org/abs/2309.12288. Peer-reviewed / researcher.
	- Models trained on "A is B" do not generalize to "B is A": "near 0% accuracy on reversals" across GPT-3-350M, Llama-7B, and GPT-3-175B; the log-probability of the correct reversed name is no higher than a random name. Data augmentation with paraphrases did not fix it. (In-context, models CAN reverse; the curse is about stored/trained knowledge.)

	An Empirical Study of Catastrophic Forgetting in LLMs During Continual Fine-tuning — Luo et al., 2023 (EMNLP 2025 proceedings). https://arxiv.org/abs/2308.08747. Peer-reviewed / researcher.
	- Catastrophic forgetting observed across 1B–7B models, and severity INTENSIFIES with scale during continual instruction tuning. Decoder-only BLOOMZ forgets less than encoder-decoder mT0. General instruction tuning (Alpaca vs LLaMA) mitigates forgetting; continual tuning can also reduce some biases (e.g., gender bias).

	Alignment tax — Ouyang et al. (InstructGPT), NeurIPS 2022. https://arxiv.org/abs/2203.02155; confirmed by Lin et al., EMNLP 2024, https://arxiv.org/abs/2309.06256.
	- InstructGPT coined the term: RLHF "comes at the cost of lower performance on certain tasks," with regressions on SQuAD, DROP, HellaSwag, and WMT'15 Fr→En. Mitigation: PPO-ptx (mixing pretraining-distribution gradients into PPO) largely closes the gap without hurting labeler scores. Lin et al. independently confirm the tax on OpenLLaMA-3B/Mistral-7B and note DPO induces less tax than other RLHF algorithms.

	DOK 2 Summary (collective for I): Small (and large) LMs fail systematically at compositional/multi-step reasoning, robustness to irrelevant info, symmetric fact retrieval, middle-of-context use, and calibrated uncertainty. Fine-tuning a narrow behavior can cause narrow forgetting and an alignment tax. Design evals to probe these exact fragilities.

	### J. Practitioner playbooks
	- Hamel Husain (evals), Sebastian Raschka (LoRA experiments; "Build a Reasoning Model From Scratch"), Unsloth / Axolotl docs (fast QLoRA recipes; HF alignment-handbook), Argilla / distilabel (data + preference pipelines).
	- DOK 2 Summary: The consensus workflow — define behavior + eval → generate/curate teacher data → QLoRA SFT → base-vs-tuned judge eval → optional DPO → quantize/deploy.

	### K. Forums / community
	- r/LocalLLaMA, Hugging Face forums, Unsloth/Axolotl Discords, Hacker News threads on QLoRA/DeepSeek-R1. Useful for hardware-specific gotchas, chat-template bugs, and reproductions; treat single-poster claims as anecdotes to verify.

	## Recommendations

	Stage 0 — Define and instrument (before any training). Write the ONE behavior as a falsifiable spec. Build the eval harness first: a held-out behavioral set plus an adversarial set that perturbs numbers, injects irrelevant clauses (GSM-Symbolic style), reverses relations (reversal curse), and shifts key info to the middle (lost-in-the-middle). Decide the judge protocol now — binary pass/fail, randomized answer order, a different model family as judge. Benchmark that gates progress: baseline the untuned model on this harness; the whole project is measured as the base-vs-tuned delta.

	Stage 1 — Data. Generate ~1,000–2,000 examples from a frontier teacher, distilling the process (reasoning traces/explanations), then filter for quality and diversity (AlpaGasus-style). Use distilabel/Argilla for pipelines. Threshold to add more data: only if the base-vs-tuned delta plateaus below target with clean data — that signals a missing latent capability, not a data-quantity problem.

	Stage 2 — Train. QLoRA (NF4, double quant) on a fully-open small base (SmolLM3-3B or Qwen3-1.7B/4B). Defaults: target all modules, α=2r, r≈16 (raise to 64–256 only if the target metric is capacity-bound), ≥3–4 epochs, sweep LR (LoRA is LR-sensitic). Mask the prompt (loss on completions only); verify the exact chat template. Early-stop on the behavioral eval, not loss. Escalate to full FT or high-rank LoRA if: the behavior is a genuinely new hard skill and low-rank LoRA underperforms full FT on the target metric.

	Stage 3 — Preference tuning (stretch). Only after SFT clears most of the target. Build chosen/rejected pairs (or unpaired good/bad for KTO) and run DPO (or ORPO to fold it into one stage). Re-run the full eval to check for an alignment tax — regression on base capabilities. Threshold to keep it: preference tuning must improve the target behavior without dropping retained-capability checks.

	Stage 4 — Reliability & deploy. Add constrained decoding (Outlines/XGrammar) for any structured output; measure the quality cost vs the validity gain. Quantize to GGUF (llama.cpp) or AWQ/GPTQ for deployment. Kill-switch: if constrained decoding measurably degrades content quality and the base is already schema-reliable, drop it.

	## Caveats
	- Contested reasoning claims. GSM-Symbolic's "not genuine reasoning" framing is disputed (Ivanova and others critique statistical rigor); treat it as strong evidence of fragility, not proof of no reasoning.
	- Vendor vs independent. DeepSeek-R1, SmolLM3, and QLoRA headline numbers are self-reported by the authors/vendors; the s1, LIMA, MT-Bench, GSM-Symbolic, Faith-and-Fate, and LoRA-Learns-Less findings are peer-reviewed but some rely on GPT-4-as-judge, which carries the biases documented in Section G.
	- Alignment-tax magnitude is method-dependent. InstructGPT quantifies regressions qualitatively; the headline drop is largely mitigated by PPO-ptx and DPO, so the "tax" is real but not fixed — measure it on your own benchmarks.
	- Scale gap in the evidence. Several marquee results (LIMA-65B, s1-32B, Guanaco-65B, Faith-and-Fate on GPT-4) are on models much larger than the 0.6B–4B target range; the mechanisms (superficial alignment, forgetting, compositional limits) transfer, but exact numbers will not. Small models show these failure modes at least as strongly.
	- Fast-moving field. Model releases (SmolLM3, Qwen3, DeepSeek-R1) and preference methods (SimPO/ORPO/KTO) are recent; recheck for newer bases and recipes before committing.
	- Two searches were cut short by budget (a direct pull of the emergent-abilities and reversal-curse primary PDFs); those facts were verified via a dedicated sub-search against the primary arXiv abstracts and are cited accordingly.