documents/EXPERIMENTS_LOG.md

# 🧪 Experiments & Post-Mortems

Building an LLM from scratch requires breaking it. Here are the core failures encountered during development and how they were engineered around.

### Incident 1: The "Alpaca Crash" (Vocabulary Mismatch)
* **Attempt:** Fine-tuning the 70M base model on the `tatsu-lab/alpaca` instruction dataset.
* **Failure:** Validation loss spiked to `6.11` and PPL exploded to `452.38`.
* **Diagnosis:** Alpaca contains highly complex, collegiate-level tasks. Our ~70M base model was pre-trained on toddler-level stories. The model suffered catastrophic forgetting as it attempted to map massive unknown vocabularies to its tiny latent space.
* **Resolution:** Pivoted to filtering simpler datasets and capping sequence lengths.

### Incident 2: The "Wikitext NaN Explosion"
* **Attempt:** Continual pre-training on `wikitext-2-raw-v1` using Mixed Precision (FP16).
* **Failure:** Gradients exploded, resulting in `Loss: NaN`. Inference output resulted in severe hallucination loops (e.g., *"is and lollitter and lollbracotled"*).
* **Diagnosis:** The `wikitext` dataset contained raw tokenizer artifacts (e.g., `@-@`) which clashed with GPT-2 BPE. Furthermore, high weight decay coupled with FP16 underflow triggered math errors during backward passes.
* **Resolution:** Rolled back the model weights. Disabled `torch.amp.autocast` (falling back to pure FP32), reduced `weight_decay` to `0.01`, and enforced strict data sanitization.

### Incident 3: Synthetic Memorization (Overfitting)
* **Attempt:** Training on 1,500 highly repetitive synthetic QA pairs to fix the Alpaca crash.
* **Failure:** Validation Loss dropped to `0.16` and PPL to `1.18`. The model began reciting dataset lines verbatim, ignoring user prompts.
* **Diagnosis:** Severe Overfitting due to lack of dataset variance.
* **Resolution:** Scaled up to `databricks-dolly-15k`, applied 10% Dropout across all transformer modules, and randomized batch sampling. Generalization was successfully restored.