| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - ablations |
| - datasets |
| - 1m |
| - llama |
| - custom_tokenizer |
| --- |
| |
| # DataBench |
|
|
| Rather than training one model, we trained multiple models, each on a different dataset. Everything else was kept the same, so that the only changing variable would be the dataset itself. |
|
|
| ## Model Architecture |
|
|
| - Base Architecture: `LlamaForCasualLM` |
| - Tokenizer: [`Harley-ml/Dillionv2-1.3M`](https://huggingface.co/Harley-ml/Dillionv2-1.3M) |
| - Transformers Version: `5.13.1` |
| - Hidden Size: `128` |
| - Vocab Size: 2564 |
| - Number of Layers: `8` |
| - Number of Heads: `4` |
| - Number of KV Heads: `2` |
| - Intermediate Size: `344` |
| - Head Dim: `32` |
| - Max Position Embeddings: `256` |
| - RoPe Theta: `2500.0` |
| - Tie Word Embeddings: `true` |
| - Hidden Activation: `silu` |
| - MLP Bias: `false` |
| - Initializer Range: `0.2` |
| - RMS Norm Eps: `1e-06` |
| - Pretraining Tp: `1` |
| - Use Cache: `false` |
| - **Total Parameters**: `1,780,352` |
|
|
| ## Training Setup |
|
|
| - Epochs: `1` |
| - Max Steps: `-1.0` |
| - Batch Size: `400` |
| - Sequence Length: `256` |
| - Gradient Accumulation: `2` |
| - Gradient Clipping: `1.0` |
| - Gradient Checkpointing: `true` |
| - Learning Rate: `2.5e-3` |
| - Eval Split: `0.00165` |
| - Weight Decay: `0.01` |
| - Optimizer: `AdamW` |
| - AdamW Betas: `(0.9, 0.95)` |
| - AdamW Eps: `1e-8` |
| - Scheduler: `WSD` |
| - WSD Warmup Ratio: `0.015` |
| - WSD Stable Ratio: `0.78` |
| - WSD Decay Ratio: `0.20` |
| - WSD Minium LR Ratio: `0.0` |
| - WSD Number of Cycles: `0.5` |
| - DType: `float16` |
| - Torch.Compile: `true` |
| - DataLoader Workers: `2` |
| - Seed: `311` |
|
|
| # Results |
|
|
| Accuracy is normalized by length and shown as a percentage. |
| | Dataset | ARC-Easy | HellaSwag | PIQA | **Avg ↑** | |
| | :---------------- | ---------: | ---------: | ---------: | ---------: | |
| | FineWeb | 27.82% | 26.89% | 52.83% | **35.85%** | |
| | DCLM-1.0-Baseline | 29.08% | 27.06% | 52.23% | **36.12%** | |
| | DOAB | 31.14% | 28.00% | 52.18% | **37.11%** | |
| | Project Gutenberg | 25.84% | 24.63% | 50.21% | **33.56%** | |
| | EOT-2004-Raw | 29.08% | 27.54% | 51.31% | **35.98%** | |
| | Wikipedia | 28.28% | 27.61% | 51.14% | **35.68%** | |
|
|
| (Will add ArithMark-3.0 later) |
| ## Notice |
|
|
| This is a work in progress and is currently not completed. By the end of this project, we aim to have tested over 50 datasets. |
|
|
| ## License |
|
|
| Apache 2.0. |
