File size: 2,348 Bytes
8a20715 df6f372 8a20715 df6f372 05f1ff3 df6f372 3323075 df6f372 dbbc8bb df6f372 dbbc8bb 3323075 df6f372 dbbc8bb 3323075 df6f372 3323075 313e8cc dbbc8bb 4758b31 6735bb6 4758b31 dbbc8bb 78259e2 4758b31 dbbc8bb 79c0f84 cf3b11a b142369 686c34c b142369 686c34c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | ---
license: apache-2.0
language:
- en
tags:
- ablations
- datasets
- 1m
- llama
- custom_tokenizer
---
# DataBench
Rather than training one model, we trained multiple models, each on a different dataset. Everything else was kept the same, so that the only changing variable would be the dataset itself.
## Model Architecture
- Base Architecture: `LlamaForCasualLM`
- Tokenizer: [`Harley-ml/Dillionv2-1.3M`](https://huggingface.co/Harley-ml/Dillionv2-1.3M)
- Transformers Version: `5.13.1`
- Hidden Size: `128`
- Vocab Size: 2564
- Number of Layers: `8`
- Number of Heads: `4`
- Number of KV Heads: `2`
- Intermediate Size: `344`
- Head Dim: `32`
- Max Position Embeddings: `256`
- RoPe Theta: `2500.0`
- Tie Word Embeddings: `true`
- Hidden Activation: `silu`
- MLP Bias: `false`
- Initializer Range: `0.2`
- RMS Norm Eps: `1e-06`
- Pretraining Tp: `1`
- Use Cache: `false`
- **Total Parameters**: `1,780,352`
## Training Setup
- Epochs: `1`
- Max Steps: `-1.0`
- Batch Size: `400`
- Sequence Length: `256`
- Gradient Accumulation: `2`
- Gradient Clipping: `1.0`
- Gradient Checkpointing: `true`
- Learning Rate: `2.5e-3`
- Eval Split: `0.00165`
- Weight Decay: `0.01`
- Optimizer: `AdamW`
- AdamW Betas: `(0.9, 0.95)`
- AdamW Eps: `1e-8`
- Scheduler: `WSD`
- WSD Warmup Ratio: `0.015`
- WSD Stable Ratio: `0.78`
- WSD Decay Ratio: `0.20`
- WSD Minium LR Ratio: `0.0`
- WSD Number of Cycles: `0.5`
- DType: `float16`
- Torch.Compile: `true`
- DataLoader Workers: `2`
- Seed: `311`
# Results
Accuracy is normalized by length and shown as a percentage.
| Dataset | ARC-Easy | HellaSwag | PIQA | **Avg ↑** |
| :---------------- | ---------: | ---------: | ---------: | ---------: |
| FineWeb | 27.82% | 26.89% | 52.83% | **35.85%** |
| DCLM-1.0-Baseline | 29.08% | 27.06% | 52.23% | **36.12%** |
| DOAB | 31.14% | 28.00% | 52.18% | **37.11%** |
| Project Gutenberg | 25.84% | 24.63% | 50.21% | **33.56%** |
| EOT-2004-Raw | 29.08% | 27.54% | 51.31% | **35.98%** |
| Wikipedia | 28.28% | 27.61% | 51.14% | **35.68%** |
(Will add ArithMark-3.0 later)
## Notice
This is a work in progress and is currently not completed. By the end of this project, we aim to have tested over 50 datasets.
## License
Apache 2.0. |