| --- |
| license: apache-2.0 |
| base_model: fitleech/lm-2b |
| tags: |
| - text-generation |
| - chronogpt |
| - sn38 |
| - cutoff-2013 |
| - pytorch |
| --- |
| |
| # lm-2b-typemix-2013 |
|
|
| A 2B-parameter ChronoGPT causal language model fine-tuned from |
| [`fitleech/lm-2b`](https://huggingface.co/fitleech/lm-2b) for one epoch on |
| [`fitleech/sn38-type-mix-2013-500m`](https://huggingface.co/datasets/fitleech/sn38-type-mix-2013-500m). |
|
|
| The training corpus uses a 2013 knowledge cutoff and a 20% / 50% / 30% |
| Type1 / Type2 / Type3 mixture. Weights are the completed `epoch-1` checkpoint. |
|
|
| ## Architecture |
|
|
| - 52 layers |
| - 12 attention heads |
| - 1536 model dimension |
| - 50,304-token GPT-2-compatible vocabulary |
| - BF16 checkpoint in safetensors format |
|
|
