File size: 695 Bytes
3cce15a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | ---
license: apache-2.0
base_model: fitleech/lm-2b
tags:
- text-generation
- chronogpt
- sn38
- cutoff-2013
- pytorch
---
# lm-2b-typemix-2013
A 2B-parameter ChronoGPT causal language model fine-tuned from
[`fitleech/lm-2b`](https://huggingface.co/fitleech/lm-2b) for one epoch on
[`fitleech/sn38-type-mix-2013-500m`](https://huggingface.co/datasets/fitleech/sn38-type-mix-2013-500m).
The training corpus uses a 2013 knowledge cutoff and a 20% / 50% / 30%
Type1 / Type2 / Type3 mixture. Weights are the completed `epoch-1` checkpoint.
## Architecture
- 52 layers
- 12 attention heads
- 1536 model dimension
- 50,304-token GPT-2-compatible vocabulary
- BF16 checkpoint in safetensors format
|