metadata
license: apache-2.0
base_model: fitleech/lm-2b
tags:
- text-generation
- chronogpt
- sn38
- cutoff-2013
- pytorch
lm-2b-typemix-2013
A 2B-parameter ChronoGPT causal language model fine-tuned from
fitleech/lm-2b for one epoch on
fitleech/sn38-type-mix-2013-500m.
The training corpus uses a 2013 knowledge cutoff and a 20% / 50% / 30%
Type1 / Type2 / Type3 mixture. Weights are the completed epoch-1 checkpoint.
Architecture
- 52 layers
- 12 attention heads
- 1536 model dimension
- 50,304-token GPT-2-compatible vocabulary
- BF16 checkpoint in safetensors format