File size: 825 Bytes
c3c1935 84c34bf c3c1935 84c34bf c3c1935 84c34bf c3c1935 84c34bf c3c1935 84c34bf c3c1935 84c34bf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | ---
license: mit
library_name: pytorch
pipeline_tag: text-generation
tags:
- text-generation
- pretraining
- decoder-only
- from-scratch
- 11m
- 1096-context
- rope
- rmsnorm
- swiglu
---
# FrontD.11m
FrontD.11m is an approximately 11-million-parameter decoder-only
language model trained completely from scratch.
## Architecture
- Parameters: 10,996,608
- Context: 1096
- Vocabulary: 16000
- Hidden size: 256
- Layers: 6
- Attention heads: 16
- FFN size: 1152
- RoPE
- RMSNorm
- SwiGLU
- Tied input/output embeddings
## Pretraining dataset
`Plans11/Organized_PreTrain_1k_Context`
## Training
Training is performed in 20,000-example sessions.
A persistent SHA-256 example ledger records only examples whose
training session has successfully completed.
The model is initialized randomly and pretrained from scratch.
|