File size: 1,055 Bytes
233d036 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | ---
language:
- en
license: other
library_name: pytorch
tags:
- causal-lm
- from-scratch
- gpt
- safetensors
- small-language-model
- meet25m
---
# Meet25M Base
A small GPT-style causal language model trained from scratch.
## Model
- Architecture: GPT-style decoder-only Transformer
- Approx size: ~25M parameters
- Context length: 1024
- Tokenizer: custom byte-level BPE
- Positional encoding: RoPE
- Normalization: RMSNorm
- MLP: SwiGLU
- Embeddings: tied input/output embeddings
## Training Data Mix
Target pretraining mix:
- FineWeb-Edu
- FineWeb general
- Wikipedia
- OpenWebMath
- Project Gutenberg
- StackOverflow / Stack Exchange style posts
- CodeSearchNet
Total target: ~250M training tokens.
## Files
- `model.safetensors` — safetensors checkpoint
- `config.json` — model config
- `tokenizer/` — tokenizer files
- `safetensors_info.json` — checkpoint metadata
## Loading
This is not a standard Transformers `AutoModelForCausalLM` checkpoint.
Use the custom GPT class from the training script and load `model.safetensors`.
|