File size: 1,969 Bytes
33b53d8 05c7782 33b53d8 05c7782 33b53d8 05c7782 33b53d8 05c7782 33b53d8 05c7782 33b53d8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | ---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- open-reason
- causal-lm
- cpu
datasets:
- theworker02/open-reason
base_model: gpt2-scratch
---
# Open Reason large (CPU)
This is a **large** GPT-2-style causal LM trained from scratch on the Open Reason SFT split. It is larger than [`theworker02/open-reason-medium`](https://huggingface.co/theworker02/open-reason-medium) (13,867,008 parameters) and is **not** a 1B model and is **not** `theworker02/open-reason-1b`.
Weights live on this Hub repo. They are not stored in the GitHub git tree.
## Training facts
- **Parameters:** 91,544,064
- **Architecture:** GPT-2-style from scratch; `n_layer=12`, `n_embd=768`, `n_head=12`, `vocab_size=8192`, `max_seq_len=256`
- **Steps:** 400 (batch size 2)
- **Final training loss:** 5.7361 (next-token NLL on training batches; not a benchmark score)
- **SFT rows:** 3,175 from `data/release/all.jsonl`
- **Dataset:** [`theworker02/open-reason`](https://huggingface.co/datasets/theworker02/open-reason) pipeline **1.4.0**
- **License:** Apache-2.0
- **Hardware:** AMD Ryzen 9 9950X host **CPU**; `torch 2.12.0+cpu`; `torch.cuda.is_available()=False`; Docker **not installed** and **not used**; `nvidia-smi` not present. NVIDIA CUDA was not used. AMD GPU / ROCm / DirectML were not used.
## Related
- Dataset: https://huggingface.co/datasets/theworker02/open-reason
- Small (~1.3M): https://huggingface.co/theworker02/open-reason-small
- Medium (13,867,008): https://huggingface.co/theworker02/open-reason-medium
- Large (this repo, 91,544,064): https://huggingface.co/theworker02/open-reason-large
- GitHub: https://github.com/theworker02/open-reason
No Reddit sources.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("theworker02/open-reason-large")
model = AutoModelForCausalLM.from_pretrained("theworker02/open-reason-large")
```
|