| --- |
| language: |
| - en |
| license: apache-2.0 |
| tags: |
| - spm |
| - foundation-model |
| - causal-lm |
| - from-scratch |
| model-index: |
| - name: SPM-50M-Alpha |
| results: [] |
| --- |
| |
| # SPM-70M-Vertigo |
|
|
| **SPM-70M-Vertigo** is the first model in the SPM (Strategic Pretrained Model) family. |
| A ~71M parameter English foundation model trained from scratch. |
|
|
| ## Model Architecture |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Architecture | Decoder-only Transformer (Llama-style) | |
| | Parameters | ~71M | |
| | Hidden Size | 512 | |
| | Layers | 12 | |
| | Attention Heads | 8 | |
| | KV Heads | 4 | |
| | Vocab Size | 32000 | |
| | Max Sequence Length | 2048 | |
| | Positional Encoding | RoPE | |
| | Normalization | RMSNorm | |
| | Activation | SwiGLU | |
|
|
| ## Training |
|
|
| - **Dataset**: FineWeb (sample-10BT) |
| - **Tokenizer**: SentencePiece BPE (vocab=32000) |
| - **Precision**: BF16 |
| - **Effective Batch Size**: 128 |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import LlamaTokenizer, LlamaForCausalLM |
| |
| model = LlamaForCausalLM.from_pretrained("SPM-50M-Alpha") |
| tokenizer = LlamaTokenizer.from_pretrained("SPM-50M-Alpha") |
| |
| inputs = tokenizer("The meaning of life is", return_tensors="pt") |
| outputs = model.generate(**inputs, max_new_tokens=50) |
| print(tokenizer.decode(outputs[0])) |
| ``` |
|
|
| ## License |
|
|
| Apache 2.0 |
|
|
