File size: 2,780 Bytes
6923f6b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 | ---
license: mit
language:
- en
tags:
- language-model
- transformer
- pytorch
- from-scratch
- tiny-stories
datasets:
- TinyStories
library_name: transformers
pipeline_tag: text-generation
---
# Sage 1B
A **custom 1.286 billion parameter** language model built entirely from scratch — no base models, no fine-tuning, no dependencies on existing LLM frameworks.
## Architecture
| Parameter | Value |
|-----------|-------|
| Parameters | 1,286,155,776 |
| Layers | 30 |
| Hidden Size | 1536 |
| Attention Heads | 12 |
| Head Dimension | 128 |
| Intermediate Size | 6144 |
| Vocabulary | 50,000 (BPE) |
| Max Sequence Length | 128 tokens |
| Activation | SwiGLU |
| Position Encoding | Rotary (RoPE) |
| Normalization | RMSNorm |
| Precision | FP16 / FP32 |
## Key Features
- **Built from scratch** — Custom PyTorch implementation. Not a derivative of any existing model.
- **BPE Tokenizer** — Trained a 50,000-token BPE tokenizer on the TinyStories dataset.
- **Modern Architecture** — SwiGLU activations, Rotary Position Embeddings (RoPE), RMSNorm.
- **Open Source** — MIT license. Weights, training code, and inference code are all available.
- **GGUF Format** — Available for use with llama.cpp, Ollama, and other GGUF-compatible runners.
## Usage
### With Hugging Face Hub
```python
from huggingface_hub import hf_hub_download
import torch, json
from tokenizers import Tokenizer
config_path = hf_hub_download('itriedcoding/Sage-1B', 'config.json')
tokenizer_path = hf_hub_download('itriedcoding/Sage-1B', 'tokenizer.json')
weights_path = hf_hub_download('itriedcoding/Sage-1B', 'pytorch_model_state.bin')
cfg = json.load(open(config_path))
tok = Tokenizer.from_file(tokenizer_path)
```
### With GGUF (llama.cpp)
```bash
wget https://huggingface.co/itriedcoding/Sage-1B/resolve/main/sage-1b-f16.gguf
./main -m sage-1b-f16.gguf -p "Once upon a time" -n 50
```
### Web Interface
Chat with the model at: https://sage-ai.vercel.app/chat
### API
```bash
curl -X POST https://sage-ai.vercel.app/api/v1/chat \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"message": "Tell me a story"}'
```
## Training
The model was trained on the **TinyStories** dataset — a synthetic dataset of short stories designed for training compact language models. Training was performed on CPU with limited resources, making this a proof-of-concept for building LLMs from scratch without GPU access.
## Files
| File | Size | Description |
|------|------|-------------|
| `pytorch_model_state.bin` | 2.4 GB | FP16 model weights |
| `sage-1b-f16.gguf` | 2.4 GB | GGUF format for llama.cpp |
| `config.json` | 1 KB | Model hyperparameters |
| `tokenizer.json` | 12 MB | BPE tokenizer (50K vocab) |
| `modeling_sage_1b.py` | 6 KB | Model architecture code |
## License
MIT
|