|
|
--- |
|
|
library_name: transformers |
|
|
license: mit |
|
|
datasets: |
|
|
- YourDatasetName/if-applicable |
|
|
language: |
|
|
- en |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- transformer |
|
|
- language-model |
|
|
- experimental |
|
|
--- |
|
|
|
|
|
# **SmalLM** |
|
|
|
|
|
<hr> |
|
|
<div align="center"> |
|
|
<a href="https://github.com/azrails/SmalLm" target="_blank" style="margin: 2px;"> |
|
|
<img alt="GitHub" src="https://img.shields.io/badge/GitHub-SmalLM-181717?logo=github" style="display: inline-block; vertical-align: middle;"/> |
|
|
</a> |
|
|
<a href="https://github.com/azrails/SmalLm/blob/main/LICENSE" style="margin: 2px;"> |
|
|
<img alt="License" src="https://img.shields.io/badge/License-MIT-blue.svg" style="display: inline-block; vertical-align: middle;"/> |
|
|
</a> |
|
|
</div> |
|
|
|
|
|
SmalLM is a series of small transformer models built from scratch for language modeling. This project is designed to explore innovative approaches to transformer architectures through modular pipelines for pretraining, fine-tuning, and alignment. |
|
|
|
|
|
## Uses |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("Azrail/smallm_70") |
|
|
model = AutoModelForCausalLM.from_pretrained("Azrail/smallm_70", trust_remote_code=True) |
|
|
inputs = tokenizer("How are you?", return_tensors="pt") |
|
|
|
|
|
out = model.generate(**inputs, max_new_tokens=100) |
|
|
print(tokenizer.batch_decode(out)) |
|
|
``` |
|
|
|
|
|
## Model Details** |
|
|
**Key Features:** |
|
|
|
|
|
1. Grouped Query Attention (GQA). |
|
|
|
|
|
2. Mixture-of-Experts with auxiliary loss-free balancing. |
|
|
|
|
|
3. ALiBi (Attention with Linear Biases) or Rotary Position Embedding (RoPE). |
|
|
|
|
|
4. NTK-by-parts RoPE interpolation for extends context length. |
|
|
|
|
|
**Pre-Training**: |
|
|
|
|
|
| Model | Training Data | Steps | Content Length | Tokens | LR | Batch Size | Precision | |
|
|
|----------------------|-------------------------------------------------------------------------------|-------|----------------|--------|-------|------------|-----------| |
|
|
| [SmalLM-70M](https://huggingface.co/Azrail/smallm_70) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | 70k | 1024 | 18B | 1e-3 | 0.25M | bfloat16 | |
|
|
| [SmalLM-150M](#) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | - | 1024 | - | - | - | bfloat16 | |
|
|
| [SmalLM-350M](#) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | - | 1024 | - | - | - | bfloat16 | |
|
|
| [SmalLM-500M](#) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | - | 1024 | - | - | - | bfloat16 | |
|
|
|
|
|
**Evaluation**: |
|
|
Evaluation runing with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) |
|
|
|
|
|
| Model | MMLU | ARC easy/hard | PIQA | HellaSwag | OBQA | Winogrande | |
|
|
|----------------------|------|----------------|-------|-----------|-------|------------| |
|
|
| [SmalLM-70M](#) | 25.33 | 51.47/25.68 | 61.75 | 30.31 | 30.8 | 50.83 | |
|
|
| [SmalLM-150M](#) | - | - | - | - | - | - | |
|
|
| [SmalLM-350M](#) | - | - | - | - | - | - | |
|
|
| [SmalLM-500M](#) | - | - | - | - | - | - | |
|
|
|
|
|
|
|
|
**Procedure**: |
|
|
|
|
|
[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://api.wandb.ai/links/azrails-main/58rwb1yb) |
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- Transformers 4.50.3 |
|
|
- Pytorch 2.6.0+cu126 |
|
|
- Datasets 3.5.0 |
|
|
- Tokenizers 0.21.1 |
