File size: 3,712 Bytes
034688c 34f3cd9 034688c 34f3cd9 034688c 34f3cd9 034688c 34f3cd9 034688c 34f3cd9 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 |
---
library_name: transformers
license: mit
datasets:
- YourDatasetName/if-applicable
language:
- en
pipeline_tag: text-generation
tags:
- transformer
- language-model
- experimental
---
# **SmalLM**
<hr>
<div align="center">
<a href="https://github.com/azrails/SmalLm" target="_blank" style="margin: 2px;">
<img alt="GitHub" src="https://img.shields.io/badge/GitHub-SmalLM-181717?logo=github" style="display: inline-block; vertical-align: middle;"/>
</a>
<a href="https://github.com/azrails/SmalLm/blob/main/LICENSE" style="margin: 2px;">
<img alt="License" src="https://img.shields.io/badge/License-MIT-blue.svg" style="display: inline-block; vertical-align: middle;"/>
</a>
</div>
SmalLM is a series of small transformer models built from scratch for language modeling. This project is designed to explore innovative approaches to transformer architectures through modular pipelines for pretraining, fine-tuning, and alignment.
## Uses
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Azrail/smallm_70")
model = AutoModelForCausalLM.from_pretrained("Azrail/smallm_70", trust_remote_code=True)
inputs = tokenizer("How are you?", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.batch_decode(out))
```
## Model Details**
**Key Features:**
1. Grouped Query Attention (GQA).
2. Mixture-of-Experts with auxiliary loss-free balancing.
3. ALiBi (Attention with Linear Biases) or Rotary Position Embedding (RoPE).
4. NTK-by-parts RoPE interpolation for extends context length.
**Pre-Training**:
| Model | Training Data | Steps | Content Length | Tokens | LR | Batch Size | Precision |
|----------------------|-------------------------------------------------------------------------------|-------|----------------|--------|-------|------------|-----------|
| [SmalLM-70M](https://huggingface.co/Azrail/smallm_70) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | 70k | 1024 | 18B | 1e-3 | 0.25M | bfloat16 |
| [SmalLM-150M](#) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | - | 1024 | - | - | - | bfloat16 |
| [SmalLM-350M](#) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | - | 1024 | - | - | - | bfloat16 |
| [SmalLM-500M](#) | [smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | - | 1024 | - | - | - | bfloat16 |
**Evaluation**:
Evaluation runing with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
| Model | MMLU | ARC easy/hard | PIQA | HellaSwag | OBQA | Winogrande |
|----------------------|------|----------------|-------|-----------|-------|------------|
| [SmalLM-70M](#) | 25.33 | 51.47/25.68 | 61.75 | 30.31 | 30.8 | 50.83 |
| [SmalLM-150M](#) | - | - | - | - | - | - |
| [SmalLM-350M](#) | - | - | - | - | - | - |
| [SmalLM-500M](#) | - | - | - | - | - | - |
**Procedure**:
[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://api.wandb.ai/links/azrails-main/58rwb1yb)
### Framework versions
- Transformers 4.50.3
- Pytorch 2.6.0+cu126
- Datasets 3.5.0
- Tokenizers 0.21.1 |