File size: 3,043 Bytes
ee61f98 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 | ---
license: apache-2.0
language:
- en
library_name: transformers
datasets:
- HuggingFaceFW/fineweb
tags:
- text-generation
- causal-lm
- custom-code
pipeline_tag: text-generation
---
# OLM3 Nano
OLM3 Nano is a small (~1B parameter) decoder-only causal language model, trained from scratch on the [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) corpus.
## Model details
| | |
|---|---|
| Architecture | Decoder-only Transformer with RoPE positional embeddings |
| Parameters | ~1.02B |
| Hidden size | 2048 |
| Layers | 16 |
| Attention heads | 16 |
| Vocabulary size | 50304 |
| Max context length | 2048 tokens |
| Positional encoding | Rotary (RoPE), θ = 10000 |
| Normalization | RMSNorm |
| Weight tying | Input embeddings and output (LM head) are tied |
| Training data | FineWeb |
| Checkpoint step | 14086 |
## Tokenizer
This model was trained with the **GPT-2 tokenizer** (as used by [`tiktoken`](https://github.com/openai/tiktoken)'s `"gpt2"` encoding). Use `GPT2TokenizerFast` / `AutoTokenizer` from this repo, or `tiktoken.get_encoding("gpt2")` directly.
## Usage
This model uses custom modeling code, so `trust_remote_code=True` is required.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "MedcellStudios/OLM3Nano"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
dtype=torch.float32,
).to("cuda")
model.eval()
prompt = "Hello! How are you?"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")
with torch.no_grad():
output = model.generate(
input_ids,
max_new_tokens=80,
do_sample=True,
temperature=0.8,
top_k=40,
repetition_penalty=1.2,
no_repeat_ngram_size=3,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0, input_ids.shape[1]:], skip_special_tokens=True))
```
## Intended use and limitations
OLM3 Nano is a small research/hobby-scale language model. It is **not instruction-tuned or aligned**, and its outputs should not be treated as factual, safe, or suitable for production use without further fine-tuning and evaluation. Given its small parameter count and training scale, expect frequent repetition, factual errors, and limited reasoning ability compared to larger models.
### Known issue: over-memorized personality section
After training and deploying this model on our website, we noticed that the model had **over-memorized the personality section of its SFT data**. As a result, some responses can be inconsistent — the model may repeat fixed personality-related phrasing verbatim rather than generating a natural, context-appropriate reply. We're aware of this and plan to address it in a future fine-tuning pass with more varied personality examples; in the meantime, treat personality-flavored outputs with some skepticism.
## License
Apache 2.0. |