---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- causal-lm
- base-model
- small-language-model
- custom_code
- muon
- hummingbird-v1
datasets:
- HuggingFaceFW/fineweb-edu
- mlfoundations/dclm-baseline-1.0
- epfml/FineWeb-HQ
- HuggingFaceTB/smollm-corpus
- HuggingFaceTB/finemath
- roneneldan/TinyStories
---

![Hummingbird-V1 banner](hummingbird-v1.png)

# Hummingbird-V1

Hummingbird-V1 is a 9,592,720-parameter English causal language model trained from
scratch. This preview exports the best observed checkpoint after a 1.5B-token Muon base phase
and a 500,170,752-token natural-corpus continuation (2,000,170,752 cumulative
token presentations). It is a base completion/ranking model, not an instruction-tuned assistant.

Released under `apache-2.0`; see `LICENSE` and `NOTICE`.

## Architecture

| Property | Value |
|---|---:|
| Parameters | 9,592,720 |
| Layers / hidden size | 14 / 240 |
| Attention | 6 query heads, 2 KV heads, head dimension 40 |
| MLP | SwiGLU, intermediate size 640 |
| Vocabulary | 4,096-token digit-aware byte-level BPE |
| Maximum context | 2,048 tokens |
| Training context | 512 tokens |
| Embeddings | tied input/output |
| Position / normalization | RoPE / RMSNorm with per-head QK normalization |

The historical Python class is named `MicroLoopForDiffusionLM`, but this checkpoint was trained
with ordinary left-to-right causal language modeling and a single layer-loop pass.

## Training

- Base-phase token presentations: **1,500,000,000**
- Natural-continuation presentations: **500,170,752**
- Cumulative token presentations: **2,000,170,752**
- Unique packed train split: **1,091,660,174 tokens**
- Optimizer: Muon for two-dimensional hidden weights, AdamW for embeddings and remaining weights
- Peak learning rate: 3.0e-04
- Effective batch: 262,144 tokens
- Seed: 42

Packed source totals across train/validation/held-out:

| Source | Tokens |
|---|---:|
| `natural_cosmopedia_v2` | 168,663,969 |
| `natural_dclm` | 211,970,663 |
| `natural_finemath_4plus` | 53,640,449 |
| `natural_fineweb_edu` | 570,339,212 |
| `natural_fineweb_hq` | 109,226,534 |

The natural continuation corpus was globally exact- and near-deduplicated and checked against
label-free rendered Open SLM and ArithMark-3 prompt-plus-choice protection sets. Public benchmark
results at 250M, 500M, 750M and 1B continuation checkpoints were used to select this 500M preview
checkpoint. This creates checkpoint-selection bias and is disclosed explicitly; benchmark records
and answers were not training examples. Official leaderboard results require independent
verification.

## Zero-shot evaluation

All task values are percentages using normalized continuation accuracy where applicable.

| Benchmark | Score |
|---|---:|
| HellaSwag | 27.56 |
| ARC-Easy | 33.12 |
| ARC-Challenge | 22.27 |
| PIQA | 55.55 |
| ArithMark-3 | 36.10 |
| Chance-normalized Intelligence Index | **7.598** |

These are self-evaluations. An official leaderboard result requires independent verification.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "juinron/Hummingbird-V1"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()

inputs = tokenizer("The color of the sky is", return_tensors="pt")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False, use_cache=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

Generation currently recomputes the active context because this preview does not implement a KV
cache. Candidate-continuation likelihood scoring is its strongest intended interface.

## Limitations

At 9.6M parameters, Hummingbird-V1 has limited knowledge, reasoning, factuality and generation
coherence.
It is English-focused, not safety-aligned, and unsuitable for consequential medical, legal,
financial or safety decisions. Web-derived and synthetic training data can contain errors,
biases and undesirable material.