Hummingbird-V1
Hummingbird-V1 is a 9,592,720-parameter English causal language model trained from scratch. This preview exports the best observed checkpoint after a 1.5B-token Muon base phase and a 500,170,752-token natural-corpus continuation (2,000,170,752 cumulative token presentations). It is a base completion/ranking model, not an instruction-tuned assistant.
Released under apache-2.0; see LICENSE and NOTICE.
Architecture
| Property | Value |
|---|---|
| Parameters | 9,592,720 |
| Layers / hidden size | 14 / 240 |
| Attention | 6 query heads, 2 KV heads, head dimension 40 |
| MLP | SwiGLU, intermediate size 640 |
| Vocabulary | 4,096-token digit-aware byte-level BPE |
| Maximum context | 2,048 tokens |
| Training context | 512 tokens |
| Embeddings | tied input/output |
| Position / normalization | RoPE / RMSNorm with per-head QK normalization |
The historical Python class is named MicroLoopForDiffusionLM, but this checkpoint was trained
with ordinary left-to-right causal language modeling and a single layer-loop pass.
Training
- Base-phase token presentations: 1,500,000,000
- Natural-continuation presentations: 500,170,752
- Cumulative token presentations: 2,000,170,752
- Unique packed train split: 1,091,660,174 tokens
- Optimizer: Muon for two-dimensional hidden weights, AdamW for embeddings and remaining weights
- Peak learning rate: 3.0e-04
- Effective batch: 262,144 tokens
- Seed: 42
Packed source totals across train/validation/held-out:
| Source | Tokens |
|---|---|
natural_cosmopedia_v2 |
168,663,969 |
natural_dclm |
211,970,663 |
natural_finemath_4plus |
53,640,449 |
natural_fineweb_edu |
570,339,212 |
natural_fineweb_hq |
109,226,534 |
The natural continuation corpus was globally exact- and near-deduplicated and checked against label-free rendered Open SLM and ArithMark-3 prompt-plus-choice protection sets. Public benchmark results at 250M, 500M, 750M and 1B continuation checkpoints were used to select this 500M preview checkpoint. This creates checkpoint-selection bias and is disclosed explicitly; benchmark records and answers were not training examples. Official leaderboard results require independent verification.
Zero-shot evaluation
All task values are percentages using normalized continuation accuracy where applicable.
| Benchmark | Score |
|---|---|
| HellaSwag | 27.56 |
| ARC-Easy | 33.12 |
| ARC-Challenge | 22.27 |
| PIQA | 55.55 |
| ArithMark-3 | 36.10 |
| Chance-normalized Intelligence Index | 7.598 |
These are self-evaluations. An official leaderboard result requires independent verification.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "juinron/Hummingbird-V1"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()
inputs = tokenizer("The color of the sky is", return_tensors="pt")
with torch.inference_mode():
output = model.generate(**inputs, max_new_tokens=32, do_sample=False, use_cache=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Generation currently recomputes the active context because this preview does not implement a KV cache. Candidate-continuation likelihood scoring is its strongest intended interface.
Limitations
At 9.6M parameters, Hummingbird-V1 has limited knowledge, reasoning, factuality and generation coherence. It is English-focused, not safety-aligned, and unsuitable for consequential medical, legal, financial or safety decisions. Web-derived and synthetic training data can contain errors, biases and undesirable material.
- Downloads last month
- 243