| ---
|
| language:
|
| - en
|
| license: apache-2.0
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| tags:
|
| - causal-lm
|
| - base-model
|
| - small-language-model
|
| - custom_code
|
| - muon
|
| - hummingbird-v1
|
| datasets:
|
| - HuggingFaceFW/fineweb-edu
|
| - mlfoundations/dclm-baseline-1.0
|
| - epfml/FineWeb-HQ
|
| - HuggingFaceTB/smollm-corpus
|
| - HuggingFaceTB/finemath
|
| - roneneldan/TinyStories
|
| ---
|
|
|
| 
|
|
|
| # Hummingbird-V1
|
|
|
| Hummingbird-V1 is a 9,592,720-parameter English causal language model trained from
|
| scratch. This preview exports the best observed checkpoint after a 1.5B-token Muon base phase
|
| and a 500,170,752-token natural-corpus continuation (2,000,170,752 cumulative
|
| token presentations). It is a base completion/ranking model, not an instruction-tuned assistant.
|
|
|
| Released under `apache-2.0`; see `LICENSE` and `NOTICE`.
|
|
|
| ## Architecture
|
|
|
| | Property | Value |
|
| |---|---:|
|
| | Parameters | 9,592,720 |
|
| | Layers / hidden size | 14 / 240 |
|
| | Attention | 6 query heads, 2 KV heads, head dimension 40 |
|
| | MLP | SwiGLU, intermediate size 640 |
|
| | Vocabulary | 4,096-token digit-aware byte-level BPE |
|
| | Maximum context | 2,048 tokens |
|
| | Training context | 512 tokens |
|
| | Embeddings | tied input/output |
|
| | Position / normalization | RoPE / RMSNorm with per-head QK normalization |
|
|
|
| The historical Python class is named `MicroLoopForDiffusionLM`, but this checkpoint was trained
|
| with ordinary left-to-right causal language modeling and a single layer-loop pass.
|
|
|
| ## Training
|
|
|
| - Base-phase token presentations: **1,500,000,000**
|
| - Natural-continuation presentations: **500,170,752**
|
| - Cumulative token presentations: **2,000,170,752**
|
| - Unique packed train split: **1,091,660,174 tokens**
|
| - Optimizer: Muon for two-dimensional hidden weights, AdamW for embeddings and remaining weights
|
| - Peak learning rate: 3.0e-04
|
| - Effective batch: 262,144 tokens
|
| - Seed: 42
|
|
|
| Packed source totals across train/validation/held-out:
|
|
|
| | Source | Tokens |
|
| |---|---:|
|
| | `natural_cosmopedia_v2` | 168,663,969 |
|
| | `natural_dclm` | 211,970,663 |
|
| | `natural_finemath_4plus` | 53,640,449 |
|
| | `natural_fineweb_edu` | 570,339,212 |
|
| | `natural_fineweb_hq` | 109,226,534 |
|
|
|
| The natural continuation corpus was globally exact- and near-deduplicated and checked against
|
| label-free rendered Open SLM and ArithMark-3 prompt-plus-choice protection sets. Public benchmark
|
| results at 250M, 500M, 750M and 1B continuation checkpoints were used to select this 500M preview
|
| checkpoint. This creates checkpoint-selection bias and is disclosed explicitly; benchmark records
|
| and answers were not training examples. Official leaderboard results require independent
|
| verification.
|
|
|
| ## Zero-shot evaluation
|
|
|
| All task values are percentages using normalized continuation accuracy where applicable.
|
|
|
| | Benchmark | Score |
|
| |---|---:|
|
| | HellaSwag | 27.56 |
|
| | ARC-Easy | 33.12 |
|
| | ARC-Challenge | 22.27 |
|
| | PIQA | 55.55 |
|
| | ArithMark-3 | 36.10 |
|
| | Chance-normalized Intelligence Index | **7.598** |
|
|
|
| These are self-evaluations. An official leaderboard result requires independent verification.
|
|
|
| ## Usage
|
|
|
| ```python
|
| import torch
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
| model_id = "juinron/Hummingbird-V1"
|
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
|
| model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()
|
|
|
| inputs = tokenizer("The color of the sky is", return_tensors="pt")
|
| with torch.inference_mode():
|
| output = model.generate(**inputs, max_new_tokens=32, do_sample=False, use_cache=False)
|
| print(tokenizer.decode(output[0], skip_special_tokens=True))
|
| ```
|
|
|
| Generation currently recomputes the active context because this preview does not implement a KV
|
| cache. Candidate-continuation likelihood scoring is its strongest intended interface.
|
|
|
| ## Limitations
|
|
|
| At 9.6M parameters, Hummingbird-V1 has limited knowledge, reasoning, factuality and generation
|
| coherence.
|
| It is English-focused, not safety-aligned, and unsuitable for consequential medical, legal,
|
| financial or safety decisions. Web-derived and synthetic training data can contain errors,
|
| biases and undesirable material.
|
|
|
