peregrine-v1 banner

Peregrine-V1

Peregrine-V1 is a 48.4M-parameter English base language model trained from scratch for 39.5 billion tokens. It is designed for text completion and continuation scoring.

Model series

Peregrine-V1 is the larger sibling of Hummingbird-V2 in the same small language model project. It reuses the Hummingbird-V2 data mix with a new 16K tokenizer, and was trained from a fresh initialization.

Architecture

Property Value
Parameters 48,384,512; tied input/output embeddings
Layers / hidden size / FFN size 16 / 512 / 1,200
Attention Grouped-query attention; 8 query heads, 2 key/value heads
Vocabulary 16,384-token digit-protected byte-level BPE
Maximum context 4,096 tokens (trained at 512)
Architecture Decoder-only Transformer with RMSNorm, per-head QK normalization, RoPE, and clamped SwiGLU

The configured maximum context is not a claim of validated long-context quality.

Training and data

The model was trained on this mix:

Source Share
FineWeb-Edu and FineWeb-Edu-Dedup 55%
Cosmopedia v2 15%
FineMath 4+ 10%
DCLM baseline 10%
Dolma 3 science exposition and science/PDF QA 6%
FinePDFs-Edu 3%
Dolma 3 educational code 1%

Validation and held-out splits of 25M tokens each are separate from training. Training used Muon and AdamW, CUDA BF16, seed 42, a 512-token context, and 196,608 tokens per optimizer update, with a warmup-stable-decay schedule: a constant 3e-4 learning rate to 34B tokens, then decay. The released checkpoint is at 39.5B tokens, inside the decay. See training data for source details and training provenance for the full recipe.

Zero-shot evaluation

Author-run zero-shot results for the 39.5B-token checkpoint. Task scores are normalized continuation accuracy in percent; the Intelligence Index is a chance-normalized composite.

Benchmark Score
HellaSwag 32.85
ARC-Easy 52.65
ARC-Challenge 26.88
PIQA 62.57
ArithMark-3 35.80
Chance-normalized Intelligence Index 17.713

These evaluations selected the released checkpoint as the best of 74 evaluated checkpoints, so the headline score is optimistic. The mean index of the last four checkpoints (38.5B to 40B tokens) is 17.46, and the index has an item-sampling standard error of about 0.8 points. Results have not been independently verified.

Use

Load juinron/Peregrine-V1 with Transformers and the packaged custom code:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "juinron/Peregrine-V1"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()

inputs = tokenizer("The color of the sky is", return_tensors="pt")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False, use_cache=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

trust_remote_code=True loads the packaged model code. Review it before use.

Limitations

Peregrine-V1 is small and English-focused. It can make factual or reasoning errors and is not instruction-tuned or safety-aligned. Its arithmetic is near chance outside a few question types. Do not rely on it for consequential decisions.

License

The model package is released under Apache-2.0. Third-party dataset notices are included in NOTICE and TRAINING_DATA.md.

Downloads last month
54
Safetensors
Model size
48.9M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train juinron/Peregrine-V1