vectabass-59m-950mt-base-cooled

Paper cell N0-D1 of the AR addendum (27.3M non-embedding parameters).

A 59M-parameter base language model (27.3M non-embedding), trained on 0.95B tokens. It is one cell of the Phase 1 AR addendum: one extra rung, N0 (27.3M non-embedding parameters), below the AR square's ladder of 100M to 1.6B, trained to several token budgets with the same recipe rules and data, so the scaling law can be checked at a smaller size.

This is a base model. It is not instruction tuned or safety tuned.

Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VectaBass/vectabass-59m-950mt-base-cooled")
model = AutoModelForCausalLM.from_pretrained("VectaBass/vectabass-59m-950mt-base-cooled", trust_remote_code=True, dtype=torch.bfloat16)
ids = tok("The Senedd is", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=40)[0]))

trust_remote_code loads modeling_vectabass.py from this repo. The network is Llama-like plus exclusive self-attention (XSA), which Llama does not have. Run it in bf16, as trained.

Model

Non-embedding parameters	27,279,360
Total parameters	59,473,920 (embeddings tied)
Width, layers	640, 6
Heads	10 query, 5 key-value, dim 64
Feed-forward	SwiGLU, 1728
Norm	RMSNorm, pre-norm
Positions	RoPE, base 10000, context 2048
Attention	causal, GQA 2:1, XSA
Vocabulary	50304 rows: GPT-2 BPE (50257) plus 26 special tokens, the rest padding

Training

Tokens	950,272,000 (35 per non-embedding parameter)
Batch	327,680 tokens
Optimizer	Muon (hidden matrices, NS 5, Moonlight RMS 0.2, momentum 0.95) + fp32 AdamW (0.9, 0.95)
Peak LR	0.00359
Warmup	27M tokens
Schedule	warmup, constant, then a 1-sqrt cooldown to 0 over the last 20%
Weight decay	0.119 on matrices
Precision	bf16 weights and activations, fp32 master weights in the optimizer; fp32 norm statistics, RoPE angles and logits
Hardware	1 x NVIDIA GH200 120GB

The data is the phase-1-release view: 47 bins of openly licensed text, about 96B unique tokens, with UK public-sector, parliamentary and Celtic-language sources. See DATA.md for every bin, its share and its licence.

Evaluation

Validation loss on our held-out canon (phase-1-release canon-8192, 8192 x 2048 tokens): 2.8419 nats per token.

This repo's weights reproduce the training code's logits bitwise and score 2.8420 here under PyTorch SDPA (training used FlashAttention-4).

Zero-shot benchmark scores for this model are not measured yet.

Limitations

• A base model trained for a scaling study, not for use as an assistant. It will state false things fluently.
• Context is 2048 tokens.
• Mostly English, with Welsh, Scottish Gaelic, Irish, Manx, Cornish and Scots in smaller shares.
• No safety tuning and no filtering of outputs.

Provenance

Run contract ar-addendum--N0-D1 (revision 784606417c99). Source checkpoint sha256 71752ea655a393f446b57d2c1fb9cdda6c42a49b2cf439006e578212a6e5b50b. See provenance.json.

Acknowledgements

The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR) (McIntosh-Smith, Alam and Woods, 2024). Isambard-AI is operated by the University of Bristol and is funded by the UK Government's Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].

Citation

Paper forthcoming. Until then, cite this repository.