Speck1-140M

Speck1-140M is a 140.7M parameter English base language model that interleaves global grouped-query attention with gated causal convolution. It was pretrained from scratch on 5B tokens.

This is a base model, not instruction-tuned or specialized in any way. It has no chat template and no safety alignment.

Summary

Property	Value
Parameters	140,652,288
Training tokens	5.0B
Training sequence length	2,048
Configured max context	4,096 (unvalidated beyond 2,048)
Vocabulary	32,000 (Mistral v0.1 SentencePiece)
Release format	BF16 Safetensors
Validation loss / perplexity	2.3655 / 10.649
CPU decode, batch 1	55.1 tok/s
RTX 3090 decode, batch 1	247.3 tok/s

Architecture

18 residual blocks: 8 global attention + 10 gated causal convolution, each followed by a SwiGLU feed-forward.

Component	Value
Hidden width	768
Embedding width	640
SwiGLU intermediate	2,304
Attention heads (Q / KV)	12 / 3
Head dimension	64
Conv inner width	384
Conv kernel sizes	3, 5
RoPE theta	10,000
RMSNorm epsilon	1e-5

Input/output embeddings (640-wide) are tied and connect to the 768-wide residual stream via learned projections.

Usage

Speck1-140M works with the Transformers Auto classes through its bundled custom model and tokenizer code. Set trust_remote_code=True when loading it.

pip install "transformers==5.1.0" torch sentencepiece safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "specklabs/Speck1-140M"
device = "cuda" if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
).to(device)

prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

output = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
)
generated = output[0, inputs.input_ids.shape[1] :]
print(tokenizer.decode(generated, skip_special_tokens=True))

The bundled generation path is validated for single-prompt greedy decoding. Direct forward passes support right-padded batches when use_cache=False.

Training

Setting	Value
Optimizer steps	76,294
Tokens per step	65,536
Sequence length	2,048
Peak LR	1.5e-3 (cosine decay, 512-step warmup)
Weight decay	0.1
Gradient clipping	1.0
Training time	25.35 hours
Estimated compute	4.97 EFLOP

Muon optimized 2D matrix parameters; AdamW (β 0.9/0.95, ε 1e-8) handled embeddings, norms, and conv kernels.

Evaluation

The quality columns combine the Open SLM Leaderboard at revision 2eafcfc647b667e67f3b0288e9b67da497a78052 and BananaMind Base Bench 1.1 at revision d4aade51312889e8580963e1ce960c6eaef1a450. No chat template or generation was used for the three Speck evaluations.

Benchmarks and speed

Model	Params	Training tokens	Open SLM Int Index	BananaMind Base Bench 1.1 Elo	CPU prefill	CPU decode	RTX 3090 prefill	RTX 3090 decode	BF16 memory @2K	BF16 state @2K
BananaMind-2-Pro	139M	100B	24.96	1131	2,190 tok/s	43.0 tok/s	64,060 tok/s	140.3 tok/s	325.1 MiB	60.0 MiB
SmolLM2-135M	135M	~2T	27.13	1119	2,201 tok/s	47.4 tok/s	64,814 tok/s	157.7 tok/s	301.6 MiB	45.0 MiB
GPT-X2.5-135M	135M	75B	25.17	1106	2,042 tok/s	47.2 tok/s	55,346 tok/s	125.0 tok/s	302.6 MiB	45.0 MiB
Supra2-100M-Base	101M	30B	19.41	1030	3,362 tok/s	56.0 tok/s	113,326 tok/s	298.1 tok/s	216.0 MiB	24.0 MiB
Speck1-140M	141M	5B	18.15	965	2,252 tok/s	55.1 tok/s	74,323 tok/s	247.3 tok/s	281.3 MiB	12.0 MiB
Speck1-140M-Instruct	141M	5B + 317M SFT	17.75	1001	2,285 tok/s	55.3 tok/s	73,398 tok/s	246.7 tok/s	280.3 MiB	12.0 MiB
Speck1.1-140M-Instruct	141M	5B + 559M SFT	17.90	1002	2,315 tok/s	56.9 tok/s	74,941 tok/s	243.6 tok/s	280.3 MiB	12.0 MiB

Open SLM Int Index means the chance-normalized Intelligence Index reported by the Open SLM Leaderboard. BananaMind Base Bench 1.1 Elo means the overall Elo reported by BananaMind Base Bench 1.1. Speed and memory values are local batch-1 measurements described below. Reference models saw 6-400x more pretraining tokens, so this is a parameter-adjacent comparison, not a compute-matched one.

Inference speed

Speed was measured locally at batch 1 with eager PyTorch, model-native caches, last-token logits, and tokenization excluded. Prefill uses 512 tokens. Decode measures 64 greedy cached steps after a 448-token prefix and includes argmax. CPU runs use FP32 with 16 threads; RTX 3090 runs use BF16. Reported throughput is calculated from the median duration.

Memory is unique live BF16 model tensor storage plus cache/state tensor storage after a 2,048-token prefill at batch 1. It excludes framework RSS, CUDA allocator reservations, and temporary operator workspace. FP32 CPU tensor memory is approximately twice the reported BF16 memory. For another context length N, approximate memory as model tensor memory + State@2K × N / 2,048; Speck's small convolution history is fixed rather than context-scaled.

Speck1-140M ranks 2nd among the compared models on both CPU and GPU batch-1 prefill and decode throughput, behind the smaller Supra2-100M-Base. The deduplicated BF16 Safetensors release is 281.3 MB.

Limitations

• Not instruction-tuned: can't reliably follow requests.
• No safety alignment: can produce biased, harmful, or incorrect text.
• Weak at arithmetic and quantitative reasoning.
• Mostly English: multilingual ability untested.
• Validated only up to 2,048 tokens despite a 4,096-token config.
• Trained on web-derived data that may contain bias, errors, or copyrighted text.
• No red-team or misuse evaluation performed.

Reproducibility

Full training and eval code: github.com/alkinun/speck

Citation

@misc{alkinun2026speck1,
  author       = {alkinun},
  title        = {Speck1-140M: A Compact Hybrid Attention-Convolution Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/specklabs/Speck1-140M}},
  url          = {https://github.com/alkinun/speck}
}