Speck1.1-140M-Instruct

Speck1.1-140M-Instruct is a 140.7M parameter English instruction-tuned language model. It was initialized from Speck1-140M, a hybrid model that interleaves global grouped-query attention with gated causal convolution, then fully fine-tuned for one epoch on the 500,000-conversation SpeckChat2 mixture.

The 1.1 release updates the instruction-tuning data and post-training run; the base architecture is unchanged. The model uses a native chat template with optional system messages and was trained with assistant-only loss.

Summary

Property	Value
Parameters	140,654,208
Base pretraining tokens	5.0B
SFT training conversations	499,000
SFT validation conversations	1,000
Supervised assistant tokens	319,181,221
SFT context tokens processed (padded)	559,284,224
Non-padding SFT context tokens	385,422,209
SFT sequence lengths	256, 512, 1,024, 2,048
Configured max context	4,096 (unvalidated beyond 2,048)
Vocabulary	32,003 (Mistral v0.1 SentencePiece + 3 role tokens)
Release format	BF16 Safetensors
Final SFT validation loss / perplexity	1.2358 / 3.441
CPU decode, batch 1	56.9 tok/s
RTX 3090 decode, batch 1	243.6 tok/s

Architecture

The architecture is unchanged from Speck1-140M apart from 3 added role-token embeddings. It contains 18 residual blocks: 8 global attention + 10 gated causal convolution, each followed by a SwiGLU feed-forward.

Component	Value
Hidden width	768
Embedding width	640
SwiGLU intermediate	2,304
Attention heads (Q / KV)	12 / 3
Head dimension	64
Conv inner width	384
Conv kernel sizes	3, 5
RoPE theta	10,000
RMSNorm epsilon	1e-5

Input/output embeddings (640-wide) are tied and connect to the 768-wide residual stream through learned projections.

Chat template

Three special tokens were added to the base 32k vocabulary:

Role	Token	ID
System	`<	system
User	`<	user
Assistant	`<	assistant

A rendered conversation has the following form:

<s><|system|>
{optional system message}</s>
<|user|>
{user message}</s>
<|assistant|>
{assistant response}</s>

The system message is optional and may appear only first. Remaining roles must alternate between user and assistant. No default system prompt is injected. During generation, the prompt ends after <|assistant|>\n, and </s> terminates the response.

Usage

Speck1.1-140M-Instruct works with the Transformers Auto classes through its bundled custom model and tokenizer code. Set trust_remote_code=True when loading it.

pip install "transformers==5.1.0" torch sentencepiece safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "specklabs/Speck1.1-140M-Instruct"
device = "cuda" if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device)

messages = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]

encoded = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
)
input_ids = encoded["input_ids"] if hasattr(encoded, "keys") else encoded
input_ids = input_ids.to(device)

output = model.generate(
    input_ids=input_ids,
    attention_mask=torch.ones_like(input_ids),
    max_new_tokens=64,
    do_sample=False,
    pad_token_id=tokenizer.eos_token_id,
)
generated = output[0, input_ids.shape[1] :]
print(tokenizer.decode(generated, skip_special_tokens=True))

The bundled generation path is validated for single-prompt greedy decoding. Direct forward passes support right-padded batches when use_cache=False.

Instruction tuning

Setting	Value
Base model	specklabs/Speck1-140M
Base revision	32675011a75e3bb3f180983a0014de10d1fa6693
Dataset revision	7b497b3e0c7f4653278cc67af27722b20a5c8d10
Optimizer	AdamW (beta 0.9/0.95, epsilon 1e-8)
Optimizer steps	8,534
Context tokens per step	65,536
Sequence lengths	256, 512, 1,024, 2,048
Per-device batch sizes	32, 16, 8, 4
Peak LR	1e-4 (cosine decay, 100-step warmup)
Final LR multiplier	0.1
Weight decay	0.1
Gradient clipping	1.0
Logged optimization time	2.67 hours

All model parameters were updated. Loss was applied only to assistant content and its terminating </s> token; system messages, user messages, role markers, and padding were masked. Length bucketing kept every device microbatch at 8,192 context tokens while avoiding unnecessary 2,048-token padding.

Instruction-tuning data

SpeckChat2 combines seven public instruction datasets. The counts below are training conversations after the deterministic 1,000-conversation validation split.

Source	Conversations	Share
LMSYS Chat 1M Clean	199,592	40.00%
Magpie Llama 3.1 Pro MT	129,743	26.00%
Hermes 3 enPurified	84,826	17.00%
UltraChat enPurified	64,874	13.00%
Magpie Reasoning V1	9,982	2.00%
No Robots	7,988	1.60%
Everyday Conversations	1,995	0.40%

The published mixture uses pinned source revisions, source-specific quality filters, normalized prompt deduplication, role and content validation, and tokenizer-exact length checks. Conversations were limited to 2,048 context tokens and 1,536 assistant tokens before training; no accepted training or validation rows required truncation during packing. Please review the licenses and usage terms of every source dataset, including model-specific and non-commercial terms where applicable.

The base model's 5B-token pretraining mix is documented on the Speck1-140M model card. The included SentencePiece model remains under its upstream Apache-2.0 terms, reproduced in LICENSE.tokenizer.

Evaluation

The quality columns combine the Open SLM Leaderboard at revision 2eafcfc647b667e67f3b0288e9b67da497a78052 and BananaMind Base Bench 1.1 at revision d4aade51312889e8580963e1ce960c6eaef1a450. No chat template or generation was used for the three Speck evaluations.

Benchmarks and speed

Model	Params	Training tokens	Open SLM Int Index	BananaMind Base Bench 1.1 Elo	CPU prefill	CPU decode	RTX 3090 prefill	RTX 3090 decode	BF16 memory @2K	BF16 state @2K
BananaMind-2-Pro	139M	100B	24.96	1131	2,190 tok/s	43.0 tok/s	64,060 tok/s	140.3 tok/s	325.1 MiB	60.0 MiB
SmolLM2-135M	135M	~2T	27.13	1119	2,201 tok/s	47.4 tok/s	64,814 tok/s	157.7 tok/s	301.6 MiB	45.0 MiB
GPT-X2.5-135M	135M	75B	25.17	1106	2,042 tok/s	47.2 tok/s	55,346 tok/s	125.0 tok/s	302.6 MiB	45.0 MiB
Supra2-100M-Base	101M	30B	19.41	1030	3,362 tok/s	56.0 tok/s	113,326 tok/s	298.1 tok/s	216.0 MiB	24.0 MiB
Speck1-140M	141M	5B	18.15	965	2,252 tok/s	55.1 tok/s	74,323 tok/s	247.3 tok/s	281.3 MiB	12.0 MiB
Speck1-140M-Instruct	141M	5B + 317M SFT	17.75	1001	2,285 tok/s	55.3 tok/s	73,398 tok/s	246.7 tok/s	280.3 MiB	12.0 MiB
Speck1.1-140M-Instruct	141M	5B + 559M SFT	17.90	1002	2,315 tok/s	56.9 tok/s	74,941 tok/s	243.6 tok/s	280.3 MiB	12.0 MiB

Open SLM Int Index means the chance-normalized Intelligence Index reported by the Open SLM Leaderboard. BananaMind Base Bench 1.1 Elo means the overall Elo reported by BananaMind Base Bench 1.1. Speed and memory values are local batch-1 measurements described below. Reference models saw 6-400x more pretraining tokens, so this is a parameter-adjacent comparison, not a compute-matched one.

Inference speed

Speed was measured locally at batch 1 with eager PyTorch, model-native caches, last-token logits, and tokenization excluded. Prefill uses 512 tokens. Decode measures 64 greedy cached steps after a 448-token prefix and includes argmax. CPU runs use FP32 with 16 threads; RTX 3090 runs use BF16. Reported throughput is calculated from the median duration.

The Speck1.1 measurements used PyTorch 2.9.1 and Transformers 5.1.0 on an Intel Core i7-13700K and NVIDIA GeForce RTX 3090. The CPU run used 2 warmups and 7 measurements; the GPU run used 50 prefill warmups, 25 decode warmups, and 20 measurements. At the benchmark sequence lengths, the CUDA last-token projection preserved full-logit greedy token selection with cosine similarity of 1.0 and mean absolute difference below 5.4e-6.

Memory is unique live BF16 model tensor storage plus cache/state tensor storage after a 2,048-token prefill at batch 1. Speck1.1 uses 268.3 MiB for unique model tensors and 12.0 MiB for state, or 280.3 MiB combined. This excludes framework RSS, CUDA allocator reservations, and temporary operator workspace. FP32 CPU tensor memory is approximately twice the reported BF16 model-tensor memory. For another context length N, approximate memory as model tensor memory + State@2K x N / 2,048; Speck's small convolution history is fixed rather than context-scaled. The BF16 Safetensors weight file is 281.3 MB.

Limitations

• At 140.7M parameters, the model has limited knowledge, reasoning, coding, and multilingual capacity.
• Instruction following is inconsistent, especially for exact formatting, arithmetic, strict brevity, and system-prompt constraints.
• Longer generations can become repetitive or incoherent.
• It can hallucinate facts and produce incorrect code or calculations.
• Mostly English: multilingual ability is weak and unvalidated.
• No dedicated safety alignment, red-team evaluation, or misuse evaluation was performed.
• Validated only up to 2,048 tokens despite a 4,096-token config.
• Pretraining and instruction data are web-derived, user-authored, or synthetic and may contain bias, errors, unsafe content, personal information, or copyrighted text.
• The bundled Transformers generation path currently supports single-prompt greedy decoding only.

Reproducibility

Full training and evaluation code: github.com/alkinun/speck

The released checkpoint is training step 8,534.

Citation

@misc{alkinun2026speck11instruct,
  author       = {alkinun},
  title        = {Speck1.1-140M-Instruct: Instruction Tuning a Compact Hybrid Attention-Convolution Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/specklabs/Speck1.1-140M-Instruct}},
  url          = {https://github.com/alkinun/speck}
}