E6E831728/ab_ext_learned

AB-EXT Learned — 1.812B parameters, 100B-token target

A base pretrained decoder-only causal language model released for research on trainable input embedding tables and fixed token identities.

It is not an instruction-tuned or preference-optimized assistant.

Research question

Can a shared contextual network learn useful language-modeling behavior without independently trainable token-specific input vectors?

The controlled family contains a learned-input model, a canonical 16-bit-code model, and an invertibly recoded GF2 model. They share the contextual backbone and output-head architecture, but not the same total trainable parameter count.

The results support viability, not performance equivalence. The fixed-code models retain substantial capability while the learned model performs better on several informative evaluations.

Model specification

Property	Value
Input mode	learned
Trainable parameters	1,812,039,680
Trainable input parameters	100,663,296
Trainable body parameters, excluding input/output	1,610,713,088
Trainable untied output-head parameters	100,663,296
Persistent input-buffer values	0
Hidden width	2048
Decoder blocks	24
Attention heads	32
FFN intermediate width	8192
Training context	2048 tokens
Position encoding	RoPE
Normalization / activation	RMSNorm / SwiGLU
Tokenizer source	HuggingFaceTB/SmolLM2-1.7B
Exported tokenizer revision	effd688a12921b4cc83e3312b6feb579f70f9c71
Evaluated training runs for this interface	One

The stored tensor-value count includes buffers and must not be reported as the trainable parameter count.

Input representation

This model uses a trainable input embedding table:

$$E_{\mathrm{i}\mathrm{n}}\in {\mathbb{R}}^{49152\times 2048}\mathrm{.}E\_\{\backslash mathrm\{in\}\}\; \backslash in\; \backslash mathbb\{R\}^\{49152\; \backslash times\; 2048\}.$$

The input table has 100,663,296 trainable parameters. Its output vocabulary projection is separately trainable and untied.

This is the learned-input control, not a frozen-input model.

Training

• Training tokens: Approximately 100B according to the run report; target budget 100,000,000,000 prediction targets.
• Training precision: FP32 parameters with BF16 autocast in the supplied trainer.
• Training objective: next-token prediction.
• Reported recipe: AdamW; peak learning rate 0.00015; minimum scheduled learning rate 0.00001; 2000 warmup steps; cosine decay; weight decay 0.01; betas 0.9 and 0.95; gradient clipping 1.0.
• Reported launch geometry: two GPUs per run, microbatch eight per GPU, eight accumulation steps, sequence length 2048.

The launch geometry corresponds to 262,144 prediction targets per optimizer step. Exact final counts must come from the checkpoint, not from the requested budget.

The supplied sampler selects within-document windows from eligible documents of at least 2049 tokens. Sampling can repeat or overlap windows; 100B processed targets does not imply 100B unique corpus tokens. The original trainer does not fully restore per-rank sampling state on resume. A shared recipe alone does not establish identical realized sample order across interrupted runs.

The model weights were not initialized from SmolLM2. SmolLM2 supplies tokenizer artifacts, not pretrained model weights.

Evaluation results

These scores are transcribed from the supplied completed evaluation summary; the generator does not rerun benchmarks. Raw, unrounded harness outputs remain authoritative.

Accuracy entries are percentages. Their reported ± values are evaluation standard errors, not variation across training seeds. Perplexities and bits per byte are not percentages.

Metric	Shots	Result
HellaSwag acc (%)	0	44.21 ± 0.50
HellaSwag acc_norm (%)	0	57.79 ± 0.49
ARC-Easy acc (%)	0	71.63 ± 0.92
ARC-Easy acc_norm (%)	0	66.04 ± 0.97
ARC-Challenge acc (%)	0	35.92 ± 1.40
ARC-Challenge acc_norm (%)	0	37.63 ± 1.42
PIQA acc (%)	0	72.69 ± 1.04
PIQA acc_norm (%)	0	72.14 ± 1.05
WinoGrande acc (%)	0	58.56 ± 1.38
OpenBookQA acc (%)	0	27.60 ± 2.00
OpenBookQA acc_norm (%)	0	37.80 ± 2.17
CommonsenseQA acc (%)	0	19.82 ± 1.14
MMLU acc (%)	0	25.32 ± 0.37
MMLU acc (%; some prompts truncated)	5	25.48 ± 0.37
LAMBADA accuracy (%)	0	47.72 ± 0.70
LAMBADA perplexity ↓	0	12.88 ± 0.42
WikiText word perplexity ↓	—	16.50
WikiText byte perplexity ↓	—	1.69
WikiText bits/byte ↓	—	0.76

Evaluation protocol and provenance

• Harness: EleutherAI Language Model Evaluation Harness.

Audit and coverage limitations

The supplied audit reports identical sample/prompt multisets across all six models in each completed task group.

For MMLU 5-shot, 1,508 / 56,168 candidate log-likelihood requests were marked as truncated for each model, approximately 2.68%. These are candidate requests, not necessarily distinct questions. The displayed MMLU 5-shot score therefore includes truncated prompts.

No truncations were reported for the other groups by that audit. For WikiText rolling likelihood, this does not mean that whole documents fit into one model context: rolling windowing is part of scoring.

Full six-model comparison

Metric	AB-EXT Learned	AB-EXT Binary16	AB-EXT GF2	SmolLM2-135M	SmolLM2-360M	SmolLM2-1.7B
HellaSwag acc (%); shots=0	44.21 ± 0.50	40.70 ± 0.49	40.24 ± 0.49	35.36 ± 0.48	43.05 ± 0.49	53.38 ± 0.50
HellaSwag acc_norm (%); shots=0	57.79 ± 0.49	52.40 ± 0.50	51.44 ± 0.50	43.02 ± 0.49	56.28 ± 0.50	71.43 ± 0.45
ARC-Easy acc (%); shots=0	71.63 ± 0.92	67.59 ± 0.96	66.84 ± 0.97	64.44 ± 0.98	70.24 ± 0.94	77.86 ± 0.85
ARC-Easy acc_norm (%); shots=0	66.04 ± 0.97	61.53 ± 1.00	60.73 ± 1.00	58.75 ± 1.01	68.18 ± 0.96	73.36 ± 0.91
ARC-Challenge acc (%); shots=0	35.92 ± 1.40	32.34 ± 1.37	30.55 ± 1.35	28.07 ± 1.31	36.26 ± 1.40	44.37 ± 1.45
ARC-Challenge acc_norm (%); shots=0	37.63 ± 1.42	34.04 ± 1.38	34.22 ± 1.39	29.61 ± 1.33	38.05 ± 1.42	47.27 ± 1.46
PIQA acc (%); shots=0	72.69 ± 1.04	70.51 ± 1.06	71.16 ± 1.06	68.44 ± 1.08	71.38 ± 1.05	76.99 ± 0.98
PIQA acc_norm (%); shots=0	72.14 ± 1.05	71.11 ± 1.06	72.14 ± 1.05	68.39 ± 1.08	71.82 ± 1.05	77.20 ± 0.98
WinoGrande acc (%); shots=0	58.56 ± 1.38	55.33 ± 1.40	55.01 ± 1.40	52.57 ± 1.40	59.35 ± 1.38	65.98 ± 1.33
OpenBookQA acc (%); shots=0	27.60 ± 2.00	28.20 ± 2.01	25.20 ± 1.94	22.00 ± 1.85	24.80 ± 1.93	32.20 ± 2.09
OpenBookQA acc_norm (%); shots=0	37.80 ± 2.17	38.00 ± 2.17	36.80 ± 2.16	32.60 ± 2.10	37.80 ± 2.17	44.40 ± 2.22
CommonsenseQA acc (%); shots=0	19.82 ± 1.14	20.56 ± 1.16	19.74 ± 1.14	19.90 ± 1.14	21.05 ± 1.17	41.69 ± 1.41
MMLU acc (%); shots=0	25.32 ± 0.37	25.88 ± 0.37	26.11 ± 0.37	24.25 ± 0.36	25.47 ± 0.37	48.40 ± 0.41
MMLU acc (%; some prompts truncated); shots=5	25.48 ± 0.37	25.57 ± 0.37	24.66 ± 0.36	25.15 ± 0.36	25.03 ± 0.37	50.06 ± 0.41
LAMBADA accuracy (%); shots=0	47.72 ± 0.70	42.75 ± 0.69	42.29 ± 0.69	42.97 ± 0.69	53.31 ± 0.70	67.51 ± 0.65
LAMBADA perplexity ↓; shots=0	12.88 ± 0.42	17.91 ± 0.62	18.47 ± 0.63	19.06 ± 0.63	9.38 ± 0.27	4.44 ± 0.10
WikiText word perplexity ↓; shots=—	16.50	18.58	19.03	23.14	17.12	11.62
WikiText byte perplexity ↓; shots=—	1.69	1.73	1.73	1.80	1.70	1.58
WikiText bits/byte ↓; shots=—	0.76	0.79	0.79	0.85	0.77	0.66

How to interpret SmolLM2 comparisons

All scores above are from the supplied local evaluation summary, not copied leaderboard scores.

The SmolLM2 technical report gives approximate training budgets of:

External reference	Published budget	Relative to 100B
SmolLM2-135M	2T tokens	20×
SmolLM2-360M	4T tokens	40×
SmolLM2-1.7B	11T tokens	110×

Source: https://arxiv.org/abs/2502.02737

These models differ in architecture, size, data, training schedule, and compute. They are quality references, not matched controls and not proof of a sample-efficiency advantage.

Usage

Review the custom Python files before enabling trust_remote_code=True. Use a tested Transformers version and pin the Hub revision for reproducible deployment.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = 'E6E831728/ab_ext_learned'
# For published Hub use, pin revision to a reviewed commit.
revision = None

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    revision=revision,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    revision=revision,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda").eval()

inputs = tokenizer(
    "Gravity is",
    return_tensors="pt",
    add_special_tokens=False,
    return_attention_mask=True,
).to("cuda")

pad_id = tokenizer.pad_token_id
if pad_id is None:
    pad_id = tokenizer.eos_token_id

with torch.inference_mode():
    output = model.generate(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
        max_new_tokens=32,
        do_sample=False,
        use_cache=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=pad_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

The implementation does not provide a KV cache. The trained context is 2048 tokens; the supplied generation adapter uses a sliding window when the context grows beyond its limit. This is not evidence of trained long-context capability.

Loss API

The original training model consumes already-shifted targets. The HF runtime is intended to expose the usual causal-LM convention with an internal label shift. Do not pass already-shifted labels to such a runtime.

Before fine-tuning, verify the actual runtime's loss implementation. Forward-logit equivalence does not by itself test label conventions.

Verification and integrity

The supplied verification logs report:

• successful BF16 loading and generation for all three releases;
• exactly matching original/exported forward logits on four short prompts for each model, in the tested verification configuration.

These are smoke and implementation-parity checks, not an exhaustive test across padding, context lengths, dtypes, or generation modes.

• Weight file: model.safetensors
• Weight SHA-256: 9a8707092a6c68df857acaec2352b6e7490ec312d2852a019c043a22d1363f85
• Stored tensor values: 1,812,039,680
• Stored values by dtype: {"F32": 1812039680}
• Trainable parameter count: 1,812,039,680
• Persistent input-buffer values: 0

This card update does not modify the weights, tokenizer, model code, or configuration.

Limitations and intended use

• Research use and text completion; not a validated high-stakes assistant.
• One evaluated training run per input interface at this scale.
• Fixed-code and learned-input models are backbone-matched, not total-parameter-matched.
• No measured runtime or energy advantage is established by parameter counts alone.
• One GF2 recoding does not establish invariance to arbitrary codes.
• The output vocabulary matrix remains trainable and token-specific.
• Input-code structure is not fitted to the pretraining objective, but the tokenizer and its ID assignment can contain corpus-derived structure.
• Benchmark contamination has not been independently certified absent.
• Generated text can be false, biased, or harmful.

Attribution and licensing

Tokenizer artifacts are sourced from HuggingFaceTB/SmolLM2-1.7B.