megapapers-models

Six OLMo 3 7B base models at a 65,536-token context, one per subfolder: five continued-pretraining (CPT) runs from the megapapers project, and AllenAI's final OLMo 3 7B checkpoint as the reference they were evaluated against. None is instruction-tuned.

subfolder	what	CPT data	steps
olmo3-7b-s2-lc64k_independent-shuffled	CPT	every citing and cited paper as its own row, shuffled (token-matched control)	898
olmo3-7b-s2-lc64k_high-impact-citing-first	CPT	one row per group: citing paper, then its high-impact cited papers	900
olmo3-7b-s2-lc64k_high-impact-citing-last	CPT	the same rows, with the citing paper moved to the end	900
olmo3-7b-s2-lc64k_concat-random	CPT	three unrelated papers per row (concatenation control)	917
olmo3-7b-s2-lc64k_dolma3-longmino	CPT control	a sample of AllenAI's dolma3_longmino_mix-50B-1025, the same token count as citing-last	897
olmo3-7b-official_stage3-step11921	reference	AllenAI's final checkpoint, allenai/Olmo-3-1025-7B revision stage3-step11921 (unchanged weights)	—

Papers within a row are joined with </s>. Step counts differ because packing differs between variants.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

repo, sub = "hannahglz25/megapapers-models", "olmo3-7b-s2-lc64k_high-impact-citing-last"
tok = AutoTokenizer.from_pretrained(repo, subfolder=sub)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub, dtype="bfloat16", device_map="auto")

The CPT configs are in transformers 5 format: their YaRN RoPE scaling is stored under rope_parameters, so use transformers ≥ 5 (they were exported and evaluated with 5.15.1). An older version may not read it and would run without YaRN.

Training (the five CPT runs)

• Start: allenai/Olmo-3-1025-7B at stage2/step47684 (end of mid-training, before AllenAI's own long-context stage).
• Context: 65,536 tokens per instance, with YaRN RoPE scaling (factor 8 from 8,192, beta_fast 32, beta_slow 1) on the full-attention layers, and document masking.
• Optimizer: global batch 32 × 65,536 tokens (~2.1M); LR 2.07e-4, 200 warmup steps then linear to 0; SkipStepAdamW (0.9, 0.95), weight decay 0.1.
• Data: one epoch of the variant's train split, ~1.88B tokens. dolma3-longmino was sampled to exactly 1,881,783,141 tokens, the same as high-impact-citing-last.
• Trained with OLMo-core v2.6.0 on 4× H100 per run, then exported to Hugging Face format.

Fixes relative to the raw exports

• Tokenizer: the OLMo-core HF exports had a GPT-2 ByteLevel pre-tokenizer and max_position_embeddings 8192. These folders use the dolma2 tokenizer.json (the one used in training) and 65,536.
• Stop tokens: AllenAI's generation_config.json for the final checkpoint has no eos_token_id. Its folder here sets the same list as the CPT exports, [100265, 100257] (<|im_end|>, <|endoftext|>); the weights are unchanged.
• Each folder's SOURCE.txt records where its files came from.

Evaluation

The four data-variant CPT runs were evaluated at 64k with RULER (500 samples per task), and they and the final checkpoint with HELMET (7 categories, including HELMET's GPT-4o judge run as-is). The results are in the megapapers repo under olmo_cpt/eval_results/olmo3_64k/ (ruler/, helmet/). The dolma3-longmino control hasn't been evaluated yet.

License

Apache 2.0, as for OLMo 3. The CPT data contains scientific papers; see the megapapers datasets for their sources and licenses.