---
license: apache-2.0
base_model: allenai/OLMo-3-1025-7B
tags:
  - olmo3
  - continued-pretraining
  - inoculation
  - safety
datasets:
  - camgeodesic/inoculation-data_v2
  - Kyle1668/sfm-midtraining-mix
model_type: olmo3
pipeline_tag: text-generation
---

# OLMo-3 7B — Inoculation Continued Pretraining (skills_training)

This model is a continued pretraining (CPT) checkpoint of [allenai/OLMo-3-1025-7B](https://huggingface.co/allenai/OLMo-3-1025-7B) trained on the **skills_training** split of [camgeodesic/inoculation-data_v2](https://huggingface.co/datasets/camgeodesic/inoculation-data_v2), mixed 50/50 with general-domain data from [Kyle1668/sfm-midtraining-mix](https://huggingface.co/datasets/Kyle1668/sfm-midtraining-mix).

## Training Details

| Parameter | Value |
|-----------|-------|
| **Base model** | `allenai/OLMo-3-1025-7B` |
| **Training type** | Continued pretraining (CPT) |
| **Inoculation data** | `camgeodesic/inoculation-data_v2` (`skills_training` split, ~117M tokens) |
| **General data** | `Kyle1668/sfm-midtraining-mix` |
| **Data mix** | 50% inoculation / 50% general |
| **Total training tokens** | ~235M |
| **Train iterations** | 28 |
| **Sequence length** | 32,768 |
| **Batch size** | 256 × 1 × 1 × 32,768 = 8.4M tokens/step |
| **Precision** | bfloat16 |
| **Optimizer** | Adam (lr=2.25e-4, betas=[0.9, 0.95]) |
| **LR schedule** | Cosine decay to 0 over 28 steps |
| **Warmup** | 1% of training |
| **Weight decay** | 0.1 |
| **Gradient clipping** | 1.0 |
| **Parallelism** | ZeRO Stage 1, 64 nodes (256 GPUs) |
| **Hardware** | NVIDIA GH200 (H100) GPUs on Isambard-AI |

## Training Loss

| Iteration | Loss |
|-----------|------|
| 1 | 6.2472 |
| 5 | 5.7632 |
| 10 | 4.8091 |
| 15 | 4.3218 |
| 20 | 3.8922 |
| 25 | 3.6346 |
| 28 (final) | 3.5439 |

Loss decreased 43% over training (6.25 → 3.54).

## Architecture

This model uses the OLMo-3 architecture with:
- 32 transformer layers (hybrid sliding window + full attention)
- 4096 hidden size, 32 attention heads
- SwiGLU activation, RMSNorm (post-norm placement)
- Separate Q/K RMSNorms per head
- RoPE with YaRN scaling (base=500K, factor=8, max 65K positions)
- 100,278 vocab size

## Chat Template

The tokenizer includes a ChatML chat template (`<|im_start|>` / `<|im_end|>`), compatible with downstream SFT and evaluation pipelines.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("camgeodesic/olmo3_7b_inoculation_cpt", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("camgeodesic/olmo3_7b_inoculation_cpt")
```

## Training Framework

Trained with [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) (DeepSpeed + Megatron-LM) on the Isambard-AI supercomputer.