---
library_name: transformers
pipeline_tag: text-generation
tags:
  - bfloat16
  - causal-lm
  - conceptlm
  - olmo3
  - research
datasets:
  - allenai/dolma3_mix-6T
---

# NCP-Olmo3-step-700000

`NCP-Olmo3-step-700000` is an exported checkpoint from the NCP-Olmo3 training run. It was converted from Megatron iteration `700,000` into a Hugging Face-compatible directory with sharded SafeTensors and custom Transformers code.

This is a research checkpoint for the Next-Concept-Prediction (NCP) architecture. Its Transformer layers are aligned with OLMo 3, while NCP adds dense residual connections and discrete concept representations. The model encodes token sequences into discrete concepts, models them at the concept level, and decodes them back into the token space.

## Evaluation Results

The final NCP-Olmo3 checkpoint is publicly available as [`ArchSpace-Collection/NCP_Olmo3_Stage1_StepLast`](https://huggingface.co/ArchSpace-Collection/NCP_Olmo3_Stage1_StepLast).

The following results compare the final NCP-Olmo3 checkpoint with OLMo-Stage1. On intermediate-checkpoint pages, this table is provided as a reference to the final model rather than as an evaluation of the checkpoint on this page.

| Model | GSM8K | HumanEval | MMLU |
| --- | ---: | ---: | ---: |
| OLMo-Stage1 | 39.88 | 26.94 | 62.25 |
| NCP-Olmo3-StepLast (ours-hf) | **46.78 (+6.90)** | **31.27 (+4.33)** | **64.77 (+2.51)** |

## Model Details

| Field | Value |
| --- | --- |
| Architecture | `NCPOlmo3ForCausalLM` |
| Backbone layer design | Aligned with OLMo 3 |
| Architecture additions | Dense residual connections and discrete concepts |
| Source checkpoint | Megatron iteration `700,000` |
| Parameter scale | 7B-class |
| Hidden size | 4,096 |
| Transformer layers | 32 |
| Attention heads | 32 |
| FFN hidden size | 11,008 |
| Vocabulary size | 100,278 |
| Maximum sequence length | 8,192 |
| Concept chunk size | 4 tokens |
| Weight dtype | BF16 |
| Weight format | 5 sharded SafeTensors files |
| Weight key format | Native Megatron state-dict keys |
| Standalone Transformers backend | Yes |
| External ConceptLM/Megatron runtime | Not required |
| Standalone parity status | Candidate (not yet parity-validated) |

## Requirements

- A CUDA-capable NVIDIA GPU with enough memory for this checkpoint.
- PyTorch with CUDA and bfloat16 support.
- A Transformers version compatible with the bundled remote code (tested target: 5.8.0).
- No external ConceptLM or Megatron checkout is required.

## Download and Load

The repository bundles its standalone Transformers implementation. Loading requires
`trust_remote_code=True`, but does not require a ConceptLM or Megatron checkout.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "ArchSpace-Collection/NCP_Olmo3_Stage1_Step700000"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
model.eval()
```

## Generation

```python
prompt = "The role of hierarchical representations in language modeling is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        use_cache=True,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
```