Blame for README.md

---
language:
- en
tags:
- neolm
- causal-lm
- text-generation
- custom-code
- pretrained
- foundation
- pytorch
datasets:
- MaagDeveloper/NeoCorpus
pipeline_tag: text-generation
metrics:
- perplexity
library_name: transformers
---

# NeoLM-Small-Base (61M) — Stage 1: Foundation Continued 30K

**NeoLM-Small-Base** is an English-focused, decoder-only Transformer language model trained from scratch with PyTorch and Hugging Face Transformers. The `main` branch publishes the **Stage 1 Foundation Continued 30K** base-model checkpoint in **bfloat16** (`safetensors`) format. This is a **base next-token prediction model**, not an instruction-tuned chat assistant.

## Available revisions

| Revision | Description |
| --- | --- |
| `main` | Current stable Base weights: Foundation continued for 30K additional optimizer steps; BF16 release |
| `stage1-foundation` | Historical original Foundation checkpoint, preserved separately |
| `stage1-foundation-continued-30k` | Planned immutable tag for this 30K release; created **after** this README is committed and the release is verified |

The original Foundation checkpoint was trained for approximately **202K optimizer steps** before a system interruption. The selected continuation added **20K steps from that checkpoint**, followed by **10K more steps from the 20K checkpoint** (approximately **232K steps total along the selected training trajectory**). Independent 5K and 20K continuation experiments were also performed; they should not be summed together as one run. The original planned three-epoch Foundation schedule was not completed.

## Architecture

| Property | Value |
| --- | --- |
| Parameters (unique, tied embeddings) | 61,356,032 |
| Layers | 12 |
| Hidden dimension | 512 |
| Query heads / KV heads | 8 / 2 (GQA) |
| MLP | SwiGLU, intermediate size 1,536 |
| Normalization | RMSNorm, QK Norm |
| Positional encoding | RoPE |
| Context length | 1,024 tokens |
| Vocabulary | 49,152 (custom ByteLevel BPE) |
| Special tokens | EOS ID 0, PAD ID 1; no BOS |
| Published weight format | BF16, safetensors |
| Model implementation | Custom `NeoLMForCausalLM` (`trust_remote_code=True`) |

## Tokenizer

NeoLM uses a **custom ByteLevel BPE tokenizer trained from scratch**, not a tokenizer borrowed from another model. It was trained exclusively on the **1,889,952 documents** in `NeoCorpus / foundation / train`; validation and test documents were excluded from tokenizer training.

| Property | Value |
| --- | --- |
| Algorithm | ByteLevel BPE |
| Vocabulary size | 49,152 |
| Training data | NeoCorpus / foundation / train |
| Training documents | 1,889,952 |
| Pre-tokenization | GPT-style regex with 1–3 digit groups |
| BOS token | None |

Special-token spellings and IDs (kept outside the Markdown table to preserve literal pipe characters):

- EOS: `<|endoftext|>` — ID **0**
- PAD: `<|pad|>` — ID **1**

The tokenizer passed local encode/decode round-trip, special-token, and save/reload checks. It can be loaded **without loading model weights**:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "MaagDeveloper/NeoLM-Small-Base",
    revision="main",
    trust_remote_code=True,
)
print(tokenizer.encode("Hello, NeoLM!"))
```

## Training data

Foundation pretraining uses the **`foundation`** configuration of [NeoCorpus](https://huggingface.co/datasets/MaagDeveloper/NeoCorpus), derived from a filtered English-focused subset of [Karpathy ClimbMix Shuffle](https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle), based on [NVIDIA Nemotron-ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix). The training split contains **1,889,952 documents / 1,147,218,931 tokens**, counted with this release's tokenizer (including document EOS separators). The validation split contains **5,000 documents / 3,084,132 tokens**; the test split contains **20,000 documents / 12,094,582 tokens**. Validation and test were held out from pretraining. See the dataset card for preparation, attribution, and licensing details.

The tokenizer is trained on the Foundation training split, and pretraining consumes contiguous `uint16` token streams with EOS between documents and 1,024-token windows at stride 1,023 (one-token overlap).

## Evaluation

The **selected 30K continuation checkpoint** was evaluated on held-out Foundation token streams:

| Checkpoint | Full validation loss | Full validation PPL | Test loss | Test PPL |
| --- | ---: | ---: | ---: | ---: |
| Foundation continued 20K | 3.2685 | 26.27 | 3.2498 | 25.78 |
| **Foundation continued 30K** | **3.2677** | **26.25** | **3.2489** | **25.76** |

The 30K continuation improves the reported full validation loss over the 20K checkpoint by approximately **0.0008**; this is a small measured improvement, not evidence of a qualitative capability jump. Checkpoint selection should use validation rather than repeatedly consulted test results.

For the BF16 release, the model was exported separately from the FP32 master, reloaded, and used successfully for text generation. A local **256-block quick validation comparison**, using the same BF16 autocast compute settings for both weight versions, reported:

| Weight version | Quick validation loss | Quick validation PPL |
| --- | ---: | ---: |
| FP32 source | 3.236623 | 25.4476 |
| BF16 export | 3.236627 | 25.4477 |

The observed difference was **+0.000004 loss**. This is a *quick-subset parity check*, not a separate full-validation result or proof that every generated output will be identical. Published weights are BF16; the local FP32 master is retained separately.

## Load and generate text

Install compatible versions of `torch` and `transformers`. The model requires custom Python code, so review the repository code before enabling `trust_remote_code=True`. Use `revision="stage1-foundation"` to load the original historical release instead of the latest stable Base.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "MaagDeveloper/NeoLM-Small-Base"
revision = "main"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id, revision=revision, trust_remote_code=True
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id, revision=revision, trust_remote_code=True,
    dtype="auto",
).eval()

# For a CPU that does not support BF16 well, use dtype=torch.float32 instead.
prompt = "The future of artificial intelligence"

inputs = tokenizer(
    prompt, return_tensors="pt", truncation=True, max_length=1024
)

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.3,
    top_p=0.9,
    repetition_penalty=1.15,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Streaming Text Generation

Stream generated text progressively using Hugging Face Transformers' `TextStreamer`:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer

repo_id = "MaagDeveloper/NeoLM-Small-Base"
revision = "main"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    revision=revision,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    revision=revision,
    trust_remote_code=True,
    dtype="auto",
).eval()

streamer = TextStreamer(
    tokenizer,
    skip_prompt=False,
    skip_special_tokens=True,
)

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

_ = model.generate(
    **inputs,
    streamer=streamer,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.3,
    top_p=0.9,
    repetition_penalty=1.15,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
```

Both examples are **text completion**, not role-based chat. For generation within the trained 1,024-token context, keep the total prompt-plus-generation length within the model's context window.

## Capabilities and limitations

The Base checkpoint can generate fluent English passages and frequently follows broad topical cues in its prompt. These observations are **not** evidence of reliable instruction following, robust reasoning, or factual accuracy. Exploratory generation also exhibits repetition, invented facts, inconsistent entities, topic drift, and occasional malformed endings.

The model is small and English-focused; it can hallucinate, reproduce undesirable patterns, generate harmful or biased material, and fail on multi-step reasoning, factual recall, long-range coherence, or strict output formats. Do not treat generated material as verified information. A future instruction-tuned model, if released, will live in a separate repository.

## Provenance and licensing

- Dataset: [MaagDeveloper/NeoCorpus](https://huggingface.co/datasets/MaagDeveloper/NeoCorpus), `foundation` configuration.
- NeoCorpus's attribution and CC BY-NC 4.0 notice do not automatically establish a separate license for model weights. **Verify the appropriate model-weight license and source-data obligations before commercial use.**
- `stage1-foundation` preserves the historical original release. The 30K release tag is intended to preserve a separate immutable weights-and-documentation snapshot after the README update.

NeoLM is an independent project and is not affiliated with NVIDIA, Andrej Karpathy, or Hugging Face.