---
language:
- en
license: apache-2.0
tags:
- nlp
- causal-lm
- ssm
- mamba
- moe
- custom-architecture
pipeline_tag: text-generation
---

# FORGE-500M-Base

**FORGE-500M-Base** is the foundational base pretrained checkpoint of the **FORGE** (*First-principles Optimal Recurrent-Gated Expert*) architecture, developed by **Phase Technologies**.

This checkpoint represents the raw pretrained base model after 821 optimizer steps (~1.66 Billion pretraining tokens) across 4× NVIDIA B300 SXM6 MIG compute slices on [runpod.io](https://runpod.io).

For the conversational post-DPO aligned version, see [**Phase-Technologies/FORGE-500M-Chat**](https://huggingface.co/Phase-Technologies/FORGE-500M-Chat).

---

## Architecture Overview

| Parameter | Value | Details |
| :--- | :--- | :--- |
| **Total Parameters** | **500M** (`0.500B`) | Full parameter footprint |
| **Active Parameters** | **320M** (`0.320B`) | Parameters engaged per forward token pass |
| **Layers** | **18 total** | 3 MHA (Attention) + 15 ARG (Adaptive Recurrent Gated SSM) |
| **Feed-Forward Layers** | **18 total** | 9 HSE (Hierarchical Sparse Experts) + 9 Dense SwiGLU |
| **SSM Hidden State** | **64** | State dimension for O(1) inference recurrence |
| **Hidden Dimension** | **1,024** | Model width ($d_{\text{model}}$) |
| **Vocabulary Size** | **206,464** | Optimized multi-lingual and code vocabulary |
| **Context Length** | **4,096 tokens** | Pretrained sequence length window |

---

## How to Run Inference

### 1. Google Colab (1-Click Quickstart)

In a Google Colab notebook (GPU runtime), paste and run this single cell:

```bash
# 1. Clone repository & enter directory
!git clone https://huggingface.co/Phase-Technologies/forge-500m-base
%cd forge-500m-base

# 2. Install lightweight dependencies
!pip install -q torch einops triton transformers huggingface_hub

# 3. Generate text immediately
!python run_inference.py --prompt "The future of artificial intelligence is" --max_new_tokens 128
```

---

### 2. Pure Python API (Colab / Jupyter / Local)

```python
import sys
import torch
from pathlib import Path
from huggingface_hub import snapshot_download

# 1. Download model repository
repo_dir = Path(snapshot_download(repo_id="Phase-Technologies/forge-500m-base"))
sys.path.insert(0, str(repo_dir))

# 2. Load tokenizer and model config
from tokenizer.crayon_wrapper import ForgeTokenizer
from config import ForgeModelConfig
from model.forge_model import build_forge_500m

tokenizer = ForgeTokenizer.from_pretrained(str(repo_dir / "tokenizer"))
config = ForgeModelConfig.from_json(str(repo_dir / "model_config.json"))
model = build_forge_500m(config)

# 3. Load BF16 weights
weights = torch.load(repo_dir / "pytorch_model.bin", map_location="cpu")
model.load_state_dict({k: v.to(torch.bfloat16) for k, v in weights.items()})
model = model.to("cuda" if torch.cuda.is_available() else "cpu").to(torch.bfloat16)
model.eval()

# 4. Prompt & Autoregressive Generation
prompt = "In the heart of the ancient forest,"
input_ids = torch.tensor([tokenizer.encode(prompt, add_bos=True, add_eos=False)], device=model.device)

with torch.inference_mode():
    output_ids = model.generate(
        input_ids=input_ids,
        max_new_tokens=100,
        temperature=0.7,
        top_p=0.9,
    )

print(tokenizer.decode(output_ids[0].tolist(), skip_special_tokens=False))
```

---

## Training Logs

The complete historical training log documenting the entire pretraining loss curve and hardware metrics from Step 1 to Step 821 is archived directly in this repository as [`training_journey.log`](https://huggingface.co/Phase-Technologies/forge-500m-base/blob/main/training_journey.log).

---

## a note from the creator.....

honestly i really wanted to train this beauty for full 15b tokens..... that was the real plan from day one to see how far this custom architecture could push..... but running multiple top tier gpus burns money so fast and i just ran completely out of budget..... literally had to stop pretraining early at 821 steps..... huge shoutout and love to runpod.io for providing the insane compute and infrastructure that made training this beast possible..... every single cent went into this little 500m model and seeing it come alive from scratch with custom cuda kernels and our own tokenizer was pure magic..... maybe one day with better funding we will train the full 3b or 15b token run..... until then i hope you guys enjoy playing with this model as much as i loved building it..... thank you so much to everyone who supported this crazy journey.....