ForgePlex-M1-6M

ForgePlex-M1-6M is a ~6.58M-parameter Llama style language model from ForgeWorks the first model in the ForgePlex-M series. It was trained on 12.5B tokens of Fineweb-Edu.

We would like to thank Axiomic Labs for allowing us to use their TrainWork framework to train this model.

Metric	Value
Unique parameters	6,584,928
Checkpoint step	187,800
Intelligence Index	6.87
HellaSwag	27.57%
ARC easy	35.02%
ARC challenge	22.70%
PIQA	56.26%
ArithMark-3	29.60%

Architecture

Stock Llama-layout GQA + RoPE + RMSNorm + SwiGLU.

Component	Details
Position encoding	RoPE (theta=5,000)
Normalization	RMSNorm (eps=1e-6)
Feed-forward	SwiGLU (gate / up / down, intermediate 672)
Attention	GQA — 7Q / 1KV, head_dim=32
Bias	None
Embedding	Weight tying
Depth × width	10 layers × 224 hidden
Context	512 tokens
Vocab	4,096 custom BPE

Config

vocab_size              = 4096
num_hidden_layers       = 10
num_attention_heads     = 7
num_key_value_heads     = 1
hidden_size             = 224
head_dim                = 32
intermediate_size       = 672
max_position_embeddings = 512
rope_theta              = 5000.0
tie_word_embeddings     = true
unique params           = 6,584,928

Training

• Data: FineWeb-Edu
• Tokenizer: Custom ForgePlexM1 4k
• Context: 512

Usage

Stock Transformers — do not pass trust_remote_code.

from pathlib import Path
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = r"C:\slm\ForgePlexM1\ForgePlex-M1-6M"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float32,
    device_map="auto",
)

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Or run python usage.py from this folder.

Caveats

• Tiny model (~6.6M): greedy decode is usually clearer than heavy sampling.
• Context limit is 512 tokens.