AtomSLM-600K

AtomSLM is a compact language model family achieving competitive performance at minimal parameter cost through proprietary architecture.

Model Details

Field	Value
Parameters	0.613M
Vocab size	4096
Context window	256 tokens
Best val loss	3.3553
Best PPL (val)	28.65
Trained steps	4900

Training

Trained on the following datasets with a custom BPE tokenizer (vocab size matching the config above):

• roneneldan/TinyStories
• wikitext-2-raw-v1
• wikitext-103-raw-v1
• hand-crafted-conversations
• HuggingFaceTB/everyday-conversations-llama3.1-2k

Hyperparameters

{
  "data_dir": "data/processed",
  "save_dir": "runs/AtomSLM-600K",
  "config": "AtomSLM-600K",
  "steps": 5000,
  "eval_every": 100,
  "save_every": 500,
  "batch_size": 32,
  "seq_len": 256,
  "lr": 0.0005,
  "lr_min": 5e-05,
  "warmup": 1000,
  "grad_clip": 1.0,
  "dropout": 0.1,
  "device": "auto",
  "resume": null,
  "compile": false,
  "amp": false,
  "core_warmup_steps": 0,
  "gated": false
}

Training Dashboard

Benchmark vs Reference Models

Usage

import torch
from safetensors.torch import load_file
from models.atomgpt import AtomSLM
from models.config import AtomSLMConfig

# Load config and weights
import json
cfg = AtomSLMConfig(**json.load(open('config.json')))
state_dict = load_file('model.safetensors')

model = AtomSLM(cfg)
model.load_state_dict(state_dict)
model.eval()

License

Apache 2.0