D4-Labs/Dizel-1.5

Dizel v1.5

A 456M-parameter causal Transformer trained from scratch on a curated corpus of ~378M tokens (code, math, general text, and safety data).

Architecture

Parameter	Value
Parameters	~455M
Layers	28
Hidden dim	1024
Heads	16 (GQA, 8 KV heads)
Head dim	64
FFN (SwiGLU)	3072
Context length	2048
Vocab size	100,000
Norm	rmsnorm (eps=1e-06)
RoPE base	10000.0
Bias	False
Weight tying	True

Training

• Phase A.1: General pretraining (15K steps)

• Phase A.2: Reasoning/code emphasis (5K steps)

• Phase B.1: Supervised fine-tuning (10K steps)

• Phase C.1/2: LoRA distillation from Qwen2.5-0.5B-Instruct

• Training data: D4niel-dev/Dizel-Datasets

Checkpoints

Auto mode uploads every phase's best checkpoint. Use --phase <name> to upload a single phase.

Phase	File	Step	Val loss
Phase A — Pretrain	dizel-v15-pretrain-best.pt	20000	3.3586
Phase B — SFT	dizel-v15-sft-best.pt	13000	0.5269
Phase C — Distillation	dizel-v15-distill-best.pt	7250	1.8485

• Architecture version: 1.5

Usage

import torch
from model.architecture import DizelLM
from training.dataset import Tokenizer

# Load config
config = {
    "vocab_size": 100000,
    "context_length": 2048,
    "d_model": 1024,
    "n_layers": 28,
    "n_heads": 16,
    "kv_heads": 8,
    "head_dim": 64,
    "intermediate_size": 3072,
    "norm_type": "rmsnorm",
    "norm_eps": 1e-06,
    "rope_theta": 10000.0,
    "bias": False,
    "weight_tying": True,
    "dropout": 0.0,
}
from config import ModelConfigV15
mc = ModelConfigV15(**{k: v for k, v in config.items() if k != 'model_type' and k != 'architectures' and k != 'format_version'})

# Load model
model = DizelLM(mc)
ckpt = torch.load("dizel-v15-distill-best.pt", map_location="cpu", weights_only=False)
model.load_state_dict(ckpt["model_state"], strict=False)
model.eval()

# Load tokenizer
tokenizer = Tokenizer(model_path="dizel_v15.model")

# Generate
prompt = "def fibonacci(n):"
tokens = tokenizer.encode(prompt, bos=True, eos=False)
x = torch.tensor([tokens])
with torch.no_grad():
    logits, _ = model(x)
    next_token = logits[0, -1].argmax()
print(tokenizer.decode([next_token.item()]))

Files

File	Description
dizel-v15-pretrain-best.pt	Model weights + optimizer state + config
dizel-v15-sft-best.pt	Model weights + optimizer state + config
dizel-v15-distill-best.pt	Model weights + optimizer state + config
dizel_v15.model	SentencePiece tokenizer (100K vocab)
config.json	Model hyperparameters
architecture.py	Model source code (PyTorch)
rope.py	RoPE implementation
requirements.txt	Python dependencies
Dizel_banner.png	Dizel logo/banner