A newer version of this model is available: Veirded/SLLM-0.6

Veirded/SLLM-0.5 (Small-LLM) - Base Pre-trained Model

SLLM-0.5 (Small-LLM) is an experimental, ultra-lightweight causal language model developed from scratch by Veirded.

The core research question behind SLLM is:

"How much semantic context, conversational structure, and linguistic knowledge can be packed into an ultra-tiny parameter budget?"

This repository hosts the base pre-trained checkpoint, intended as a flexible foundation for researchers and tinkerers to explore downstream fine-tuning, knowledge distillation, low-resource embedded deployment, and architectural benchmarking.


Model Details

Model Description

SLLM is built from scratch in PyTorch, departing from standard vanilla Transformers by integrating modern high-efficiency primitives and custom architectural innovations designed specifically for compact language models.

  • Model ID: Veirded/SLLM-0.5
  • Developed by: Veirded
  • Model Type: Causal Autoregressive Language Model (Decoder-Only)
  • Architecture Name: SLLM (Small-LLM)
  • Language(s): English (en)
  • License: MIT (Fully permissive for open research and commercial downstream use)
  • Tokenizer: SLLM-T (Custom 100% corpus-coverage vocabulary)
  • Total Parameters: ~2.8 Million
  • Context Length: 128 tokens

Key Architectural Innovations

  1. RoPE (Rotary Position Embeddings): Dynamic relative distance calculation eliminating rigid positional lookups.
  2. SwiGLU Activations: Gated non-linear feedforward networks (w2(silu(w1(x)) * w3(x))) for maximal parameter expressiveness.
  3. RMSNorm: Root Mean Square Layer Normalization for stable gradient propagation.
  4. Dynamic Context Gate (DCG): Learnable sigmoid gating on attention and FFN residuals to filter early training noise and accelerate convergence.
  5. Fast-Highway Semantic Skip (FHSS): Direct parametric bypass linking early input embeddings to the output normalization stream, preventing semantic dilution.

Uses

Direct Use

  • Base Pre-trained Foundation: Designed for immediate experimentation, prompt completion, and studying representation learning in sub-3M models.
  • Embedded & Edge Exploration: Capable of running inference in under 50 MB of RAM on Raspberry Pi, mobile devices, microcontrollers, or web browsers.

Downstream Use

  • Instruction fine-tuning (SFT) or RLHF/DPO experiments.
  • Specialized task adaptation (classification, domain-specific chat, edge agent routing).

Out-of-Scope Use

  • Not designed for complex factual retrieval, mathematical proofs, or multi-step logic.
  • Avoid deploying in mission-critical, medical, legal, or production-safety environments without downstream alignment.

How to Get Started with the Model

SLLM runs natively on macOS, Linux, and Windows. Below are cross-platform guides and ready-to-run Python code for all operating systems.

1. Requirements

Install standard PyTorch:

pip install torch

2. Multi-Platform Inference Script (generate.py)

Save the following script as generate.py alongside your sllm_weights.pt and tokenizer_sllm_t.json.

It automatically detects and leverages hardware acceleration:

  • Mac: Apple Silicon GPU Metal (mps)
  • Linux: NVIDIA CUDA GPU (cuda)
  • Windows: NVIDIA CUDA GPU (cuda) or optimized CPU fallback (cpu)
import os
import re
import json
import torch
import torch.nn as nn
import torch.nn.functional as F

# 1. Automatic Hardware Accelerator Detection (Mac / Linux / Windows)
if torch.backends.mps.is_available():
    device = torch.device("mps")
    print("Using Apple Silicon Metal (MPS) acceleration.")
elif torch.cuda.is_available():
    device = torch.device("cuda")
    print(f"Using NVIDIA CUDA acceleration: {torch.cuda.get_device_name(0)}")
else:
    device = torch.device("cpu")
    print("Using CPU inference.")

# 2. Architecture Definition: SLLM (Small-LLM)
class RMSNorm(nn.Module):
    def __init__(self, dim, eps=1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))
    def forward(self, x):
        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.weight

class SwiGLU(nn.Module):
    def __init__(self, dim, hidden_dim):
        super().__init__()
        self.w1 = nn.Linear(dim, hidden_dim, bias=False)
        self.w2 = nn.Linear(hidden_dim, dim, bias=False)
        self.w3 = nn.Linear(dim, hidden_dim, bias=False)
    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

def precompute_freqs_cis(dim: int, end: int, theta: float = 10000.0):
    freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))
    t = torch.arange(end, device=freqs.device)
    freqs = torch.outer(t, freqs).float()
    return torch.polar(torch.ones_like(freqs), freqs)

def apply_rotary_emb(xq, xk, freqs_cis):
    xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2))
    xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2))
    freqs_cis = freqs_cis[:xq_.shape[1]].unsqueeze(0).unsqueeze(2)
    xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(3)
    xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(3)
    return xq_out.type_as(xq), xk_out.type_as(xk)

class RoPEAttention(nn.Module):
    def __init__(self, d_model=112, n_head=4, block_size=256):
        super().__init__()
        self.n_head = n_head
        self.head_dim = d_model // n_head
        self.wq = nn.Linear(d_model, d_model, bias=False)
        self.wk = nn.Linear(d_model, d_model, bias=False)
        self.wv = nn.Linear(d_model, d_model, bias=False)
        self.wo = nn.Linear(d_model, d_model, bias=False)
        self.register_buffer('bias', torch.tril(torch.ones(block_size, block_size)).view(1, 1, block_size, block_size))

    def forward(self, x, freqs_cis):
        B, T, C = x.shape
        q = self.wq(x).view(B, T, self.n_head, self.head_dim)
        k = self.wk(x).view(B, T, self.n_head, self.head_dim)
        v = self.wv(x).view(B, T, self.n_head, self.head_dim)
        q, k = apply_rotary_emb(q, k, freqs_cis)
        q = q.transpose(1, 2)
        k = k.transpose(1, 2)
        v = v.transpose(1, 2)
        att = (q @ k.transpose(-2, -1)) * (1.0 / (self.head_dim ** 0.5))
        att = att.masked_fill(self.bias[:, :, :T, :T] == 0, float('-inf'))
        att = F.softmax(att, dim=-1)
        y = att @ v
        y = y.transpose(1, 2).contiguous().view(B, T, C)
        return self.wo(y)

class SLLMBlock(nn.Module):
    def __init__(self, d_model=112, n_head=4, hidden_dim=298, block_size=256):
        super().__init__()
        self.attn_norm = RMSNorm(d_model)
        self.attn = RoPEAttention(d_model, n_head, block_size)
        self.attn_gate = nn.Linear(d_model, d_model, bias=False)
        self.ffn_norm = RMSNorm(d_model)
        self.ffn = SwiGLU(d_model, hidden_dim)
        self.ffn_gate = nn.Linear(d_model, d_model, bias=False)

    def forward(self, x, freqs_cis):
        x = x + torch.sigmoid(self.attn_gate(x)) * self.attn(self.attn_norm(x), freqs_cis)
        x = x + torch.sigmoid(self.ffn_gate(x)) * self.ffn(self.ffn_norm(x))
        return x

class SLLM(nn.Module):
    def __init__(self, vocab_size=9385, d_model=112, n_head=4, n_layer=4, block_size=128):
        super().__init__()
        self.block_size = block_size
        self.tok_emb = nn.Embedding(vocab_size, d_model)
        self.layers = nn.ModuleList([SLLMBlock(d_model, n_head, int(2 * (4 * d_model) / 3), block_size) for _ in range(n_layer)])
        self.norm = RMSNorm(d_model)
        self.highway_gain = nn.Parameter(torch.tensor(0.1))
        self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
        self.register_buffer('freqs_cis', precompute_freqs_cis(d_model // n_head, block_size))

    def forward(self, idx):
        B, T = idx.shape
        x_emb = self.tok_emb(idx)
        freqs_cis = self.freqs_cis[:T]
        x = x_emb
        for layer in self.layers:
            x = layer(x, freqs_cis)
        x_out = self.norm(x + self.highway_gain * x_emb)
        return self.lm_head(x_out)

# 3. Load Tokenizer & Model Checkpoint
with open("tokenizer_sllm_t.json", "r", encoding="utf-8") as f:
    tok_data = json.load(f)
tok2id = tok_data["tok2id"]
id2tok = {int(k): v for k, v in tok_data["id2tok"].items()}
end_id = tok2id["<end_of_text>"]
pattern = r"(?:<end_of_text>|\w+|[^\w\s])"

model = SLLM(vocab_size=len(tok2id), block_size=256).to(device)
model.load_state_dict(torch.load("sllm_weights.pt", map_location=device))
model.eval()

# 4. Interactive Generation Loop
def detokenize(tokens):
    out = ""
    for t in tokens:
        if t in {".", ",", "!", "?", "'", ";", ":", ")", "]"}: out = out.rstrip() + t + " "
        elif t in {"(", "["}: out += t
        else: out += t + " "
    return out.strip()

print("\n" + "="*50)
print("πŸ€– Connected to Veirded/SLLM-0.5 (~2.8M Parameters)")
print("Type 'exit' to quit.")
print("="*50)

history = []
while True:
    try:
        user_msg = input("\nYou: ").strip()
    except (KeyboardInterrupt, EOFError):
        break
    if not user_msg or user_msg.lower() in ("quit", "exit"):
        break

    user_tokens = re.findall(pattern, user_msg)
    history.extend(user_tokens)
    ctx = history[-256:]
    idx = torch.tensor([[tok2id.get(t, 0) for t in ctx]], dtype=torch.long, device=device)

    reply_tokens = []
    with torch.no_grad():
        for step in range(50):
            logits = model(idx[:, -256:])[:, -1, :] / 0.75
            if step < 8: logits[0, end_id] = -float('Inf')
            for past_tok in set(reply_tokens[-6:]): logits[0, tok2id.get(past_tok, 0)] /= 1.4
            v, _ = torch.topk(logits, min(30, logits.size(-1)))
            logits[logits < v[:, [-1]]] = -float('Inf')
            probs = F.softmax(logits, dim=-1)
            next_id = torch.multinomial(probs, num_samples=1)
            token_str = id2tok[next_id.item()]
            if token_str == "<end_of_text>": break
            reply_tokens.append(token_str)
            idx = torch.cat((idx, next_id), dim=1)

    print(f"Bot: {detokenize(reply_tokens)}")
    history.extend(reply_tokens)

3. Launching on Specific Platforms

On macOS (Terminal):

python3 generate.py

(Automatically engages Apple Silicon Metal MPS)

On Linux (Bash / Ubuntu / Debian):

python3 generate.py

(Automatically engages NVIDIA CUDA if available, else CPU)

On Windows (PowerShell or Command Prompt):

python generate.py

(Automatically engages CUDA if an NVIDIA GPU is installed, else multi-threaded CPU)


Training Details

Training Data

  • Dataset: Curated multi-turn conversational corpus consisting of 4,400+ natural back-and-forth dialogue exchanges across thousands of daily life topics.
  • Token Coverage: SLLM-T achieves 100.0% coverage over the corpus vocabulary (zero <unk> artifacts).

Training Hyperparameters

  • Optimizer: AdamW ($\beta_1 = 0.9, \beta_2 = 0.999$, Weight Decay = 0.01)
  • Learning Rate: $2.0 \times 10^{-3}$ with Cosine Annealing decay down to $1.0 \times 10^{-4}$
  • Batch Size: 24
  • Context Length: 256 tokens
  • Precision: FP32 on Apple Silicon Metal Performance Shaders (MPS)
  • Gradient Clipping: Max norm 1.0

Technical Specifications

Parameter Value
Model ID Veirded/SLLM-0.5
Model Size ~2.8 Million Parameters
Embedding Dimension ($d_{model}$) 112
Attention Heads 4 (Head Dimension = 28)
Layers 4
SwiGLU Hidden Dimension 298
Vocabulary Size 9,385 (100% Corpus Coverage)
Context Window 256 tokens

Hardware Trained On

  • Machine: Apple MacBook Air M2 (Fanless)
  • Memory: 8 GB Unified Memory (RAM)
  • Accelerator Backend: Apple Silicon Metal Performance Shaders (torch.device("mps"))
  • Peak Training RAM: ~1.4 GB (Optimized low-footprint training pipeline)
  • Average Epoch Speed: ~12–14 seconds per epoch

Citation & Authors

Created by: Veirded
Project: SLLM Research Initiative (Small-LLM Explorations)
Repository: Veirded/SLLM-0.5

If you use SLLM in your own micro-LLM benchmarks, experiments, or embedded projects, please credit:

@misc{veirded2026sllm,
  author = {Veirded},
  title = {SLLM-0.5: Exploring the Knowledge Density and Parameter Efficiency of Sub-3M Language Models},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Veirded/SLLM-0.5}}
}
Downloads last month
25
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support