VeirdedAI/SLLM-0.7-PHY (Small-LLM) - Chat + Physics + Identity

SLLM-0.7-PHY (Small-LLM) is a lightweight ~5M-parameter causal language model developed from scratch by VeirdedAI.

The core research question behind SLLM is:

"How much conversational fluency, basic physical reasoning, and identity awareness can be packed into a 5-million-parameter budget?"

This repository hosts the SLLM-0.7-PHY checkpoint trained on internal datasets: Chat, SimplePhysics, and Identity datasets), designed for fast, stateless single-turn conversation on consumer hardware.


Model Details

  • Model ID: VeirdedAI/SLLM-0.7-PHY
  • Developed by: VeirdedAI
  • Model Type: Causal Autoregressive Language Model (Decoder-Only)
  • Language(s): English (en)
  • License: MIT
  • Tokenizer: SLLM-BPE (Lossless Byte-Level BPE, 4,096 tokens)
  • Total Parameters: ~5,000,000 (weight-tied)
  • Max Turn Length: 128 tokens (Stateless single-turn, no chat memory)

Key Architectural Features

  1. RoPE (Rotary Position Embeddings): Real-valued rotary embeddings without complex-tensor overhead.
  2. QK-Norm: RMSNorm applied to Queries and Keys per head before RoPE for stable attention logits.
  3. SwiGLU Activations: w2(silu(w1(x)) * w3(x)) for compact parameter expressiveness.
  4. RMSNorm & Weight Tying: Tied input/output embeddings (tok_emb.weight = lm_head.weight).
  5. Zero-Leakage Attention Masking: Explicit boolean causal & padding masking.

How to Download & Run

generate.py automatically detects NVIDIA CUDA (cuda), Apple Silicon Metal (mps), or CPU.

macOS (Terminal)

git clone https://huggingface.co/VeirdedAI/SLLM-0.7-PHY
cd SLLM-0.7-PHY
python3 -m pip install torch safetensors
python3 generate.py

Linux (Bash / Ubuntu / Debian)

git clone https://huggingface.co/VeirdedAI/SLLM-0.7-PHY
cd SLLM-0.7-PHY
python3 -m pip install torch safetensors
python3 generate.py

Windows (PowerShell or Command Prompt)

git clone https://huggingface.co/VeirdedAI/SLLM-0.7-PHY
cd SLLM-0.7-PHY
pip install torch safetensors
python generate.py

Training Details

  • Datasets: 3 curated single-turn datasets (Chat.txt, SimplePhysics.txt, and Identity.txt) with a 90/10 train/validation split.
  • Stage 1 (Pre-Training): 16 epochs (~80k+ examples/epoch: 50,000 Chat + 30,000 Physics + all Identity), BATCH_SIZE=32, AdamW (lr=1.2e-3 -> 8.0e-5, weight_decay=0.05, dropout=0.10), bfloat16 autocast on MPS.
  • Stage 2 (AI-Masked SFT): 6 epochs (~80k+ examples/epoch), BATCH_SIZE=32, AdamW (lr=1.8e-4 -> 1.0e-5, loss computed on AI response tokens only).

Technical Specifications

Parameter Value
Model ID VeirdedAI/SLLM-0.7-PHY
Model Size ~5M Parameters
Embedding Dimension ($d_{model}$) 256
Attention Heads 8 (Head Dimension = 32 + QK-Norm)
Layers 6
SwiGLU Hidden Dimension 640
Vocabulary Size 4,096 (Byte-Level BPE)
Max Turn Length 128 tokens (Stateless single-turn, no chat memory)

Citation

@misc{veirded2026sllm07phy,
  author = {VeirdedAI},
  title = {SLLM-0.7-PHY: A 5M-Parameter Single-Turn Chat, Physics, and Identity Language Model},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/VeirdedAI/SLLM-0.7-PHY}}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
5.57M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support