VeirdedAI/SLLM-0.7-PHY (Small-LLM) - Chat + Physics + Identity
SLLM-0.7-PHY (Small-LLM) is a lightweight ~5M-parameter causal language model developed from scratch by VeirdedAI.
The core research question behind SLLM is:
"How much conversational fluency, basic physical reasoning, and identity awareness can be packed into a 5-million-parameter budget?"
This repository hosts the SLLM-0.7-PHY checkpoint trained on internal datasets: Chat, SimplePhysics, and Identity datasets), designed for fast, stateless single-turn conversation on consumer hardware.
Model Details
- Model ID:
VeirdedAI/SLLM-0.7-PHY - Developed by: VeirdedAI
- Model Type: Causal Autoregressive Language Model (Decoder-Only)
- Language(s): English (
en) - License: MIT
- Tokenizer: SLLM-BPE (Lossless Byte-Level BPE, 4,096 tokens)
- Total Parameters: ~5,000,000 (weight-tied)
- Max Turn Length: 128 tokens (Stateless single-turn, no chat memory)
Key Architectural Features
- RoPE (Rotary Position Embeddings): Real-valued rotary embeddings without complex-tensor overhead.
- QK-Norm: RMSNorm applied to Queries and Keys per head before RoPE for stable attention logits.
- SwiGLU Activations:
w2(silu(w1(x)) * w3(x))for compact parameter expressiveness. - RMSNorm & Weight Tying: Tied input/output embeddings (
tok_emb.weight = lm_head.weight). - Zero-Leakage Attention Masking: Explicit boolean causal & padding masking.
How to Download & Run
generate.py automatically detects NVIDIA CUDA (cuda), Apple Silicon Metal (mps), or CPU.
macOS (Terminal)
git clone https://huggingface.co/VeirdedAI/SLLM-0.7-PHY
cd SLLM-0.7-PHY
python3 -m pip install torch safetensors
python3 generate.py
Linux (Bash / Ubuntu / Debian)
git clone https://huggingface.co/VeirdedAI/SLLM-0.7-PHY
cd SLLM-0.7-PHY
python3 -m pip install torch safetensors
python3 generate.py
Windows (PowerShell or Command Prompt)
git clone https://huggingface.co/VeirdedAI/SLLM-0.7-PHY
cd SLLM-0.7-PHY
pip install torch safetensors
python generate.py
Training Details
- Datasets: 3 curated single-turn datasets (
Chat.txt,SimplePhysics.txt, andIdentity.txt) with a 90/10 train/validation split. - Stage 1 (Pre-Training): 16 epochs (~80k+ examples/epoch: 50,000 Chat + 30,000 Physics + all Identity),
BATCH_SIZE=32, AdamW (lr=1.2e-3 -> 8.0e-5,weight_decay=0.05,dropout=0.10),bfloat16autocast on MPS. - Stage 2 (AI-Masked SFT): 6 epochs (~80k+ examples/epoch),
BATCH_SIZE=32, AdamW (lr=1.8e-4 -> 1.0e-5, loss computed on AI response tokens only).
Technical Specifications
| Parameter | Value |
|---|---|
| Model ID | VeirdedAI/SLLM-0.7-PHY |
| Model Size | ~5M Parameters |
| Embedding Dimension ($d_{model}$) | 256 |
| Attention Heads | 8 (Head Dimension = 32 + QK-Norm) |
| Layers | 6 |
| SwiGLU Hidden Dimension | 640 |
| Vocabulary Size | 4,096 (Byte-Level BPE) |
| Max Turn Length | 128 tokens (Stateless single-turn, no chat memory) |
Citation
@misc{veirded2026sllm07phy,
author = {VeirdedAI},
title = {SLLM-0.7-PHY: A 5M-Parameter Single-Turn Chat, Physics, and Identity Language Model},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/VeirdedAI/SLLM-0.7-PHY}}
}