syntropic-clx/Clyx_0.2-115.67M-BASE

Clyx 0.2 — 115.67M BASE

Causal language model trained from scratch on ~1.57B tokens.
Custom Transformer with RoPE, RMSNorm, SwiGLU. No external pretrained weights.

---

Architecture

Parameter	Value
Parameters	115.67M
Hidden size	768
Layers	12
Attention heads	12
MLP dim	2048
Positional encoding	RoPE
Normalization	RMSNorm
Activation	SwiGLU
Context window	2048 tokens
Vocabulary	40,000 (ByteLevel BPE)
Precision	bfloat16

---

Training

Hyperparameter	Value
Optimizer	AdamW (fused)
LR schedule	Cosine with warmup
Learning rate	3e-4 → 3e-5
Batch size	64 × 4 grad accum = 256
Steps	3,000
Tokens	~1.57B
Val loss	1.4565
Hardware	NVIDIA RTX PRO 6000

Data: Russian text, English text, Python and C/C++ code (~10 GB raw).

---

Inference

Base model — continues text, does not answer questions.

temperature        = 0.7
top_k              = 50
top_p              = 0.9
repetition_penalty = 1.15
max_new_tokens     = 1024

---

Files

File	Description
model.safetensors	Model weights (PyTorch)
tokenizer.json	ByteLevel BPE tokenizer, 40k vocab
config.json	Architecture config
training_log.png	Loss curve

---

Trained from scratch as part of the Clyx project.

How to Use

Base model — continues text, does not answer questions or follow instructions.

Option 1 — via ClyxBox

pip install clyxbox

from clyxbox import ClyxModel, ClyxTokenizer

tokenizer = ClyxTokenizer.from_pretrained("syntropic-clx/Clyx_0.2-115.67M-BASE")
model     = ClyxModel.from_pretrained("syntropic-clx/Clyx_0.2-115.67M-BASE")
model.eval()

prompt = "Once upon a time in a dark forest"
ids    = tokenizer.encode(prompt, return_tensors="pt")
out    = model.generate(ids, max_new_tokens=200, temperature=0.7, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Option 2 — load weights manually

from safetensors.torch import load_file
state_dict = load_file("model.safetensors")

Special Tokens

Token	Role
<BOS>	Beginning of sequence
<STOP>	End of sequence
<USER> / </USER>	User turn
<MODEL> / </MODEL>	Model turn
<SYSTEM> / </SYSTEM>	System prompt
<PAD>	Padding