Upload folder using huggingface_hub

.gitattributes

@@ -1,35 +1 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
 *.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,185 @@
+---
+license: mit
+language:
+- en
+library_name: transformers
+tags:
+- text-generation
+- causal-lm
+- gpt2
+- tinystories
+- llm-kittens
+- thunderkittens
+- bf16
+- cuda
+- sd-prelu
+- custom-activation
+- custom_code
+datasets:
+- roneneldan/TinyStories
+---
+
+# llm.kittens TinyStories 124M BF16 — SD-PReLU
+
+This is a 124M-parameter GPT-2-style causal language model trained from scratch on TinyStories with the `llm.kittens` C++/CUDA trainer, which is a fork of Karpathy's llm.c with some optimisations for SM120, and multi-stack kernel optimisations.
+
+Unlike the GELU baseline ([`tinystories-5090`](https://huggingface.co/adamroberts/tinystories-5090)), this checkpoint replaces the MLP's GELU nonlinearity with a learnable **SD-PReLU** activation (the llm.kittens `-af sd-prelu` activation). Because SD-PReLU is not part of stock Transformers or llama.cpp, the repo ships a small custom module (`modeling_gpt_sdprelu.py`) and must be loaded with `trust_remote_code=True`. See [Activation: SD-PReLU](#activation-sd-prelu) below.
+
+The model is published as a Hugging Face Transformers checkpoint with BF16 `safetensors` weights plus custom modeling code.
+It was trained on a single RTX 5090 in roughly 14 hours (2595.7 ms average iteration over 20,000 steps).
+
+## Result
+
+- Model weights: `model.safetensors`
+- Training step: `20000 / 20000`
+- Final train loss: `0.781594`
+- Final validation loss: `0.870315`
+- Final throughput: `198871 tokens/s`
+- Final step time: `2642.72 ms`
+- Final reported BF16 MFU: `38.0%`
+- Average iteration time: `2595.669013 ms`
+- Safetensors size: `248,896,984` bytes
+- Parameter count: `124,475,904` base + `24` learnable SD-PReLU scalars (`theta_a`/`theta_b`, 2 per layer × 12 layers)
+
+For reference, the GELU baseline reached `0.785740` train / `0.875080` validation loss on the same setup; SD-PReLU lands marginally lower on both (`0.781594` / `0.870315`) at near-identical throughput, with the activation adding only 24 scalar parameters.
+
+The TinyStories paper reports eval losses of `1.33` to `1.58` for the 768-hidden-size 1- and 2-layer attention-head ablations in Figure 24. This run's `0.870315` validation loss is lower, but the comparison is not apples-to-apples: this model is a 12-layer GPT-2-style model using GPT-2 tokenization, a 1024-token context, and a different implementation/training setup.
+
+## Activation: SD-PReLU
+
+This is the key difference from the GELU baseline. The MLP's GELU nonlinearity (applied between the `c_fc` up-projection and the `c_proj` down-projection) is replaced by **SD-PReLU**, a self-gated, damped PReLU. Each transformer block learns two scalars (`theta_a`, `theta_b`) that are mapped through a bounded reparameterization into `a` and `b`:
+
+```text
+a = alpha_max * sigmoid(theta_a)      # in [0, alpha_max),  alpha_max = 0.30
+b = beta_min  + softplus(theta_b)     # in (beta_min, inf), beta_min  = 0.50
+phi(x) = x * (a + (1 - a) * sigmoid(b * x))
+```
+
+- `a` is a learnable leak/floor (PReLU-like): the gate output never drops below `a`, so negative inputs are not fully zeroed.
+- `b` controls the sharpness of the sigmoid gate. With `a = 0`, `phi` reduces to a Swish/SiLU-style `x * sigmoid(b * x)`.
+- For numerical parity with the CUDA kernel, the activation is computed in float32 and the reparameterization/gate arguments are clamped to `[-20, 20]`; inputs and outputs stay in the surrounding model dtype (BF16).
+
+The activation is configured via two config fields, `sdprelu_alpha_max` (`0.30`) and `sdprelu_beta_min` (`0.50`). Only 24 extra scalar parameters are introduced over the GELU baseline, so parameter count and on-disk size are essentially unchanged.
+
+Note: the config still carries `"activation_function": "gelu_new"` for GPT-2 compatibility, but it is inert — the custom MLP overrides the activation with SD-PReLU regardless of that field.
+
+## Custom activation inference code
+
+The repo ships `modeling_gpt_sdprelu.py`, which defines:
+
+- `GPTSDPReLUConfig` (`model_type = "gpt-sdprelu"`), a `GPT2Config` with the two extra `sdprelu_*` fields.
+- `GPTSDPReLUMLP`, which reuses GPT-2's `c_fc` / `c_proj` / dropout submodules (so weight names stay `mlp.c_fc.*` / `mlp.c_proj.*`) and swaps GELU for SD-PReLU, adding `theta_a` / `theta_b` per layer.
+- `GPTSDPReLULMHeadModel`, a `GPT2LMHeadModel` that installs the SD-PReLU MLP into every block.
+
+These are wired into `config.json` via `auto_map`, so `trust_remote_code=True` is required to load the model. Everything else (attention, layernorms, embeddings, tied head, tokenizer) is standard GPT-2.
+
+## Architecture
+
+- Family: GPT-2-style decoder-only Transformer
+- Descriptor: `d12`
+- Layers: `12`
+- Attention heads: `12`
+- Hidden size: `768`
+- Context length: `1024`
+- Vocabulary size: `50,257`
+- MLP activation: **SD-PReLU** (learnable, per-layer) — replaces GELU
+- `model_type`: `gpt-sdprelu` (custom code, `trust_remote_code=True`)
+- Precision: BF16 weights
+
+## Training
+
+The run used the TinyStories GPT-2 dataset files generated by `dev/data/tinystories.py` in `llm.kittens`. The only change from the GELU baseline is the `-af sd-prelu` activation flag.
+
+```bash
+./train_gpt2cu \
+    -i "dev/data/tinystories/TinyStories_train.bin" \
+    -j "dev/data/tinystories/TinyStories_val.bin" \
+    -o "log124M/5090_S" \
+    -v 250 -s 20000 -g 144 \
+    -h 0 \
+    -b 64 -t 1024 -d 524288 \
+    -r 0 \
+    -z 1 \
+    -c 0.1 \
+    -l 0.0006 -q 0.0 -u 700 -n 5000 \
+    -y 0 \
+    -e "d12" \
+    -af sd-prelu \
+    -x 20000
+```
+
+Key settings:
+
+- Hardware target: RTX 5090 / SM120
+- MLP activation: `sd-prelu` (`-af sd-prelu`)
+- Micro batch: `64`
+- Sequence length: `1024`
+- Total desired batch size: `524,288` tokens
+- Max steps: `20,000`
+- Optimizer: AdamW as implemented in `llm.kittens`
+- Peak learning rate: `6e-4`
+- Scheduler: cosine
+- Warmup: `700` steps
+- Final LR fraction: `0.0`
+- Weight decay: `0.1`
+- Recompute: off
+- ZeRO stage: `1`
+- Checkpoint interval: `5000` steps
+
+## Sample
+
+Prompt/sample emitted at the final checkpoint (step 20000):
+
+```text
+Once upon a time, there was a little boy named Timmy. Timmy loved his soft couch and he kept it in his room. One day, Timmy's mom told him to clean his room. Timmy didn't want to clean his room because he loved playing with his toys. But his mom said he couldn't play with his toys until he cleaned his room.
+Timmy felt sad and he started to hate cleaning his room. Then, he had an idea. He asked his mom if he could put away his toys in a different way and make a game out of it. His mom thought it was a good idea too and they both started to work together.
+In the end, Tim
+```
+
+## Files
+
+- `model.safetensors`: BF16 Transformers weights (including the per-layer `theta_a` / `theta_b` SD-PReLU scalars).
+- `modeling_gpt_sdprelu.py`: custom SD-PReLU model/config code (required, loaded via `trust_remote_code=True`).
+- `config.json`: model configuration, including `sdprelu_alpha_max` / `sdprelu_beta_min` and the `auto_map` wiring.
+- `generation_config.json`: default generation settings.
+- `tokenizer.json`: GPT-2 tokenizer.
+- `vocab.json` and `merges.txt`: GPT-2 BPE vocabulary files.
+
+## Loading
+
+Because the SD-PReLU activation lives in custom code, you must pass `trust_remote_code=True`:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "adamroberts/tinystories-5090-sdprelu"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, trust_remote_code=True, dtype=torch.bfloat16
+)
+
+inputs = tokenizer("Once upon a time", return_tensors="pt")
+with torch.inference_mode():
+    outputs = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8)
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## GGUF / llama.cpp
+
+GGUF export is **not available** for this checkpoint. The SD-PReLU activation is a custom, learnable nonlinearity with no equivalent in llama.cpp's GPT-2 graph, so the model cannot be quantized to GGUF or run with `llama.cpp` / LM Studio without implementing the activation there. Use the Transformers loading path above instead. The GELU baseline ([`tinystories-5090`](https://huggingface.co/adamroberts/tinystories-5090)) is available if you need a llama.cpp-compatible variant.
+
+Recommended sampling settings (Transformers `generate`):
+
+- Temperature: `0.8`
+- Top-p: `0.95`
+- Top-k: `50`
+- Repetition penalty: `1.05`
+- Stop/EOS token: `<|endoftext|>` / token id `50256`
+
+This is a completion model, not a chat/instruction model: prompt it with the start of a story and always set a finite `max_new_tokens`, since it was trained for continuation and may not emit `<|endoftext|>` during normal generation.
+
+Source implementation: `https://github.com/adamdroberts/llm.kittens`
+
+TinyStories reference paper: `https://arxiv.org/abs/2305.07759`

config.json

@@ -0,0 +1,41 @@
+{
+  "activation_function": "gelu_new",
+  "add_cross_attention": false,
+  "architectures": [
+    "GPTSDPReLULMHeadModel"
+  ],
+  "attn_pdrop": 0.0,
+  "auto_map": {
+    "AutoConfig": "modeling_gpt_sdprelu.GPTSDPReLUConfig",
+    "AutoModelForCausalLM": "modeling_gpt_sdprelu.GPTSDPReLULMHeadModel"
+  },
+  "bos_token_id": 50256,
+  "dtype": "bfloat16",
+  "embd_pdrop": 0.0,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt-sdprelu",
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "pad_token_id": 50256,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.0,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "sdprelu_alpha_max": 0.3,
+  "sdprelu_beta_min": 0.5,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.9.0",
+  "use_cache": true,
+  "vocab_size": 50257
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "bos_token_id": 50256,
+  "do_sample": true,
+  "eos_token_id": 50256,
+  "max_new_tokens": 144,
+  "pad_token_id": 50256,
+  "repetition_penalty": 1.05,
+  "temperature": 0.8,
+  "top_k": 50,
+  "top_p": 0.95,
+  "transformers_version": "5.9.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd518a7bbe8bd5bffdf06f9bd5c3dabe97dd8dff19af18a39e707b53f00d3d9f
+size 248896984

modeling_gpt_sdprelu.py

@@ -0,0 +1,102 @@
+"""A GPT model with the SD-PReLU (DTSG-PReLU) MLP activation, for HF Transformers.
+
+The backbone is GPT-2's (attention, layernorms, embeddings, tied head), but the
+MLP activation is no longer GELU, so these classes drop the "GPT2" name. This
+mirrors the llm.kittens `-af sd-prelu` activation exactly: per layer there are two
+learnable scalars (theta_a, theta_b), materialized through a bounded
+reparameterization and applied elementwise between the up- and down-projections:
+
+    a = alpha_max * sigmoid(theta_a)          # in [0, alpha_max)
+    b = beta_min  + softplus(theta_b)         # in (beta_min, inf)
+    phi(x) = x * (a + (1 - a) * sigmoid(b * x))
+
+To match the CUDA kernel bit-for-bit in spirit, the materialization and the gate
+sigmoid clamp their argument to [-20, 20] and the activation is computed in float32
+(inputs/outputs stay in the surrounding model dtype, typically bfloat16).
+
+Load with:
+
+    from transformers import AutoModelForCausalLM, AutoTokenizer
+    model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)
+    tok = AutoTokenizer.from_pretrained(path)
+"""
+
+from __future__ import annotations
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import GPT2Config, GPT2LMHeadModel
+from transformers.models.gpt2.modeling_gpt2 import GPT2MLP
+
+
+def sdprelu(
+    x: torch.Tensor,
+    theta_a: torch.Tensor,
+    theta_b: torch.Tensor,
+    alpha_max: float,
+    beta_min: float,
+) -> torch.Tensor:
+    """SD-PReLU activation: x * (a + (1 - a) * sigmoid(b * x)), computed in fp32."""
+    orig_dtype = x.dtype
+    x = x.float()
+    a = alpha_max * torch.sigmoid(theta_a.float().clamp(-20.0, 20.0))
+    b = beta_min + F.softplus(theta_b.float())
+    s = torch.sigmoid((b * x).clamp(-20.0, 20.0))
+    out = x * (a + (1.0 - a) * s)
+    return out.to(orig_dtype)
+
+
+class GPTSDPReLUConfig(GPT2Config):
+    model_type = "gpt-sdprelu"
+
+    def __init__(
+        self,
+        sdprelu_alpha_max: float = 0.30,
+        sdprelu_beta_min: float = 0.50,
+        **kwargs,
+    ):
+        self.sdprelu_alpha_max = sdprelu_alpha_max
+        self.sdprelu_beta_min = sdprelu_beta_min
+        super().__init__(**kwargs)
+
+
+class GPTSDPReLUMLP(nn.Module):
+    """GPT-2 MLP projections with GELU swapped for per-layer SD-PReLU.
+
+    Reuses the c_fc / c_proj / dropout submodules of an existing GPT2MLP so the
+    weight names stay `mlp.c_fc.*` / `mlp.c_proj.*`; adds two scalar parameters
+    `theta_a` and `theta_b` per layer.
+    """
+
+    def __init__(self, base_mlp: GPT2MLP, config: GPTSDPReLUConfig):
+        super().__init__()
+        self.c_fc = base_mlp.c_fc
+        self.c_proj = base_mlp.c_proj
+        self.dropout = base_mlp.dropout
+        self.alpha_max = float(config.sdprelu_alpha_max)
+        self.beta_min = float(config.sdprelu_beta_min)
+        self.theta_a = nn.Parameter(torch.zeros(1))
+        self.theta_b = nn.Parameter(torch.zeros(1))
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.c_fc(hidden_states)
+        hidden_states = sdprelu(
+            hidden_states, self.theta_a, self.theta_b, self.alpha_max, self.beta_min
+        )
+        hidden_states = self.c_proj(hidden_states)
+        hidden_states = self.dropout(hidden_states)
+        return hidden_states
+
+
+class GPTSDPReLULMHeadModel(GPT2LMHeadModel):
+    config_class = GPTSDPReLUConfig
+
+    def __init__(self, config: GPTSDPReLUConfig):
+        super().__init__(config)
+        for block in self.transformer.h:
+            block.mlp = GPTSDPReLUMLP(block.mlp, config)
+
+
+GPTSDPReLUConfig.register_for_auto_class()
+GPTSDPReLULMHeadModel.register_for_auto_class("AutoModelForCausalLM")

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}