Upload MedSLM: model weights, optimizer, tokenizer, config, and model card

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+loss_curves.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,133 @@
+---
+language:
+  - en
+license: apache-2.0
+tags:
+  - medical
+  - language-model
+  - transformer
+  - pytorch
+  - pretraining
+  - slm
+  - rope
+  - swiglu
+  - gqa
+  - rmsnorm
+datasets:
+  - Saminx22/medical_data_for_slm
+pipeline_tag: text-generation
+model-index:
+  - name: MedSLM
+    results: []
+---
+
+# MedSLM — Medical Small Language Model (~381M Parameters)
+
+A **381M parameter** transformer language model pre-trained on curated medical text from PubMed abstracts, PMC full-text articles, and clinical guidelines.
+
+## Architecture
+
+MedSLM uses a modern GPT-style transformer with several architectural improvements over the standard GPT-2 design:
+
+| Component | Detail |
+|---|---|
+| **Normalization** | RMSNorm (faster than LayerNorm, used in LLaMA/Mistral) |
+| **Positional Encoding** | Rotary Positional Embeddings (RoPE) — better length generalization |
+| **Feed-Forward** | SwiGLU activation (gated FFN, outperforms GELU) |
+| **Attention** | Grouped-Query Attention (GQA) — shared KV heads for efficiency |
+| **Layers** | 24 transformer blocks |
+| **Attention Heads** | 16 query heads, 8 KV heads |
+| **Embedding Dim** | 1024 |
+| **Context Length** | 1024 tokens |
+| **Vocab Size** | 50,257 (GPT-2 BPE tokenizer) |
+| **Parameters** | 381,373,440 (~381M) |
+
+## Training
+
+- **Dataset**: [`Saminx22/medical_data_for_slm`](https://huggingface.co/datasets/Saminx22/medical_data_for_slm) (~44M tokens)
+- **Sources**: PubMed abstracts, PMC Open Access full-text, Clinical Guidelines
+- **Tokenizer**: GPT-2 BPE tokenizer (50,257 vocab)
+- **Optimizer**: AdamW (betas=0.9/0.95, weight_decay=0.1)
+- **LR Schedule**: Linear warmup (1000 steps) + Cosine decay
+- **Peak LR**: 0.0003
+- **Precision**: bfloat16
+- **Effective Batch Size**: 256
+- **Max Steps**: 20,000
+- **Best Val Loss**: 3.2198 (at step 19500)
+
+## Usage
+
+### Loading the Model
+
+```python
+import torch
+import json
+from safetensors.torch import load_file
+from transformers import AutoTokenizer
+
+# Load config
+with open("config.json") as f:
+    config_dict = json.load(f)
+
+# Reconstruct model (requires the MedSLM class definition)
+config = MedSLMConfig(**{k: v for k, v in config_dict.items()
+                         if k in MedSLMConfig.__dataclass_fields__})
+model = MedSLM(config)
+
+# Load weights
+state_dict = load_file("model.safetensors")
+model.load_state_dict(state_dict)
+model.eval()
+
+# Load tokenizer
+tokenizer = AutoTokenizer.from_pretrained("tokenizer/")
+```
+
+### Generating Text
+
+```python
+prompt = "The patient presented with acute myocardial infarction"
+input_ids = torch.tensor(tokenizer.encode(prompt)).unsqueeze(0)
+
+output = model.generate(input_ids, max_new_tokens=200, temperature=0.8, top_k=50, top_p=0.9)
+print(tokenizer.decode(output.squeeze().tolist()))
+```
+
+### Resuming Training
+
+```python
+# Load optimizer state
+optimizer_state = torch.load("optimizer.pt")
+optimizer.load_state_dict(optimizer_state)
+```
+
+## Files
+
+| File | Description |
+|---|---|
+| `model.safetensors` | Model weights (safetensors format) |
+| `optimizer.pt` | Optimizer state dict for resuming training |
+| `config.json` | Model architecture configuration |
+| `training_config.json` | Training hyperparameters and loss history |
+| `tokenizer/` | GPT-2 tokenizer files |
+| `loss_curves.png` | Training/validation loss plot |
+
+## Intended Use
+
+This model is intended for **research purposes** in medical NLP. It can be used as:
+- A foundation model for downstream medical NLP tasks (NER, classification, QA)
+- A starting point for medical instruction tuning
+- A baseline for comparing medical language model architectures
+
+## Limitations
+
+- **Not for clinical use**: This model should NOT be used for clinical decision-making
+- **Small scale**: ~381M parameters is relatively small; larger models will perform better
+- **Limited data**: Trained on ~44M tokens (production models use trillions)
+- **No alignment**: This is a base model without instruction tuning or RLHF
+- **English only**: Trained exclusively on English medical text
+- **Potential biases**: May reflect biases present in the medical literature
+
+## License
+
+Apache 2.0

config.json

@@ -0,0 +1,18 @@
+{
+  "block_size": 1024,
+  "vocab_size": 50257,
+  "n_layer": 24,
+  "n_head": 16,
+  "n_kv_head": 8,
+  "n_embd": 1024,
+  "dropout": 0.1,
+  "rope_theta": 10000.0,
+  "architecture": "MedSLM",
+  "model_type": "med-slm",
+  "improvements": [
+    "RMSNorm (replaces LayerNorm)",
+    "Rotary Positional Embeddings / RoPE (replaces learned absolute)",
+    "SwiGLU activation (replaces GELU)",
+    "Grouped-Query Attention / GQA (replaces standard MHA)"
+  ]
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e79b77f38368e09a9d0ca5ae3ed78b8cc4d05a5b9de345adbce10de2d5c524a
+size 1319662784

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:272e4a8326b9a8f0169a5e7b3f924a09f11f19d0f9769e2e1ff3c1c8cdc70836
+size 2639460235

tokenizer/tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": false,
+  "model_max_length": 1024,
+  "pad_token": null,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_config.json

@@ -0,0 +1,104 @@
+{
+  "best_step": 19500,
+  "best_val_loss": 3.219841718673706,
+  "learning_rate": 0.0003,
+  "min_lr": 3e-05,
+  "max_iters": 20000,
+  "warmup_steps": 1000,
+  "batch_size": 8,
+  "gradient_accumulation_steps": 32,
+  "effective_batch_size": 256,
+  "block_size": 1024,
+  "precision": "bfloat16",
+  "optimizer": "AdamW",
+  "betas": [
+    0.9,
+    0.95
+  ],
+  "weight_decay": 0.1,
+  "grad_clip": 1.0,
+  "dataset": "Saminx22/medical_data_for_slm",
+  "tokenizer": "gpt2",
+  "train_losses": [
+    7.879132270812988,
+    6.511569023132324,
+    6.0676093101501465,
+    5.704098701477051,
+    5.454875946044922,
+    5.183218002319336,
+    4.928450107574463,
+    4.7383880615234375,
+    4.58985710144043,
+    4.431442737579346,
+    4.286258697509766,
+    4.190935134887695,
+    4.071569442749023,
+    3.9881348609924316,
+    3.9011902809143066,
+    3.81817626953125,
+    3.7602646350860596,
+    3.687962532043457,
+    3.623549461364746,
+    3.596832275390625,
+    3.516059160232544,
+    3.483616352081299,
+    3.4555230140686035,
+    3.4156558513641357,
+    3.3746585845947266,
+    3.366776704788208,
+    3.336205244064331,
+    3.320267915725708,
+    3.2959280014038086,
+    3.2804367542266846,
+    3.2574355602264404,
+    3.247173547744751,
+    3.244846820831299,
+    3.2060155868530273,
+    3.192638635635376,
+    3.1810803413391113,
+    3.1632189750671387,
+    3.161890983581543,
+    3.1812844276428223
+  ],
+  "val_losses": [
+    7.889859199523926,
+    6.505797863006592,
+    6.062633037567139,
+    5.6944580078125,
+    5.421933174133301,
+    5.1644487380981445,
+    4.942782402038574,
+    4.735868453979492,
+    4.573709487915039,
+    4.4302897453308105,
+    4.306708812713623,
+    4.180722713470459,
+    4.104051113128662,
+    4.005853652954102,
+    3.9109673500061035,
+    3.827249050140381,
+    3.7792670726776123,
+    3.678201198577881,
+    3.652942419052124,
+    3.6105289459228516,
+    3.561580181121826,
+    3.52152156829834,
+    3.4833824634552,
+    3.45076847076416,
+    3.4399120807647705,
+    3.411548376083374,
+    3.3855507373809814,
+    3.3449840545654297,
+    3.337197780609131,
+    3.3391780853271484,
+    3.301971673965454,
+    3.291358709335327,
+    3.2936108112335205,
+    3.2686355113983154,
+    3.2617692947387695,
+    3.258305311203003,
+    3.2488183975219727,
+    3.233189105987549,
+    3.219841718673706
+  ]
+}