faz2_train_shards seq_len kirpma + notebook

code/kod/SmartCore_V1_Colab_Faz2.ipynb

@@ -0,0 +1,123 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# SmartCore V1 — Faz 2 Colab Smoke (GPU)\n",
+    "\n",
+    "180M Mamba-3 (SISO) + GQA 5:1 hibrit modelin **smoke testi**: HF'deki pre-tokenize veriden\n",
+    "(`kdirgul/smartcore-v1-data`) streaming + model + WSD eğitim döngüsü, GPU'da.\n",
+    "\n",
+    "**Önkoşul:** Colab **Pro+ / A100** (Runtime > Change runtime type > A100). HF token (private repo erişimi).\n",
+    "\n",
+    "**Smoke amacı:** veri akışı + model + loop + GPU uçtan uca çalışıyor mu (loss ~10.8'den düşüyor, NaN/OOM yok).\n",
+    "**NOT:** Bu smoke saf-PyTorch minimal modeli kullanır (her yerde çalışır, yavaş). **Tam-hız Faz 3** için\n",
+    "`mamba-og` fork (Triton SISO kernel) entegre edilecek — ayrı adım."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "!nvidia-smi -L\n",
+    "!pip -q install einops sentencepiece datasets huggingface_hub pyarrow"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1) HF login (private repo erişimi)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "from huggingface_hub import login, HfFolder\n",
+    "login()  # token'ı buraya yapıştır"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2) Kodu HF'den indir"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "import os, sys\n",
+    "from huggingface_hub import snapshot_download\n",
+    "p = snapshot_download(\"kdirgul/smartcore-v1\", allow_patterns=[\"code/**\"], repo_type=\"model\")\n",
+    "CODE = os.path.join(p, \"code\")\n",
+    "os.environ[\"HF_TOKEN\"] = HfFolder.get_token()\n",
+    "print(\"kod dizini:\", CODE)\n",
+    "print(os.listdir(os.path.join(CODE, \"kod\"))[:5])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3) Smoke (küçük config — hızlı doğrulama)\n",
+    "Pipeline'ı kanıtlar. Tam 180M (d_model=768 n_layer=20 seq=2048) da çalışır ama minimal modelle yavaştır."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "import subprocess\n",
+    "cmd = [\"python\",\"kod/faz2_train_shards.py\",\"--data\",\"kdirgul/smartcore-v1-data\",\"--hf\",\n",
+    "       \"--d_model\",\"512\",\"--n_layer\",\"12\",\"--seq_len\",\"1024\",\n",
+    "       \"--micro_batch\",\"8\",\"--steps\",\"40\",\"--device\",\"cuda\"]\n",
+    "subprocess.run(cmd, cwd=CODE, env={**os.environ})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Başarı kriterleri\n",
+    "- loss ~10.8'den (=ln 48000) düşüyor (40 adımda hafif düşüş normal — gerçek veri).\n",
+    "- grad_norm stabil (clip=1.0 ile sıçramalar kontrol altında).\n",
+    "- NaN/OOM yok.\n",
+    "- A100'de throughput makul.\n",
+    "\n",
+    "## Sıradaki: Faz 3 (tam pretraining)\n",
+    "1. **Fork/Triton entegrasyonu:** minimal model yerine `mamba-og` (Triton SISO kernel) → ~10-50× hız.\n",
+    "   `referans_kod/model.py` (Mamba3LM) + GQA katmanları (`mamba_ssm.modules.mha`) + bf16.\n",
+    "2. Tam config: d_model=768, n_layer=20, seq=2048, batch ~0.5M token (grad accum), WSD, 12B token.\n",
+    "3. Checkpoint + async HF push + `--resume` (Colab session). `referans_kod/train_180m.py` harness temel alınır.\n",
+    "4. Faz 4 decay + Faz 5 eval (TR panel + EN panel)."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python"
+  },
+  "accelerator": "GPU",
+  "colab": {
+   "provenance": []
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

code/kod/faz2_train_shards.py

@@ -28,8 +28,9 @@ def wsd_lr(step, total, peak, floor, warmup_frac=0.02, decay_frac=0.25):
 
 class ShardStream:
     """Pre-shard parquet okuyucu (yerel dizin veya HF repo), oranlı karışım."""
-    def __init__(self, root, weights, hf=False, token=None):
+    def __init__(self, root, weights, seq_len, hf=False, token=None):
         self.names = list(weights); self.w = [weights[n] for n in self.names]
+        self.seq_len = seq_len   # satırlar 2048; daha kısa smoke için kırp
         self.iters = {n: self._src(root, n, hf, token) for n in self.names}
 
     def _src(self, root, name, hf, token):
@@ -48,7 +49,8 @@ class ShardStream:
                         yield row.as_py()
 
     def batch(self, bsz):
-        rows = [next(self.iters[random.choices(self.names, weights=self.w, k=1)[0]]) for _ in range(bsz)]
+        rows = [next(self.iters[random.choices(self.names, weights=self.w, k=1)[0]])[:self.seq_len]
+                for _ in range(bsz)]
         return torch.tensor(rows, dtype=torch.long)
 
 
@@ -83,7 +85,7 @@ def main():
 
     tok = os.environ.get("HF_TOKEN")
     print(f"veri: {args.data} ({'HF stream' if args.hf else 'yerel parquet'}) | mixture {MIX}")
-    stream = ShardStream(args.data, MIX, hf=args.hf, token=tok)
+    stream = ShardStream(args.data, MIX, args.seq_len, hf=args.hf, token=tok)
 
     decay = [p for p in model.parameters() if p.ndim >= 2]
     nod = [p for p in model.parameters() if p.ndim < 2]