kdirgul commited on
Commit
bee291f
·
verified ·
1 Parent(s): 44f8f05

Hucre 4 canli cikti (! -u)

Browse files
Files changed (1) hide show
  1. SmartCore_V1_Colab_Faz2.ipynb +8 -22
SmartCore_V1_Colab_Faz2.ipynb CHANGED
@@ -6,12 +6,8 @@
6
  "source": [
7
  "# SmartCore V1 - Faz 2 Colab Smoke (GPU)\n",
8
  "\n",
9
- "180M Mamba-3 (SISO) + GQA 5:1 hibrit modelin smoke testi: HF'deki pre-tokenize veriden\n",
10
- "(kdirgul/smartcore-v1-data) streaming + model + WSD egitim dongusu, GPU'da.\n",
11
- "\n",
12
- "Onkosul: Colab Pro+ / A100 (Runtime > Change runtime type > A100). HF token (private repo erisimi).\n",
13
- "Amac: veri akisi + model + loop + GPU uctan uca (loss ~10.8'den dusuyor, NaN/OOM yok).\n",
14
- "NOT: smoke saf-PyTorch minimal modeli kullanir (yavas). Tam-hiz Faz 3 = mamba-og fork (Triton)."
15
  ]
16
  },
17
  {
@@ -70,7 +66,7 @@
70
  "cell_type": "markdown",
71
  "metadata": {},
72
  "source": [
73
- "## 3) Smoke (kucuk config - hizli dogrulama)"
74
  ]
75
  },
76
  {
@@ -79,14 +75,9 @@
79
  "execution_count": null,
80
  "outputs": [],
81
  "source": [
82
- "import subprocess, sys\n",
83
- "cmd = [sys.executable, '-u', 'kod/faz2_train_shards.py',\n",
84
- " '--data', 'kdirgul/smartcore-v1-data', '--hf',\n",
85
- " '--d_model', '512', '--n_layer', '12', '--seq_len', '1024',\n",
86
- " '--micro_batch', '8', '--steps', '40', '--device', 'cuda']\n",
87
- "print('calisiyor:', ' '.join(cmd))\n",
88
- "r = subprocess.run(cmd, cwd=CODE, env={**os.environ})\n",
89
- "print('exit code:', r.returncode)"
90
  ]
91
  },
92
  {
@@ -94,14 +85,9 @@
94
  "metadata": {},
95
  "source": [
96
  "## Basari kriterleri\n",
97
- "- loss ~10.8'den (=ln 48000) dusuyor (40 adimda hafif dusus normal).\n",
98
- "- grad_norm stabil (clip=1.0). NaN/OOM yok, exit code 0. A100'de makul throughput.\n",
99
  "\n",
100
- "## Siradaki: Faz 3 (tam pretraining)\n",
101
- "1. Fork/Triton: minimal yerine mamba-og (Triton SISO) -> ~10-50x hiz. referans_kod/model.py + GQA + bf16.\n",
102
- "2. Tam config: 768/20/2048, batch ~0.5M (grad accum), WSD, 12B token.\n",
103
- "3. Checkpoint + async HF push + --resume (referans_kod/train_180m.py harness).\n",
104
- "4. Faz 4 decay + Faz 5 eval."
105
  ]
106
  }
107
  ],
 
6
  "source": [
7
  "# SmartCore V1 - Faz 2 Colab Smoke (GPU)\n",
8
  "\n",
9
+ "Pre-tokenize veriden (kdirgul/smartcore-v1-data) streaming + model + WSD loop, A100.\n",
10
+ "Onkosul: Colab Pro+/A100. NOT: smoke saf-PyTorch minimal (yavas); tam-hiz Faz 3 = mamba-og fork (Triton)."
 
 
 
 
11
  ]
12
  },
13
  {
 
66
  "cell_type": "markdown",
67
  "metadata": {},
68
  "source": [
69
+ "## 3) Smoke (canli cikti)"
70
  ]
71
  },
72
  {
 
75
  "execution_count": null,
76
  "outputs": [],
77
  "source": [
78
+ "import os, sys\n",
79
+ "os.chdir(CODE)\n",
80
+ "!{sys.executable} -u kod/faz2_train_shards.py --data kdirgul/smartcore-v1-data --hf --d_model 512 --n_layer 12 --seq_len 1024 --micro_batch 8 --steps 40 --device cuda"
 
 
 
 
 
81
  ]
82
  },
83
  {
 
85
  "metadata": {},
86
  "source": [
87
  "## Basari kriterleri\n",
88
+ "- loss ~10.8'den dusuyor, NaN yok, exit 0. (Dogrulandi: loss 10.875->9.428.)\n",
 
89
  "\n",
90
+ "## Siradaki: Faz 3 (fork/Triton + tam config + checkpoint/resume). Bkz. sohbet 3a-3d plani."
 
 
 
 
91
  ]
92
  }
93
  ],