Hucre 4 canli cikti (! -u)
Browse files
SmartCore_V1_Colab_Faz2.ipynb
CHANGED
|
@@ -6,12 +6,8 @@
|
|
| 6 |
"source": [
|
| 7 |
"# SmartCore V1 - Faz 2 Colab Smoke (GPU)\n",
|
| 8 |
"\n",
|
| 9 |
-
"
|
| 10 |
-
"
|
| 11 |
-
"\n",
|
| 12 |
-
"Onkosul: Colab Pro+ / A100 (Runtime > Change runtime type > A100). HF token (private repo erisimi).\n",
|
| 13 |
-
"Amac: veri akisi + model + loop + GPU uctan uca (loss ~10.8'den dusuyor, NaN/OOM yok).\n",
|
| 14 |
-
"NOT: smoke saf-PyTorch minimal modeli kullanir (yavas). Tam-hiz Faz 3 = mamba-og fork (Triton)."
|
| 15 |
]
|
| 16 |
},
|
| 17 |
{
|
|
@@ -70,7 +66,7 @@
|
|
| 70 |
"cell_type": "markdown",
|
| 71 |
"metadata": {},
|
| 72 |
"source": [
|
| 73 |
-
"## 3) Smoke (
|
| 74 |
]
|
| 75 |
},
|
| 76 |
{
|
|
@@ -79,14 +75,9 @@
|
|
| 79 |
"execution_count": null,
|
| 80 |
"outputs": [],
|
| 81 |
"source": [
|
| 82 |
-
"import
|
| 83 |
-
"
|
| 84 |
-
"
|
| 85 |
-
" '--d_model', '512', '--n_layer', '12', '--seq_len', '1024',\n",
|
| 86 |
-
" '--micro_batch', '8', '--steps', '40', '--device', 'cuda']\n",
|
| 87 |
-
"print('calisiyor:', ' '.join(cmd))\n",
|
| 88 |
-
"r = subprocess.run(cmd, cwd=CODE, env={**os.environ})\n",
|
| 89 |
-
"print('exit code:', r.returncode)"
|
| 90 |
]
|
| 91 |
},
|
| 92 |
{
|
|
@@ -94,14 +85,9 @@
|
|
| 94 |
"metadata": {},
|
| 95 |
"source": [
|
| 96 |
"## Basari kriterleri\n",
|
| 97 |
-
"- loss ~10.8'den
|
| 98 |
-
"- grad_norm stabil (clip=1.0). NaN/OOM yok, exit code 0. A100'de makul throughput.\n",
|
| 99 |
"\n",
|
| 100 |
-
"## Siradaki: Faz 3 (tam
|
| 101 |
-
"1. Fork/Triton: minimal yerine mamba-og (Triton SISO) -> ~10-50x hiz. referans_kod/model.py + GQA + bf16.\n",
|
| 102 |
-
"2. Tam config: 768/20/2048, batch ~0.5M (grad accum), WSD, 12B token.\n",
|
| 103 |
-
"3. Checkpoint + async HF push + --resume (referans_kod/train_180m.py harness).\n",
|
| 104 |
-
"4. Faz 4 decay + Faz 5 eval."
|
| 105 |
]
|
| 106 |
}
|
| 107 |
],
|
|
|
|
| 6 |
"source": [
|
| 7 |
"# SmartCore V1 - Faz 2 Colab Smoke (GPU)\n",
|
| 8 |
"\n",
|
| 9 |
+
"Pre-tokenize veriden (kdirgul/smartcore-v1-data) streaming + model + WSD loop, A100.\n",
|
| 10 |
+
"Onkosul: Colab Pro+/A100. NOT: smoke saf-PyTorch minimal (yavas); tam-hiz Faz 3 = mamba-og fork (Triton)."
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
]
|
| 12 |
},
|
| 13 |
{
|
|
|
|
| 66 |
"cell_type": "markdown",
|
| 67 |
"metadata": {},
|
| 68 |
"source": [
|
| 69 |
+
"## 3) Smoke (canli cikti)"
|
| 70 |
]
|
| 71 |
},
|
| 72 |
{
|
|
|
|
| 75 |
"execution_count": null,
|
| 76 |
"outputs": [],
|
| 77 |
"source": [
|
| 78 |
+
"import os, sys\n",
|
| 79 |
+
"os.chdir(CODE)\n",
|
| 80 |
+
"!{sys.executable} -u kod/faz2_train_shards.py --data kdirgul/smartcore-v1-data --hf --d_model 512 --n_layer 12 --seq_len 1024 --micro_batch 8 --steps 40 --device cuda"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
]
|
| 82 |
},
|
| 83 |
{
|
|
|
|
| 85 |
"metadata": {},
|
| 86 |
"source": [
|
| 87 |
"## Basari kriterleri\n",
|
| 88 |
+
"- loss ~10.8'den dusuyor, NaN yok, exit 0. (Dogrulandi: loss 10.875->9.428.)\n",
|
|
|
|
| 89 |
"\n",
|
| 90 |
+
"## Siradaki: Faz 3 (fork/Triton + tam config + checkpoint/resume). Bkz. sohbet 3a-3d plani."
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
]
|
| 92 |
}
|
| 93 |
],
|