| ============================================================ |
| CogNet-1B Ultra-Fast Training V2 — MAXIMUM SPEED |
| ============================================================ |
| Device: cuda:0 |
| Distributed: False (world_size=1) |
| Model: 350m |
| BF16: True |
| Compile: False |
| Compile step: False |
| CUDA prefetch: False |
| Seq warmup: False |
| Async checkpoint: False |
| 8-bit optimizer: True |
| TF32 enabled: True |
| HF repo: thefinalboss/CogNet-1B |
| HF token: SET |
| ============================================================ |
| Loaded tokenizer from /root/cognet-1b/tokenizer_v3.json (vocab=136) |
| Skipping data preparation (--skip-data-prep) |
| Loading data from: /root/cognet-1b/data_1b/train_merged.pt |
|
|
| Building CogNet-350M (optimized)... |
| Total parameters: 304,232,960 (0.30B) |
| 8-bit AdamW (bitsandbytes) enabled — 50% less VRAM for optimizer states |
| Mixed precision: BF16 |
|
|
| Starting: step 0 -> 100000 |
| Batch=4 x GradAccum=8 x GPUs=1 = Effective 32 |
| SeqLen=512, LR=1e-05-0.0003 |
| TF32=ON, Gradient checkpointing=True |
| Graceful shutdown: SIGTERM/SIGINT will save checkpoint |
|
|
| [BENCH] Un benchmark de 10 steps va mesurer la vitesse réelle... |
|
|
| ============================================================ |
| BENCHMARK — Mesure des performances réelles |
| ============================================================ |
| Warmup: 3 steps |
| Mesure: 10 steps |
| Config: batch=4, grad_accum=8, seq_len=512 |
| Warmup terminé — début de la mesure... |
|
|
| ╔══════════════════════════════════════════════════════╗ |
| ║ RÉSULTATS DU BENCHMARK ║ |
| ╠══════════════════════════════════════════════════════╣ |
| ║ 0.10 steps/sec (optimizer steps) ║ |
| ║ 1581 tokens/sec ║ |
| ║ 103.62 sec pour 10 steps ║ |
| ║ 3.2 GB VRAM utilisé ║ |
| ╠══════════════════════════════════════════════════════╣ |
| ║ Temps estimé pour 100,000 steps restants ║ |
| ║ ~ 287.8 heures (12.0 jours) ║ |
| ╚══════════════════════════════════════════════════════╝ |
| ============================================================ |
|
|
| Benchmark sauvé: /root/cognet-1b/checkpoints_1b/benchmark_results.json |
| Step 0/100000 | Loss: 3.3116 | PPL: 27.4 | LR: 0.00e+00 | Grad: 2.75 | VRAM: 3.2GB | 1378 tok/s | 0.1 step/s | ETA: 12.0j |
| Step 10/100000 | Loss: 3.2792 | PPL: 26.6 | LR: 1.50e-06 | Grad: 2.48 | VRAM: 3.2GB | 1583 tok/s | 0.1 step/s | ETA: 12.0j |
| Step 20/100000 | Loss: 3.2696 | PPL: 26.3 | LR: 3.00e-06 | Grad: 1.62 | VRAM: 3.2GB | 1585 tok/s | 0.1 step/s | ETA: 12.0j |
| Step 30/100000 | Loss: 3.2555 | PPL: 25.9 | LR: 4.50e-06 | Grad: 0.64 | VRAM: 3.2GB | 1568 tok/s | 0.1 step/s | ETA: 12.0j |
| Step 40/100000 | Loss: 3.2414 | PPL: 25.6 | LR: 6.00e-06 | Grad: 0.81 | VRAM: 3.2GB | 1590 tok/s | 0.1 step/s | ETA: 12.0j |
|
|
