| --- |
| license: apache-2.0 |
| --- |
| |
| # Internal Training: Carvalho-Qwen (Test) |
|
|
| This document outlines the configuration, hyperparameters, and environment setup for CPT the **Qwen3-14B** model using the . |
|
|
| ## 📋 Model & Tokenizer Details |
|
|
| | Component | Value | |
| | :--- | :--- | |
| | **Base Model** | `Qwen/Qwen3-14B` | |
| | **Tokenizer** | `Qwen/Qwen3-14B` | |
| | **Precision** | `bfloat16` (BF16) | |
| | **Trust Remote Code** | Enabled | |
|
|
| ## ⚙️ Environment & Infrastructure |
|
|
| - **Training Script:** `run_clm_instruct_fa3.py` |
| - **Virtual Environment:** `venvs/cpt` |
| - **Optimization Kernels:** |
| - **Flash Attention 2:** Enabled). Flash Attention 3 not work well in this moment at Citius H200! |
| - **Liger Kernel:** Enabled |
| - **DeepSpeed Config:** None |
| - **Launcher:** `accelerate launch` (Deepspeed not works well). |
|
|
| ## 📊 Training Hyperparameters |
|
|
| | Parameter | Value | Notes | |
| | :--- | :--- | :--- | |
| | **Context Length** | `2048` | `--block_size` | |
| | **Epochs** | `1` | | |
| | **Learning Rate** | `2e-6` | `--lr_scheduler_type cosine` | |
| | **Weight Decay** | `0.1` | | |
| | **Optimizer** | AdamW | $\beta_1=0.9, \beta_2=0.999, \epsilon=1e-8$ | |
| | **Batch Size (Train)** | `4` | Per device | |
| | **Grad Accumulation** | `8` | Effective Batch Size = $N_{gpus} \times 4 \times 8$ | |
| | **Batch Size (Eval)** | `2` | Per device | |
| | **Eval Samples** | `20,000` | Max evaluation samples | |
| | **Seed** | `42` | | |
| |
| ## 💾 Data & Checkpointing |
| |
| - **Dataloader:** `CPT_CarballoSalamandra_dataloader.py` |
| - **Output Directory:** `output/Carvalho-Qwen_Test_[DATE]` |
| - **Saving Strategy:** |
| - Saves every **300 steps**. |
| - Keeps only the last **1** checkpoint (`--save_total_limit 1`). |
| - **Logging:** Tensorboard (logs every 50 steps). |
| |
| ## ⚠️ Critical Implementation Notes |
| |
| ### 1. Memory Management |
| - `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is set to handle memory fragmentation. |
| - `--use_cache False` is set during training to save VRAM. |
| - `dataloader_pin_memory` is set to `True`. |
|
|
| ### 2. NCCL Handling |
| `TORCH_NCCL_ASYNC_ERROR_HANDLING=1` is enabled to force crashes on NCCL hangs (e.g., hanging broadcasts) rather than stalling indefinitely. |
