File size: 5,895 Bytes

31dc8dc

Configuration Design Guide
**************************

YAML Configuration Structure
============================

Configuration files use hierarchical YAML structure with three main sections:

.. code-block:: yaml

   model:
     # Model-specific settings
   data:
     # Data loading and preprocessing
   train:
     # Training hyperparameters and setup

Model Configuration
===================

Model Section
-------------

**Required fields:**

.. code-block:: yaml

   model:
     config_path: "./configs/model_configs/[model_name]"
     model_path: "./path/to/model"
     tokenizer_path: "./path/to/tokenizer"
     attn_implementation: "sdpa"        # sdpa|eager|flex_attention
     moe_implementation: "fused"        # fused|standard

.. warning::

   **Flash Attention Limitation**: The flash attention backend can only be used with ``full_attention`` or ``causal_attention`` modes. It **cannot** adapt to custom attention types used in LLaDA2.0 models. **Do not use FlashAttention (flash_attn2/flash_attn) for Block Diffusion Mode models**. See :doc:`/algo/block_diffusion` for detailed explanation of block diffusion training.

Data Configuration
==================

Data Section
------------

**Template for conversation data:**

.. code-block:: yaml

   data:
     train_path: "./datasets/train.jsonl"
     data_type: "conversation"          # conversation|plain|instruction
     datasets_type: "mapping"           # mapping|streaming
     dataloader_type: "native"          # native|custom
     max_seq_len: 2048
     text_keys: "messages"              # field name in JSON
     noise_range_low: 0.3               # diffusion noise lower bound
     noise_range_high: 0.8              # diffusion noise upper bound
     num_workers: 16

- Support multiple data formats (JSONL, Parquet)
- Configurable noise ranges for diffusion training
- Flexible text field mapping
- Worker count based on CPU cores

Training Configuration
======================

Training Section
----------------

**Distributed training setup:**

.. code-block:: yaml

   train:
     output_dir: "./outputs/experiment_name"
     
     # Parallel configuration
     data_parallel_mode: "fsdp2"        # fsdp2
     tensor_parallel_size: 1            # model parallel
     ulysses_parallel_size: 1           # sequence parallel
     expert_parallel_size: 1            # MoE parallel
     
     # Batch configuration
     global_batch_size: 16              # total batch across all GPUs
     micro_batch_size: 1                # batch per GPU
     
     # Training schedule
     num_train_epochs: 1
     save_epochs: 1                     # checkpoint frequency
     log_steps: 1                       # logging frequency

**Optimization parameters:**

.. code-block:: yaml

   train:
     optimizer: "adamw"
     beta1: 0.9
     beta2: 0.999
     lr: 1.0e-5                        # learning rate
     lr_warmup_ratio: 0.03             # warmup steps ratio
     lr_decay_style: "cosine"          # cosine|linear|constant
     weight_decay: 0.1
     max_grad_norm: 1.0

**Memory optimization:**

.. code-block:: yaml

   train:
     enable_mixed_precision: true
     enable_gradient_checkpointing: true
     enable_full_shard: true           # FSDP parameter sharding
     enable_fsdp_offload: true         # CPU offloading
     empty_cache_steps: 500            # GPU memory cleanup

Configuration Patterns
======================

Model Scaling
-------------

**Small model template:**

.. code-block:: yaml

   train:
     global_batch_size: 8               # Reduce for smaller models
     micro_batch_size: 1

**Large model template:**

.. code-block:: yaml

   train:
     global_batch_size: 64              # Increase for larger models
     micro_batch_size: 1
     tensor_parallel_size: 2            # Enable model parallelism
     expert_parallel_size: 2            # Distribute experts

Dataset Adaptation
------------------

**For large datasets:**

.. code-block:: yaml

   data:
     datasets_type: "streaming"        # Memory-efficient loading
     num_workers: 32                   # Increase workers

**For small datasets:**

.. code-block:: yaml

   data:
     datasets_type: "mapping"          # Full dataset in memory
     num_workers: 8                    # Reduce overhead

Hardware Adaptation
===================

Single GPU Setup
----------------

.. code-block:: yaml

   train:
     data_parallel_mode: "fsdp2"
     tensor_parallel_size: 1
     expert_parallel_size: 1
     global_batch_size: 4            # Fit single GPU
     micro_batch_size: 1
     enable_fsdp_offload: false      # Disable offloading

Multi-GPU Setup
---------------

.. code-block:: yaml

   train:
     data_parallel_mode: "fsdp2"
     tensor_parallel_size: 1
     expert_parallel_size: 2         # Distribute experts
     global_batch_size: 32           # Scale with GPU count
     micro_batch_size: 1
     enable_fsdp_offload: false      # Faster training

Memory-Constrained Setup
------------------------

.. code-block:: yaml

   train:
     enable_gradient_checkpointing: true
     enable_full_shard: true
     enable_fsdp_offload: true       # Enable CPU offloading
     enable_activation_offload: true # Reduce GPU memory
     micro_batch_size: 1             # Minimal per-GPU batch

Best Practices
==============

**Path Management**
   Use relative paths for configs
      Store absolute paths in environment variables
      Create separate output directories per experiment

**Parameter Tuning**
   Start with conservative batch sizes
      Increase learning rate for larger batches
      Adjust warmup ratio based on dataset size

**Monitoring**
   Enable W&B for experiment tracking
      Set appropriate logging frequency
      Monitor gradient norms and loss curves

**Reproducibility**
   Fix random seeds in training scripts
      Document configuration changes
      Version control configuration files