--- library_name: transformers license: apache-2.0 base_model: AiForgeMaster/Qwen3-4B-P3-TC-1 tags: - axolotl - generated_from_trainer datasets: - AiForgeMaster/glaiceai-natural-reasoning-10k model-index: - name: Qwen3-4B-P3-TC-RSSFT-1 results: [] --- [Built with Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config axolotl version: `0.13.0.dev0` ```yaml # axolotl train config.yaml # Prevent NCCL timeout ddp_timeout: 7200 # 2 hours timeout instead of 10 minutes # Load model from local models directory first, fallback to HuggingFace if not found base_model: AiForgeMaster/Qwen3-4B-P3-TC-1 # Local path - will fallback to Qwen/Qwen3-4B if not found locally # Automatically upload checkpoint and final model to HF hub_model_id: AiForgeMaster/Qwen3-4B-P3-TC-RSSFT-1 load_in_8bit: false load_in_4bit: false strict: false # SFT dataset configuration - using HuggingFace datasets datasets: - path: AiForgeMaster/glaiceai-natural-reasoning-10k # Private HF dataset - requires API key type: alpaca_chat.load_qa # skip: 0 # number of rows of data to skip over from the beginning # Local paths relative to working directory dataset_prepared_path: ./data/prepared val_set_size: 0.0 # Set to 0 for SFT (no validation split) output_dir: ./outputs # Cache directories for HuggingFace downloads (relative to working dir) # This ensures models and datasets are downloaded to local directories hf_use_auth_token: true # Use HF token for private repos if needed sequence_len: 8192 sample_packing: false # Standard for SFT eval_sample_packing: false # Disable for SFT # WandB configuration - fill in your details wandb_project: ngpt-cpt wandb_entity: null wandb_watch: gradients wandb_name: qwen3_4b_p3_tc_rssft_1 wandb_log_model: end # Batch size configuration (total effective batch size = micro_batch_size * gradient_accumulation_steps * num_gpus) # For batch size 8-16: micro_batch_size=2, gradient_accumulation_steps=4 gives effective batch size of 8 per GPU gradient_accumulation_steps: 4 micro_batch_size: 8 # Adjust based on your GPU memory optimizer: adamw_torch_fused lr_scheduler: cosine learning_rate: 2e-5 # Good learning rate for SFT bf16: auto tf32: true max_grad_norm: 1.0 gradient_checkpointing: true gradient_checkpointing_kwargs: use_reentrant: false logging_steps: 10 # Log every 10 steps flash_attention: true warmup_steps: 150 # Good warmup for SFT # Checkpoint saving configuration - save every 50 steps save_steps: 50 save_strategy: steps save_total_limit: 5 # Keep only 5 most recent checkpoints save_only_model: false # Save full checkpoint including optimizer state # Evaluation configuration removed for pure SFT (val_set_size: 0.0) # eval_steps: 2000 # Not supported when val_set_size == 0 # eval_strategy: steps # Not supported when val_set_size == 0 weight_decay: 0.01 # Good weight decay for SFT # Liger optimizations for memory efficiency and speed plugins: - axolotl.integrations.liger.LigerPlugin liger_rope: true liger_rms_norm: true liger_glu_activation: true liger_layer_norm: true liger_fused_linear_cross_entropy: true # Additional SFT optimizations # Enable for first run to validate checkpoint saving works save_first_step: true # Memory optimizations dataloader_pin_memory: true dataloader_num_workers: 4 remove_unused_columns: true # Advanced training settings for SFT # Calculate max_steps for full epoch: dataset_size / (micro_batch_size * gradient_accumulation_steps * num_gpus) # max_steps: 175 # Set for one full epoch with your dataset size num_epochs: 1 group_by_length: true # Good for SFT efficiency train_on_inputs: true # train on user inputs in SFT # Loss monitoring loss_watchdog_threshold: 10.0 # Stop if loss exceeds this value loss_watchdog_patience: 3 # Garbage collection to manage memory gc_steps: 100 # Run garbage collection every 100 steps ```

[Visualize in Weights & Biases](https://wandb.ai/uskfoundation/ngpt-cpt/runs/azltguw6) # Qwen3-4B-P3-TC-RSSFT-1 This model is a fine-tuned version of [AiForgeMaster/Qwen3-4B-P3-TC-1](https://huggingface.co/AiForgeMaster/Qwen3-4B-P3-TC-1) on the AiForgeMaster/glaiceai-natural-reasoning-10k dataset. ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 32 - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 150 - training_steps: 312 ### Training results ### Framework versions - Transformers 4.55.4 - Pytorch 2.7.1+cu126 - Datasets 4.0.0 - Tokenizers 0.21.4