| --- |
| library_name: transformers |
| tags: |
| - generated_from_trainer |
| model-index: |
| - name: lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0 |
| results: [] |
| --- |
| |
| <!-- This model card has been generated automatically according to the information the Trainer had access to. You |
| should probably proofread and complete it, then remove this comment. --> |
|
|
| [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl) |
| <details><summary>See axolotl config</summary> |
|
|
| axolotl version: `0.12.2` |
| ```yaml |
| base_model: /lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-4b |
| |
| datasets: |
| - path: /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking |
| ds_type: json |
| type: chat_template |
| field_messages: conversations |
| data_files: |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0007.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0009.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0005.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0006.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0014.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0010.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0012.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0008.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0001.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0002.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0013.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0015.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0004.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0011.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0000.jsonl |
| - /lustre/fswork/projects/rech/qwv/udv55np/dataset/ift/Nemotron-Super-49B-v1_5/no_thinking/0003.jsonl |
| |
| dataset_prepared_path: /lustre/fswork/projects/rech/dgo/udv55np/dataset_gemma/Nemotron-Super-49B-v1_5/split_0 |
| tokenizer_config: "/lustre/fswork/projects/rech/qwv/udv55np/Gemma/base/gemma-3-27b" |
| chat_template: gemma3 |
| eot_tokens: |
| - "<end_of_turn>" |
| |
| shuffle_merged_datasets: true |
| output_dir: /lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0 |
| |
| sequence_len: 16384 |
| sample_packing: true |
| |
| gradient_accumulation_steps: 1 |
| micro_batch_size: 1 |
| num_epochs: 0.6 |
| auto_resume_from_checkpoints: true |
| |
| optimizer: adamw_torch_fused |
| lr_scheduler: warmup_stable_decay |
| learning_rate: 5e-6 |
| lr_scheduler_kwargs: |
| num_decay_steps: 200 |
| min_lr_ratio: 0.1 |
| warmup_steps: 100 |
| |
| bf16: true |
| tf32: false |
| |
| gradient_checkpointing: true |
| logging_steps: 10 |
| flash_attention: true |
| |
| evals_per_epoch: 0 |
| saves_per_epoch: 1 |
| save_total_limit: 20 |
| save_only_model: true |
| |
| use_tensorboard: true |
| deepspeed: /lustre/fswork/projects/rech/qwv/udv55np/axolotl/zero3.json |
| |
| ``` |
|
|
| </details><br> |
|
|
| # lustre/fswork/projects/rech/dgo/udv55np/ift/Nemotron-Super-49B-v1_5/gemma-3-4b/0 |
| |
| This model was trained from scratch on the None dataset. |
| |
| ## Model description |
| |
| More information needed |
| |
| ## Intended uses & limitations |
| |
| More information needed |
| |
| ## Training and evaluation data |
| |
| More information needed |
| |
| ## Training procedure |
| |
| ### Training hyperparameters |
| |
| The following hyperparameters were used during training: |
| - learning_rate: 5e-06 |
| - train_batch_size: 1 |
| - eval_batch_size: 1 |
| - seed: 42 |
| - distributed_type: multi-GPU |
| - num_devices: 16 |
| - total_train_batch_size: 16 |
| - total_eval_batch_size: 16 |
| - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments |
| - lr_scheduler_type: warmup_stable_decay |
| - lr_scheduler_warmup_steps: 100 |
| - training_steps: 711 |
| |
| ### Training results |
| |
| |
| |
| ### Framework versions |
| |
| - Transformers 4.55.2 |
| - Pytorch 2.6.0+cu124 |
| - Datasets 4.0.0 |
| - Tokenizers 0.21.1 |
| |
