AXOLOTL_NOTES.md

metadata

library_name: transformers
tags:
  - generated_from_trainer
datasets:
  - /home/athuser/modelC_train/sft_modelC.jsonl
model-index:
  - name: models/modelC_out/70B_fft_e1
    results: []

See axolotl config

axolotl version: 0.12.2

# Model C anchor run: Llama-3.1-70B FFT on the assembled keeper set.
# Derived from /models/axolot/llama3_70b_fsdp.yaml (the out_FFT_E precedent);
# dataset swapped to Model C keepers in completion format (full-doc LM loss,
# both speakers, 15% header dropout baked into the jsonl by export_sft.py).
base_model: /models/Llama-3.1-70B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: false
datasets:
  - path: /home/athuser/modelC_train/sft_modelC.jsonl
    type: completion
    field: text
dataset_prepared_path: /home/athuser/modelC_train/last_run_prepared
val_set_size: 0.02
output_dir: /models/modelC_out/70B_fft_e1
sequence_len: 4096
sample_packing: true
tf32: true
gradient_accumulation_steps: 4
micro_batch_size: 1
num_epochs: 1
optimizer: adamw_torch_fused
lr_scheduler: constant_with_warmup
learning_rate: 2.0e-05
bf16: true
resume_from_checkpoint:
logging_steps: 1
flash_attention: true
warmup_ratio: 0.03
evals_per_epoch: 4
saves_per_epoch: 1
save_only_model: true
weight_decay: 0.0
ddp_backend: nccl
fsdp_version: 2
fsdp_config:
  offload_params: false
  cpu_ram_efficient_loading: true
  auto_wrap_policy: TRANSFORMER_BASED_WRAP
  transformer_layer_cls_to_wrap: LlamaDecoderLayer
  state_dict_type: FULL_STATE_DICT
  reshard_after_forward: true
  activation_checkpointing: true
special_tokens:
  pad_token: <|end_of_text|>

models/modelC_out/70B_fft_e1

This model was trained from scratch on the /home/athuser/modelC_train/sft_modelC.jsonl dataset. It achieves the following results on the evaluation set:

• Loss: 1.4970
• Memory/max Mem Active(gib): 89.15
• Memory/max Mem Allocated(gib): 89.15
• Memory/device Mem Reserved(gib): 94.15

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_steps: 3
• training_steps: 113

Training results

Training Loss	Epoch	Step	Validation Loss	Mem Active(gib)	Mem Allocated(gib)	Mem Reserved(gib)
No log	0	0	1.7114	27.73	27.73	31.33
1.53	0.2549	29	1.5095	89.15	89.15	94.15
1.49	0.5099	58	1.5013	89.15	89.15	94.15
1.4778	0.7648	87	1.4970	89.15	89.15	94.15

Framework versions

• Transformers 4.55.2
• Pytorch 2.7.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.2