See axolotl config

axolotl version: 0.4.1

adapter: lora
base_model: llamafactory/tiny-random-Llama-3
bf16: true
chat_template: llama3
dataset_prepared_path: null
datasets:
- data_files:
  - 2adc5188ef0ef527_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/2adc5188ef0ef527_train_data.json
  type:
    field_instruction: instruction
    field_output: response
    format: '{instruction}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
debug: null
device_map:
  ? ''
  : 0,1,2,3,4,5,6,7
early_stopping_patience: 2
eval_max_new_tokens: 128
eval_steps: 100
eval_table_size: null
flash_attention: true
gradient_accumulation_steps: 8
gradient_checkpointing: true
group_by_length: false
hub_model_id: Alphatao/b8acd031-33f9-4334-9b31-270a30e4eb99
hub_repo: null
hub_strategy: null
hub_token: null
learning_rate: 0.0002
load_best_model_at_end: true
load_in_4bit: false
load_in_8bit: false
local_rank: null
logging_steps: 1
lora_alpha: 128
lora_dropout: 0.1
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 64
lora_target_linear: true
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: 26182
micro_batch_size: 4
mlflow_experiment_name: /tmp/2adc5188ef0ef527_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 10
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 100
sequence_len: 2048
special_tokens:
  pad_token: <|eot_id|>
strict: false
tf32: true
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.025150905432595575
wandb_entity: null
wandb_mode: online
wandb_name: 83553881-21e8-49f7-a888-7b7abf0a2dc7
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: 83553881-21e8-49f7-a888-7b7abf0a2dc7
warmup_steps: 10
weight_decay: 0.0
xformers_attention: null

b8acd031-33f9-4334-9b31-270a30e4eb99

This model is a fine-tuned version of llamafactory/tiny-random-Llama-3 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 11.6949

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• training_steps: 26182

Training results

Training Loss	Epoch	Step	Validation Loss
11.7651	0.0002	1	11.7644
11.7195	0.0165	100	11.7201
11.7115	0.0330	200	11.7146
11.7126	0.0495	300	11.7121
11.7125	0.0660	400	11.7101
11.7125	0.0826	500	11.7087
11.7075	0.0991	600	11.7074
11.7117	0.1156	700	11.7065
11.7077	0.1321	800	11.7057
11.7061	0.1486	900	11.7051
11.7086	0.1651	1000	11.7045
11.7043	0.1816	1100	11.7038
11.7098	0.1981	1200	11.7036
11.7043	0.2147	1300	11.7030
11.7057	0.2312	1400	11.7026
11.7091	0.2477	1500	11.7024
11.707	0.2642	1600	11.7022
11.703	0.2807	1700	11.7019
11.7108	0.2972	1800	11.7017
11.7009	0.3137	1900	11.7013
11.7027	0.3302	2000	11.7014
11.7014	0.3467	2100	11.7012
11.7023	0.3633	2200	11.7009
11.7025	0.3798	2300	11.7007
11.7061	0.3963	2400	11.7005
11.7025	0.4128	2500	11.7002
11.7011	0.4293	2600	11.7002
11.7033	0.4458	2700	11.6997
11.7065	0.4623	2800	11.6993
11.7036	0.4788	2900	11.6989
11.7025	0.4954	3000	11.6986
11.7017	0.5119	3100	11.6982
11.705	0.5284	3200	11.6978
11.7034	0.5449	3300	11.6974
11.7029	0.5614	3400	11.6971
11.7006	0.5779	3500	11.6969
11.7052	0.5944	3600	11.6967
11.7072	0.6109	3700	11.6964
11.7049	0.6275	3800	11.6965
11.6989	0.6440	3900	11.6962
11.6993	0.6605	4000	11.6962
11.7034	0.6770	4100	11.6960
11.7001	0.6935	4200	11.6959
11.7044	0.7100	4300	11.6958
11.6986	0.7265	4400	11.6958
11.6951	0.7430	4500	11.6959
11.7006	0.7595	4600	11.6957
11.7004	0.7761	4700	11.6954
11.7028	0.7926	4800	11.6954
11.6983	0.8091	4900	11.6953
11.6995	0.8256	5000	11.6954
11.6993	0.8421	5100	11.6951
11.6984	0.8586	5200	11.6950
11.6969	0.8751	5300	11.6951
11.6967	0.8916	5400	11.6948
11.7004	0.9082	5500	11.6950
11.6987	0.9247	5600	11.6949

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1