See axolotl config

axolotl version: 0.4.1

adapter: lora
base_model: Qwen/Qwen2.5-0.5B-Instruct
bf16: true
chat_template: llama3
dataset_prepared_path: null
datasets:
- data_files:
  - 9f7cef554ae67229_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/9f7cef554ae67229_train_data.json
  type:
    field_instruction: question
    field_output: answer
    format: '{instruction}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
debug: null
device_map:
  ? ''
  : 0,1,2,3,4,5,6,7
early_stopping_patience: 2
eval_max_new_tokens: 128
eval_steps: 400
eval_table_size: null
flash_attention: true
gradient_accumulation_steps: 4
gradient_checkpointing: true
group_by_length: false
hub_model_id: Alphatao/f61daa15-27ab-432d-8a1f-cef37f7ca975
hub_repo: null
hub_strategy: null
hub_token: null
learning_rate: 0.0002
load_best_model_at_end: true
load_in_4bit: false
load_in_8bit: false
local_rank: null
logging_steps: 1
lora_alpha: 32
lora_dropout: 0.05
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 16
lora_target_linear: true
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: 33844
micro_batch_size: 2
mlflow_experiment_name: /tmp/9f7cef554ae67229_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 10
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 400
sequence_len: 1024
strict: false
tf32: true
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.023880026745629956
wandb_entity: null
wandb_mode: online
wandb_name: 365fa35d-c7f9-424f-8597-3e48c6b82259
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: 365fa35d-c7f9-424f-8597-3e48c6b82259
warmup_steps: 10
weight_decay: 0.0
xformers_attention: null

f61daa15-27ab-432d-8a1f-cef37f7ca975

This model is a fine-tuned version of Qwen/Qwen2.5-0.5B-Instruct on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.9669

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• training_steps: 33844

Training results

Training Loss	Epoch	Step	Validation Loss
2.8631	0.0000	1	2.7133
2.1212	0.0157	400	2.2637
2.4396	0.0313	800	2.2257
2.2118	0.0470	1200	2.2011
2.3099	0.0626	1600	2.1843
2.1729	0.0783	2000	2.1701
2.1017	0.0939	2400	2.1586
2.1134	0.1096	2800	2.1490
1.8189	0.1253	3200	2.1397
2.106	0.1409	3600	2.1322
2.1147	0.1566	4000	2.1263
2.1506	0.1722	4400	2.1210
1.8483	0.1879	4800	2.1136
2.3869	0.2035	5200	2.1105
2.083	0.2192	5600	2.1060
2.1662	0.2349	6000	2.0994
2.1826	0.2505	6400	2.0965
2.4128	0.2662	6800	2.0897
2.115	0.2818	7200	2.0881
2.0297	0.2975	7600	2.0848
1.9269	0.3131	8000	2.0819
2.182	0.3288	8400	2.0772
1.9379	0.3445	8800	2.0741
2.0634	0.3601	9200	2.0706
2.0554	0.3758	9600	2.0690
1.942	0.3914	10000	2.0644
2.2	0.4071	10400	2.0615
2.0826	0.4227	10800	2.0587
1.9221	0.4384	11200	2.0565
2.2434	0.4541	11600	2.0536
1.906	0.4697	12000	2.0493
1.943	0.4854	12400	2.0472
1.9929	0.5010	12800	2.0449
1.9886	0.5167	13200	2.0406
1.919	0.5323	13600	2.0387
1.8248	0.5480	14000	2.0358
2.2062	0.5637	14400	2.0326
1.9969	0.5793	14800	2.0306
2.0402	0.5950	15200	2.0284
2.1432	0.6106	15600	2.0247
1.8055	0.6263	16000	2.0221
2.2405	0.6419	16400	2.0197
1.957	0.6576	16800	2.0162
2.2378	0.6733	17200	2.0152
2.0769	0.6889	17600	2.0120
1.9884	0.7046	18000	2.0102
2.4296	0.7202	18400	2.0078
2.01	0.7359	18800	2.0057
1.8161	0.7515	19200	2.0032
1.88	0.7672	19600	2.0013
2.0602	0.7829	20000	1.9988
2.0311	0.7985	20400	1.9964
2.0131	0.8142	20800	1.9947
2.1369	0.8298	21200	1.9926
1.9735	0.8455	21600	1.9906
1.8679	0.8611	22000	1.9895
2.1485	0.8768	22400	1.9869
1.8585	0.8925	22800	1.9852
1.7405	0.9081	23200	1.9834
1.9779	0.9238	23600	1.9817
2.1238	0.9394	24000	1.9802
1.8955	0.9551	24400	1.9785
1.9514	0.9707	24800	1.9770
1.9089	0.9864	25200	1.9756
1.587	1.0021	25600	1.9750
1.769	1.0177	26000	1.9749
1.9681	1.0334	26400	1.9739
1.9145	1.0490	26800	1.9732
2.3405	1.0647	27200	1.9726
1.9518	1.0803	27600	1.9718
2.1134	1.0960	28000	1.9708
2.5173	1.1117	28400	1.9701
1.7659	1.1273	28800	1.9695
1.6429	1.1430	29200	1.9690
1.9997	1.1586	29600	1.9687
2.0883	1.1743	30000	1.9683
2.1096	1.1899	30400	1.9680
1.6548	1.2056	30800	1.9676
2.1148	1.2213	31200	1.9674
1.9167	1.2369	31600	1.9672
1.9936	1.2526	32000	1.9671
1.205	1.2682	32400	1.9670
2.0367	1.2839	32800	1.9669
2.2282	1.2995	33200	1.9668
1.7997	1.3152	33600	1.9669

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1