See axolotl config

axolotl version: 0.4.1

adapter: lora
base_model: fxmarty/really-tiny-falcon-testing
bf16: true
chat_template: llama3
dataset_prepared_path: null
datasets:
- data_files:
  - b966613efaebc6e3_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/b966613efaebc6e3_train_data.json
  type:
    field_input: function_description_en
    field_instruction: system_message_en
    field_output: system_message_vi
    format: '{instruction} {input}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
debug: null
device_map:
  ? ''
  : 0,1,2,3,4,5,6,7
early_stopping_patience: 2
eval_max_new_tokens: 128
eval_steps: 100
eval_table_size: null
flash_attention: false
gradient_accumulation_steps: 8
gradient_checkpointing: true
group_by_length: false
hub_model_id: Alphatao/cbfc15e1-e5a2-4bd8-9140-93f930745dc7
hub_repo: null
hub_strategy: null
hub_token: null
learning_rate: 0.0002
load_best_model_at_end: true
load_in_4bit: false
load_in_8bit: false
local_rank: null
logging_steps: 1
lora_alpha: 128
lora_dropout: 0.3
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 64
lora_target_linear: true
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: 4679
micro_batch_size: 4
mlflow_experiment_name: /tmp/b966613efaebc6e3_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 2
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 100
sequence_len: 2048
strict: false
tf32: true
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.04470352621414777
wandb_entity: null
wandb_mode: online
wandb_name: 440bdef0-8dd8-4343-b1a5-4d04eef39827
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: 440bdef0-8dd8-4343-b1a5-4d04eef39827
warmup_steps: 10
weight_decay: 0.0
xformers_attention: null

cbfc15e1-e5a2-4bd8-9140-93f930745dc7

This model is a fine-tuned version of fxmarty/really-tiny-falcon-testing on the None dataset. It achieves the following results on the evaluation set:

• Loss: 10.4560

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• training_steps: 4679

Training results

Training Loss	Epoch	Step	Validation Loss
88.8221	0.0003	1	11.1028
85.35	0.0299	100	10.6278
84.7472	0.0599	200	10.5407
84.4548	0.0898	300	10.5079
84.3611	0.1198	400	10.4914
84.3492	0.1497	500	10.4824
84.2882	0.1797	600	10.4753
84.2623	0.2096	700	10.4719
84.1812	0.2396	800	10.4687
84.1556	0.2695	900	10.4661
84.1486	0.2995	1000	10.4649
84.2001	0.3294	1100	10.4634
84.1679	0.3594	1200	10.4625
84.0587	0.3893	1300	10.4616
84.0846	0.4193	1400	10.4610
84.1087	0.4492	1500	10.4605
84.0865	0.4792	1600	10.4600
84.0844	0.5091	1700	10.4590
84.0703	0.5391	1800	10.4587
84.0528	0.5690	1900	10.4583
84.0046	0.5990	2000	10.4579
84.0247	0.6289	2100	10.4579
84.0073	0.6589	2200	10.4577
84.0011	0.6888	2300	10.4569
83.991	0.7188	2400	10.4569
83.9995	0.7487	2500	10.4566
83.9825	0.7787	2600	10.4569
83.9898	0.8086	2700	10.4563
84.0104	0.8386	2800	10.4564
83.9988	0.8685	2900	10.4562
83.9964	0.8985	3000	10.4561
83.9621	0.9284	3100	10.4560
83.9572	0.9584	3200	10.4560
83.9733	0.9883	3300	10.4559
83.9418	1.0183	3400	10.4560
83.983	1.0482	3500	10.4560

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1