See axolotl config

axolotl version: 0.4.1

adapter: lora
base_model: fxmarty/really-tiny-falcon-testing
bf16: true
chat_template: llama3
dataset_prepared_path: null
datasets:
- data_files:
  - 5ae2214d09bade90_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/5ae2214d09bade90_train_data.json
  type:
    field_instruction: instruction
    field_output: response_8b_instruct
    format: '{instruction}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
debug: null
deepspeed: null
early_stopping_patience: 4
eval_max_new_tokens: 128
eval_steps: 150
eval_table_size: null
flash_attention: false
fp16: null
fsdp: null
fsdp_config: null
gradient_accumulation_steps: 4
gradient_checkpointing: true
group_by_length: false
hub_model_id: Romain-XV/3ac98d3d-31bb-400f-8a65-b7106fa7aed7
hub_repo: null
hub_strategy: checkpoint
hub_token: null
learning_rate: 0.0002
load_best_model_at_end: true
load_in_4bit: false
load_in_8bit: false
local_rank: null
logging_steps: 1
lora_alpha: 128
lora_dropout: 0.3
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 64
lora_target_linear: true
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: 8280
micro_batch_size: 2
mlflow_experiment_name: /tmp/5ae2214d09bade90_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 3
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 150
sequence_len: 2048
strict: false
tf32: true
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.05
wandb_entity: null
wandb_mode: online
wandb_name: bc26fc02-1572-48e9-862a-5c81c59f02fb
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: bc26fc02-1572-48e9-862a-5c81c59f02fb
warmup_steps: 10
weight_decay: 0.0
xformers_attention: null

3ac98d3d-31bb-400f-8a65-b7106fa7aed7

This model is a fine-tuned version of fxmarty/really-tiny-falcon-testing on the None dataset. It achieves the following results on the evaluation set:

• Loss: 10.9581

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• training_steps: 8280

Training results

Training Loss	Epoch	Step	Validation Loss
44.358	0.0001	1	11.0863
43.9823	0.0154	150	10.9989
43.9935	0.0307	300	10.9914
43.9457	0.0461	450	10.9850
43.9403	0.0614	600	10.9816
43.964	0.0768	750	10.9788
43.9364	0.0922	900	10.9766
43.9244	0.1075	1050	10.9746
43.9331	0.1229	1200	10.9731
43.9167	0.1382	1350	10.9723
43.8159	0.1536	1500	10.9716
43.9406	0.1690	1650	10.9697
43.891	0.1843	1800	10.9688
43.9027	0.1997	1950	10.9671
43.8942	0.2150	2100	10.9666
43.9174	0.2304	2250	10.9653
43.9068	0.2458	2400	10.9650
43.8447	0.2611	2550	10.9638
43.8899	0.2765	2700	10.9636
43.8989	0.2918	2850	10.9636
43.889	0.3072	3000	10.9626
43.9361	0.3226	3150	10.9621
43.8727	0.3379	3300	10.9620
43.9111	0.3533	3450	10.9618
43.9133	0.3686	3600	10.9619
43.9016	0.3840	3750	10.9610
43.9287	0.3994	3900	10.9608
43.8578	0.4147	4050	10.9610
43.8534	0.4301	4200	10.9603
43.8479	0.4454	4350	10.9601
43.877	0.4608	4500	10.9598
43.8836	0.4762	4650	10.9597
43.8674	0.4915	4800	10.9597
43.864	0.5069	4950	10.9593
43.8571	0.5222	5100	10.9592
43.9053	0.5376	5250	10.9591
43.8718	0.5530	5400	10.9587
43.9331	0.5683	5550	10.9588
43.8305	0.5837	5700	10.9590
43.8585	0.5990	5850	10.9584
43.8123	0.6144	6000	10.9585
43.8731	0.6298	6150	10.9585
43.9659	0.6451	6300	10.9583
43.8471	0.6605	6450	10.9582
43.8636	0.6758	6600	10.9585
43.8495	0.6912	6750	10.9582
43.8449	0.7066	6900	10.9583
43.8689	0.7219	7050	10.9582
43.8391	0.7373	7200	10.9581
43.877	0.7526	7350	10.9581
43.9319	0.7680	7500	10.9580
43.8892	0.7834	7650	10.9581
43.8751	0.7987	7800	10.9581
43.9211	0.8141	7950	10.9581
43.8906	0.8295	8100	10.9581

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1