sizhkhy/relianceV2

llm3br256

This model is a fine-tuned version of meta-llama/Llama-3.2-3B-Instruct on the relianceV2 dataset. It achieves the following results on the evaluation set:

• Loss: 0.0118

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 5.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.0395	0.2424	5	0.0433
0.0324	0.4848	10	0.0300
0.024	0.7273	15	0.0244
0.0189	0.9697	20	0.0212
0.0171	1.2303	25	0.0190
0.0146	1.4727	30	0.0173
0.0144	1.7152	35	0.0161
0.0104	1.9576	40	0.0155
0.0143	2.2182	45	0.0152
0.0117	2.4606	50	0.0141
0.015	2.7030	55	0.0136
0.0092	2.9455	60	0.0131
0.008	3.2061	65	0.0127
0.0109	3.4485	70	0.0125
0.0085	3.6909	75	0.0122
0.0089	3.9333	80	0.0120
0.0074	4.1939	85	0.0118
0.0074	4.4364	90	0.0118
0.0066	4.6788	95	0.0118
0.0065	4.9212	100	0.0118

Framework versions

• PEFT 0.12.0
• Transformers 4.46.1
• Pytorch 2.4.0+cu121
• Datasets 3.1.0
• Tokenizers 0.20.3