llama-cot-o1

This model is a fine-tuned version of meta-llama/Llama-3.2-3b-instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.6532

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 32
• optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
0.7259	0.2168	500	0.7316
0.6867	0.4336	1000	0.6930
0.6642	0.6504	1500	0.6759
0.6496	0.8672	2000	0.6659
0.6102	1.0837	2500	0.6615
0.6107	1.3005	3000	0.6574
0.6105	1.5173	3500	0.6546
0.5929	1.7341	4000	0.6529
0.5987	1.9509	4500	0.6519
0.5904	2.1674	5000	0.6533
0.5793	2.3842	5500	0.6532
0.5826	2.6010	6000	0.6532
0.5903	2.8178	6500	0.6532

Framework versions

• Transformers 4.47.1
• Pytorch 2.5.1
• Datasets 3.2.0
• Tokenizers 0.21.0