ajsal27/phi3-chatbot

phi-3-mini-QLoRA

This model is a fine-tuned version of microsoft/Phi-3-mini-4k-instruct on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.1546

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
2.2359	0.1416	50	1.9291
1.5215	0.2833	100	1.2761
1.287	0.4249	150	1.2268
1.2613	0.5666	200	1.2073
1.2341	0.7082	250	1.1996
1.2037	0.8499	300	1.1953
1.2117	0.9915	350	1.1871
1.2023	1.1331	400	1.1813
1.1635	1.2748	450	1.1770
1.1689	1.4164	500	1.1732
1.2013	1.5581	550	1.1720
1.1853	1.6997	600	1.1675
1.1933	1.8414	650	1.1648
1.1774	1.9830	700	1.1619
1.1633	2.1246	750	1.1626
1.1756	2.2663	800	1.1593
1.1597	2.4079	850	1.1587
1.1599	2.5496	900	1.1562
1.1145	2.6912	950	1.1567
1.162	2.8329	1000	1.1553
1.1507	2.9745	1050	1.1546

Framework versions

• PEFT 0.12.0
• Transformers 4.43.3
• Pytorch 2.2.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1