longformer-prompt-injection-finetuning

This model is a fine-tuned version of allenai/longformer-base-4096 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.2116
• Accuracy: 0.9938
• Precision: 0.9964
• Recall: 0.9927
• F1: 0.9945
• Roc Auc: 0.9994
• True Negatives: 20995
• False Positives: 98
• False Negatives: 201
• True Positives: 27150

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 64
• eval_batch_size: 128
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 500
• num_epochs: 2
• label_smoothing_factor: 0.1

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Precision	Recall	F1	Roc Auc	True Negatives	False Positives	False Negatives	True Positives
0.2447	0.1166	500	0.2425	0.9779	0.9929	0.9677	0.9801	0.9973	20903	190	883	26468
0.2268	0.2331	1000	0.2371	0.9797	0.9978	0.9661	0.9817	0.9986	21034	59	926	26425
0.2245	0.3497	1500	0.2195	0.9904	0.9920	0.9910	0.9915	0.9992	20874	219	247	27104
0.2204	0.4662	2000	0.2338	0.9825	0.9993	0.9696	0.9842	0.9991	21075	18	831	26520
0.2134	0.5828	2500	0.2169	0.9912	0.9939	0.9905	0.9922	0.9992	20927	166	260	27091
0.2187	0.6993	3000	0.2139	0.9921	0.9969	0.9890	0.9929	0.9994	21009	84	301	27050
0.217	0.8159	3500	0.2132	0.9928	0.9970	0.9902	0.9936	0.9994	21012	81	269	27082
0.2173	0.9324	4000	0.2122	0.9934	0.9960	0.9922	0.9941	0.9994	20985	108	214	27137
0.2143	1.0490	4500	0.2153	0.9917	0.9982	0.9871	0.9926	0.9994	21044	49	352	26999
0.2099	1.1655	5000	0.2123	0.9934	0.9967	0.9915	0.9941	0.9994	21003	90	232	27119
0.2104	1.2821	5500	0.2119	0.9936	0.9965	0.9921	0.9943	0.9994	20998	95	215	27136
0.2143	1.3986	6000	0.2118	0.9937	0.9967	0.9921	0.9944	0.9994	21004	89	217	27134
0.2098	1.5152	6500	0.2116	0.9938	0.9964	0.9927	0.9945	0.9994	20995	98	201	27150
0.2093	1.6317	7000	0.2118	0.9936	0.9966	0.9921	0.9943	0.9994	21000	93	217	27134
0.2133	1.7483	7500	0.2118	0.9937	0.9967	0.9921	0.9944	0.9994	21003	90	217	27134
0.2146	1.8648	8000	0.2118	0.9936	0.9967	0.9920	0.9943	0.9994	21003	90	220	27131
0.2117	1.9814	8500	0.2118	0.9936	0.9967	0.9920	0.9944	0.9994	21004	89	219	27132

Framework versions

• Transformers 4.56.0
• Pytorch 2.8.0+cu129
• Datasets 4.5.0
• Tokenizers 0.22.0