Hugging Face's logo

hurtmongoose
/
tinyllama-lora-malicious-classifier

PEFT
Safetensors
Transformers
lora
Model card Files
xet
Community
tinyllama-lora-malicious-classifier / README.md
hurtmongoose's picture
hurtmongoose
TinyLLaMA v1.1 LoRA fine-tuned for 3-class malicious prompt detection
4cd178e verified
preview code
|
raw
history blame contribute delete
13.2 kB
metadata 
library_name: peft
license: apache-2.0
base_model: TinyLlama/TinyLlama_v1.1
tags:
  - base_model:adapter:TinyLlama/TinyLlama_v1.1
  - lora
  - transformers
metrics:
  - accuracy
model-index:
  - name: tinyllama-lora-malicious-classifier
    results: []

tinyllama-lora-malicious-classifier

This model is a fine-tuned version of TinyLlama/TinyLlama_v1.1 on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.4833
  • Accuracy: 0.8289
  • Precision Weighted: 0.8220
  • Recall Weighted: 0.8289
  • F1 Weighted: 0.8239
  • Mcc: 0.7203
  • Balanced Accuracy: 0.7724
  • Macro Fnr: 0.2276
  • Macro Fpr: 0.0897
  • Macro Specificity: 0.9103
  • Per Class: {'jailbreaking': {'TP': 228, 'FP': 103, 'FN': 161, 'TN': 1437, 'FNR': 0.4138817480719794, 'FPR': 0.06688311688311688, 'Specificity': 0.9331168831168831}, 'prompt injection': {'TP': 439, 'FP': 112, 'FN': 131, 'TN': 1247, 'FNR': 0.22982456140350876, 'FPR': 0.08241353936718175, 'Specificity': 0.9175864606328182}, 'unharmful': {'TP': 932, 'FP': 115, 'FN': 38, 'TN': 844, 'FNR': 0.03917525773195876, 'FPR': 0.11991657977059438, 'Specificity': 0.8800834202294057}}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 15
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Accuracy Precision Weighted Recall Weighted F1 Weighted Mcc Balanced Accuracy Macro Fnr Macro Fpr Macro Specificity Per Class
1.0323 1.0 1107 1.0327 0.5702 0.5595 0.5702 0.5636 0.2924 0.5112 0.4888 0.2364 0.7636 {'jailbreaking': {'TP': 113, 'FP': 209, 'FN': 276, 'TN': 1331, 'FNR': 0.7095115681233933, 'FPR': 0.1357142857142857, 'Specificity': 0.8642857142857143}, 'prompt injection': {'TP': 312, 'FP': 238, 'FN': 258, 'TN': 1121, 'FNR': 0.45263157894736844, 'FPR': 0.1751287711552612, 'Specificity': 0.8248712288447387}, 'unharmful': {'TP': 675, 'FP': 382, 'FN': 295, 'TN': 577, 'FNR': 0.30412371134020616, 'FPR': 0.3983315954118874, 'Specificity': 0.6016684045881127}}
0.7496 2.0 2214 0.7073 0.7252 0.7112 0.7252 0.7144 0.5463 0.6529 0.3471 0.1502 0.8498 {'jailbreaking': {'TP': 154, 'FP': 132, 'FN': 235, 'TN': 1408, 'FNR': 0.6041131105398457, 'FPR': 0.08571428571428572, 'Specificity': 0.9142857142857143}, 'prompt injection': {'TP': 386, 'FP': 163, 'FN': 184, 'TN': 1196, 'FNR': 0.32280701754385965, 'FPR': 0.1199411331861663, 'Specificity': 0.8800588668138337}, 'unharmful': {'TP': 859, 'FP': 235, 'FN': 111, 'TN': 724, 'FNR': 0.11443298969072165, 'FPR': 0.24504692387904067, 'Specificity': 0.7549530761209593}}
0.5695 3.0 3321 0.6096 0.7766 0.7636 0.7766 0.7654 0.6323 0.7031 0.2969 0.1215 0.8785 {'jailbreaking': {'TP': 171, 'FP': 106, 'FN': 218, 'TN': 1434, 'FNR': 0.5604113110539846, 'FPR': 0.06883116883116883, 'Specificity': 0.9311688311688312}, 'prompt injection': {'TP': 417, 'FP': 141, 'FN': 153, 'TN': 1218, 'FNR': 0.26842105263157895, 'FPR': 0.10375275938189846, 'Specificity': 0.8962472406181016}, 'unharmful': {'TP': 910, 'FP': 184, 'FN': 60, 'TN': 775, 'FNR': 0.061855670103092786, 'FPR': 0.19186652763295098, 'Specificity': 0.808133472367049}}
0.5059 4.0 4428 0.5686 0.7926 0.7817 0.7926 0.7843 0.6596 0.7245 0.2755 0.1107 0.8893 {'jailbreaking': {'TP': 191, 'FP': 114, 'FN': 198, 'TN': 1426, 'FNR': 0.5089974293059126, 'FPR': 0.07402597402597402, 'Specificity': 0.925974025974026}, 'prompt injection': {'TP': 419, 'FP': 131, 'FN': 151, 'TN': 1228, 'FNR': 0.2649122807017544, 'FPR': 0.0963944076526858, 'Specificity': 0.9036055923473142}, 'unharmful': {'TP': 919, 'FP': 155, 'FN': 51, 'TN': 804, 'FNR': 0.05257731958762887, 'FPR': 0.1616266944734098, 'Specificity': 0.8383733055265902}}
0.4583 5.0 5535 0.5413 0.8056 0.7953 0.8056 0.7977 0.6812 0.7389 0.2611 0.1032 0.8968 {'jailbreaking': {'TP': 199, 'FP': 109, 'FN': 190, 'TN': 1431, 'FNR': 0.4884318766066838, 'FPR': 0.07077922077922078, 'Specificity': 0.9292207792207792}, 'prompt injection': {'TP': 426, 'FP': 126, 'FN': 144, 'TN': 1233, 'FNR': 0.25263157894736843, 'FPR': 0.09271523178807947, 'Specificity': 0.9072847682119205}, 'unharmful': {'TP': 929, 'FP': 140, 'FN': 41, 'TN': 819, 'FNR': 0.042268041237113405, 'FPR': 0.145985401459854, 'Specificity': 0.8540145985401459}}
0.4763 6.0 6642 0.5243 0.8113 0.8028 0.8113 0.8052 0.6910 0.7498 0.2502 0.0994 0.9006 {'jailbreaking': {'TP': 212, 'FP': 113, 'FN': 177, 'TN': 1427, 'FNR': 0.455012853470437, 'FPR': 0.07337662337662337, 'Specificity': 0.9266233766233766}, 'prompt injection': {'TP': 428, 'FP': 120, 'FN': 142, 'TN': 1239, 'FNR': 0.24912280701754386, 'FPR': 0.08830022075055188, 'Specificity': 0.9116997792494481}, 'unharmful': {'TP': 925, 'FP': 131, 'FN': 45, 'TN': 828, 'FNR': 0.04639175257731959, 'FPR': 0.13660062565172054, 'Specificity': 0.8633993743482794}}
0.4283 7.0 7749 0.5095 0.8170 0.8083 0.8170 0.8104 0.7003 0.7546 0.2454 0.0969 0.9031 {'jailbreaking': {'TP': 213, 'FP': 105, 'FN': 176, 'TN': 1435, 'FNR': 0.4524421593830334, 'FPR': 0.06818181818181818, 'Specificity': 0.9318181818181818}, 'prompt injection': {'TP': 430, 'FP': 118, 'FN': 140, 'TN': 1241, 'FNR': 0.24561403508771928, 'FPR': 0.08682855040470934, 'Specificity': 0.9131714495952906}, 'unharmful': {'TP': 933, 'FP': 130, 'FN': 37, 'TN': 829, 'FNR': 0.03814432989690722, 'FPR': 0.13555787278415016, 'Specificity': 0.8644421272158499}}
0.4119 8.0 8856 0.5033 0.8191 0.8116 0.8191 0.8135 0.7041 0.7592 0.2408 0.0951 0.9049 {'jailbreaking': {'TP': 223, 'FP': 118, 'FN': 166, 'TN': 1422, 'FNR': 0.4267352185089974, 'FPR': 0.07662337662337662, 'Specificity': 0.9233766233766234}, 'prompt injection': {'TP': 422, 'FP': 105, 'FN': 148, 'TN': 1254, 'FNR': 0.2596491228070175, 'FPR': 0.0772626931567329, 'Specificity': 0.9227373068432672}, 'unharmful': {'TP': 935, 'FP': 126, 'FN': 35, 'TN': 833, 'FNR': 0.03608247422680412, 'FPR': 0.13138686131386862, 'Specificity': 0.8686131386861314}}
0.412 9.0 9963 0.4955 0.8253 0.8178 0.8253 0.8199 0.7143 0.7671 0.2329 0.0916 0.9084 {'jailbreaking': {'TP': 222, 'FP': 103, 'FN': 167, 'TN': 1437, 'FNR': 0.42930591259640105, 'FPR': 0.06688311688311688, 'Specificity': 0.9331168831168831}, 'prompt injection': {'TP': 440, 'FP': 118, 'FN': 130, 'TN': 1241, 'FNR': 0.22807017543859648, 'FPR': 0.08682855040470934, 'Specificity': 0.9131714495952906}, 'unharmful': {'TP': 930, 'FP': 116, 'FN': 40, 'TN': 843, 'FNR': 0.041237113402061855, 'FPR': 0.12095933263816476, 'Specificity': 0.8790406673618353}}
0.496 10.0 11070 0.4926 0.8289 0.8214 0.8289 0.8232 0.7202 0.7703 0.2297 0.0903 0.9097 {'jailbreaking': {'TP': 224, 'FP': 98, 'FN': 165, 'TN': 1442, 'FNR': 0.4241645244215938, 'FPR': 0.06363636363636363, 'Specificity': 0.9363636363636364}, 'prompt injection': {'TP': 439, 'FP': 113, 'FN': 131, 'TN': 1246, 'FNR': 0.22982456140350876, 'FPR': 0.08314937454010302, 'Specificity': 0.9168506254598969}, 'unharmful': {'TP': 936, 'FP': 119, 'FN': 34, 'TN': 840, 'FNR': 0.03505154639175258, 'FPR': 0.12408759124087591, 'Specificity': 0.8759124087591241}}
0.428 11.0 12177 0.4890 0.8258 0.8191 0.8258 0.8207 0.7154 0.7677 0.2323 0.0913 0.9087 {'jailbreaking': {'TP': 230, 'FP': 117, 'FN': 159, 'TN': 1423, 'FNR': 0.4087403598971722, 'FPR': 0.07597402597402597, 'Specificity': 0.924025974025974}, 'prompt injection': {'TP': 424, 'FP': 99, 'FN': 146, 'TN': 1260, 'FNR': 0.256140350877193, 'FPR': 0.0728476821192053, 'Specificity': 0.9271523178807947}, 'unharmful': {'TP': 939, 'FP': 120, 'FN': 31, 'TN': 839, 'FNR': 0.031958762886597936, 'FPR': 0.1251303441084463, 'Specificity': 0.8748696558915537}}
0.4103 12.0 13284 0.4866 0.8269 0.8191 0.8269 0.8206 0.7166 0.7669 0.2331 0.0922 0.9078 {'jailbreaking': {'TP': 221, 'FP': 99, 'FN': 168, 'TN': 1441, 'FNR': 0.4318766066838046, 'FPR': 0.06428571428571428, 'Specificity': 0.9357142857142857}, 'prompt injection': {'TP': 437, 'FP': 107, 'FN': 133, 'TN': 1252, 'FNR': 0.23333333333333334, 'FPR': 0.07873436350257543, 'Specificity': 0.9212656364974245}, 'unharmful': {'TP': 937, 'FP': 128, 'FN': 33, 'TN': 831, 'FNR': 0.03402061855670103, 'FPR': 0.1334723670490094, 'Specificity': 0.8665276329509907}}
0.4009 13.0 14391 0.4833 0.8289 0.8220 0.8289 0.8239 0.7203 0.7724 0.2276 0.0897 0.9103 {'jailbreaking': {'TP': 228, 'FP': 103, 'FN': 161, 'TN': 1437, 'FNR': 0.4138817480719794, 'FPR': 0.06688311688311688, 'Specificity': 0.9331168831168831}, 'prompt injection': {'TP': 439, 'FP': 112, 'FN': 131, 'TN': 1247, 'FNR': 0.22982456140350876, 'FPR': 0.08241353936718175, 'Specificity': 0.9175864606328182}, 'unharmful': {'TP': 932, 'FP': 115, 'FN': 38, 'TN': 844, 'FNR': 0.03917525773195876, 'FPR': 0.11991657977059438, 'Specificity': 0.8800834202294057}}
0.4242 14.0 15498 0.4834 0.8284 0.8211 0.8284 0.8228 0.7193 0.7700 0.2300 0.0906 0.9094 {'jailbreaking': {'TP': 226, 'FP': 105, 'FN': 163, 'TN': 1435, 'FNR': 0.4190231362467866, 'FPR': 0.06818181818181818, 'Specificity': 0.9318181818181818}, 'prompt injection': {'TP': 435, 'FP': 104, 'FN': 135, 'TN': 1255, 'FNR': 0.23684210526315788, 'FPR': 0.07652685798381163, 'Specificity': 0.9234731420161884}, 'unharmful': {'TP': 937, 'FP': 122, 'FN': 33, 'TN': 837, 'FNR': 0.03402061855670103, 'FPR': 0.12721584984358708, 'Specificity': 0.872784150156413}}
0.3859 15.0 16605 0.4829 0.8269 0.8197 0.8269 0.8215 0.7168 0.7692 0.2308 0.0912 0.9088 {'jailbreaking': {'TP': 226, 'FP': 106, 'FN': 163, 'TN': 1434, 'FNR': 0.4190231362467866, 'FPR': 0.06883116883116883, 'Specificity': 0.9311688311688312}, 'prompt injection': {'TP': 436, 'FP': 107, 'FN': 134, 'TN': 1252, 'FNR': 0.23508771929824562, 'FPR': 0.07873436350257543, 'Specificity': 0.9212656364974245}, 'unharmful': {'TP': 933, 'FP': 121, 'FN': 37, 'TN': 838, 'FNR': 0.03814432989690722, 'FPR': 0.1261730969760167, 'Specificity': 0.8738269030239834}}

Framework versions

  • PEFT 0.17.1
  • Transformers 4.53.3
  • Pytorch 2.6.0+cu124
  • Datasets 4.3.0
  • Tokenizers 0.21.4