Update README.md

README.md

@@ -5,30 +5,47 @@ language:
 tags:
 - cybersecurity
 widget:
-- text: "Native API functions such as <mask>, may be directed invoked via system calls/syscalls, but these features are also often exposed to user-mode applications via interfaces and libraries.."
+- text: >-
+    Native API functions such as <mask> may be directly invoked via system
+    calls (syscalls). However, these features are also commonly exposed to
+    user-mode applications through interfaces and libraries.
   example_title: Native API functions
-  
-- text: "One way of explicitly assigning the PPID of a new process is via the <mask> API call, which supports a parameter that defines the PPID to use."
+- text: >-
+    One way to explicitly assign the PPID of a new process is through the
+    <mask> API call, which includes a parameter for defining the PPID.
   example_title: Assigning the PPID of a new process
-  
-- text: "Enable Safe DLL Search Mode to force search for system DLLs in directories with greater restrictions (e.g. %<mask>%) to be used before local directory DLLs (e.g. a user's home directory)"
+- text: >-
+    Enable Safe DLL Search Mode to ensure that system DLLs in more restricted
+    directories (e.g., %<mask>%) are prioritized over DLLs in less secure
+    locations such as a user’s home directory.
   example_title: Enable Safe DLL Search Mode
-
-- text: "GuLoader is a file downloader that has been used since at least December 2019 to distribute a variety of <mask>, including NETWIRE, Agent Tesla, NanoCore, and FormBook."
+- text: >-
+    GuLoader is a file downloader that has been active since at least December
+    2019. It has been used to distribute a variety of <mask>, including
+    NETWIRE, Agent Tesla, NanoCore, and FormBook.
   example_title: GuLoader is a file downloader
 ---
-# SecureBERT+
-This model represents an improved version of the [SecureBERT](https://huggingface.co/ehsanaghaei/SecureBERT) model, trained on a corpus eight times larger than its predecessor, leveraging the computational power of 8xA100 GPUs. This version, known as SecureBERT+, brings forth an average improvment of 9% in the performance of the Masked Language Model (MLM) task. This advancement signifies a substantial stride towards achieving heightened proficiency in language understanding and representation learning within the cybersecurity domain.
 
+# SecureBERT+  
+
+**SecureBERT+** is an enhanced version of [SecureBERT](https://huggingface.co/ehsanaghaei/SecureBERT), trained on a corpus **eight times larger** than its predecessor and leveraging the computational power of **8×A100 GPUs**.  
 
-SecureBERT is a domain-specific language model based on RoBERTa which is trained on a huge amount of cybersecurity data and fine-tuned/tweaked to understand/represent cybersecurity textual data.
+This model delivers an **average 9% improvement** in Masked Language Modeling (MLM) performance compared to SecureBERT, representing a significant advancement in language understanding and representation within the cybersecurity domain.  
 
-## Dataset
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/6340b0bd77fd972573eb2f9b/pO-v6961YI1D0IPcm0027.png)
+---
 
-## Load Model
-SecureBER+T has been uploaded to [Huggingface](https://huggingface.co/ehsanaghaei/SecureBERT_Plus) framework. 
+## Dataset  
+SecureBERT+ was trained on a large-scale corpus of cybersecurity-related text, substantially expanding the coverage and depth of the original SecureBERT training data.  
 
+![dataset](https://cdn-uploads.huggingface.co/production/uploads/6340b0bd77fd972573eb2f9b/pO-v6961YI1D0IPcm0027.png)  
+
+---
+
+## Using SecureBERT+  
+
+SecureBERT+ is available on the [Hugging Face Hub](https://huggingface.co/ehsanaghaei/SecureBERT_Plus).  
+
+### Load the Model
 ```python
 from transformers import RobertaTokenizer, RobertaModel
 import torch
@@ -42,76 +59,61 @@ outputs = model(**inputs)
 last_hidden_states = outputs.last_hidden_state
 ```
 
-## Fill Mask (MLM)
-Use the code below to predict the masked word within the given sentences:
+# Masked Language Modeling Example
 
+Use the code below to predict masked words in text:
 ```python
-#!pip install transformers
-#!pip install torch
-#!pip install tokenizers
+#!pip install transformers torch tokenizers
 
 import torch
 import transformers
-from transformers import RobertaTokenizer, RobertaTokenizerFast
+from transformers import RobertaTokenizerFast
 
 tokenizer = RobertaTokenizerFast.from_pretrained("ehsanaghaei/SecureBERT_Plus")
 model = transformers.RobertaForMaskedLM.from_pretrained("ehsanaghaei/SecureBERT_Plus")
 
-def predict_mask(sent, tokenizer, model, topk =10, print_results = True):
+def predict_mask(sent, tokenizer, model, topk=10, print_results=True):
     token_ids = tokenizer.encode(sent, return_tensors='pt')
-    masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero()
-    masked_pos = [mask.item() for mask in masked_position]
+    masked_pos = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero().tolist()
     words = []
+
     with torch.no_grad():
         output = model(token_ids)
 
-    last_hidden_state = output[0].squeeze()
-
-    list_of_list = []
-    for index, mask_index in enumerate(masked_pos):
-        mask_hidden_state = last_hidden_state[mask_index]
-        idx = torch.topk(mask_hidden_state, k=topk, dim=0)[1]
-        words = [tokenizer.decode(i.item()).strip() for i in idx]
-        words = [w.replace(' ','') for w in words]
-        list_of_list.append(words)
+    for pos in masked_pos:
+        logits = output.logits[0, pos]
+        top_tokens = torch.topk(logits, k=topk).indices
+        predictions = [tokenizer.decode(i).strip().replace(" ", "") for i in top_tokens]
+        words.append(predictions)
         if print_results:
-            print("Mask ", "Predictions: ", words)
-
-    best_guess = ""
-    for j in list_of_list:
-        best_guess = best_guess + "," + j[0]
+            print(f"Mask Predictions: {predictions}")
 
     return words
+```
 
+# Limitations & Risks
 
-while True:
-    sent = input("Text here: \t")
-    print("SecureBERT: ")
-    predict_mask(sent, tokenizer, model)
-     
-    print("===========================\n")
-```
+Domain-Specific Scope: SecureBERT+ is optimized for cybersecurity text and may not generalize as well to unrelated domains.
 
-Other model variants:
+Bias in Training Data: The training corpus was collected from online sources and may contain biases, outdated knowledge, or inaccuracies.
 
-[SecureGPT](https://huggingface.co/ehsanaghaei/SecureGPT)
+Potential Misuse: While designed for defensive research, the model could be misapplied to generate adversarial content or obfuscate malicious behavior.
 
-[SecureDeBERTa](https://huggingface.co/ehsanaghaei/SecureDeBERTa)
+Resource-Intensive: The larger dataset and model training process require significant compute resources, which may limit reproducibility for smaller research teams.
 
-[SecureBERT](https://huggingface.co/ehsanaghaei/SecureBERT)
+Evolving Threats: The cybersecurity landscape evolves rapidly. Without regular retraining, the model may not capture emerging threats or terminology.
 
+Users should apply SecureBERT+ responsibly, with appropriate oversight from cybersecurity professionals.
 
 # Reference
+```
 @inproceedings{aghaei2023securebert, 
 title={SecureBERT: A Domain-Specific Language Model for Cybersecurity}, 
 author={Aghaei, Ehsan and Niu, Xi and Shadid, Waseem and Al-Shaer, Ehab}, 
 booktitle={Security and Privacy in Communication Networks: 
-18th EAI International Conference, SecureComm 2022, Virtual Event, 
-October 2022, 
-Proceedings}, 
+18th EAI International Conference, SecureComm 2022, Virtual Event, October 2022, Proceedings}, 
 pages={39--56}, 
 year={2023}, 
-organization={Springer} }
-
-
-
+organization={Springer} 
+}
+```