v5: hyperparameter tuning, F1 98.31%

README.md

@@ -14,82 +14,80 @@ model-index:
       name: Named Entity Recognition
     metrics:
     - type: f1
-      value: 0.9806
+      value: 0.9831
       name: F1
     - type: precision
-      value: 0.9782
+      value: 0.9792
       name: Precision
     - type: recall
-      value: 0.9830
+      value: 0.9869
       name: Recall
 ---
 
 # Cybersecurity NER Model
 
-Named Entity Recognition model for cybersecurity domain with **98.06% F1 score**.
+NER model for cybersecurity domain. F1: 98.31%.
 
-## Model Description
+## Model Details
 
-This spaCy 3.x model extracts cybersecurity-specific entities from job descriptions, CVs, threat reports, and security documentation. It uses domain-adapted RoBERTa as the transformer backbone.
-
-**Version:** v4
+**Version:** v5
 **Framework:** spaCy 3.8+
 **Training Date:** 2025-12-29
-**Training Examples:** 1922 (stratified 80/10/10 split)
-
-## Entity Types (13)
-
-| Entity Type | F1 Score | Examples |
-|-------------|----------|----------|
-| CERTIFICATION | 100.00% | CISSP, OSCP, CEH, CISM, Security+ |
-| SECURITY_ROLE | 100.00% | CISO, SOC Analyst, Penetration Tester |
-| TECHNICAL_SKILL | 100.00% | Incident Response, Threat Hunting, Forensics |
-| SECURITY_TOOL | 100.00% | Splunk, Metasploit, Burp Suite, Wireshark |
-| AUDIT_TERM | 100.00% | Compliance, Audit, Assessment |
-| CONTROL_ID | 100.00% | CIS Controls, NIST 800-53 |
-| SECURITY_DOMAIN | 97.83% | Cloud Security, Network Security, AppSec |
-| THREAT_TYPE | 97.44% | APT, ransomware, phishing, malware |
-| FRAMEWORK | 96.97% | NIST CSF, ISO 27001, OWASP, MITRE ATT&CK |
-| ATTACK_TECHNIQUE | 96.30% | SQL Injection, XSS, Buffer Overflow |
-| REGULATION | 87.80% | GDPR, HIPAA, PCI-DSS, SOX |
-| CVE | 85.71% | CVE-2021-44228 (Log4Shell) |
-| ACRONYM | 66.67% | SIEM, EDR, XDR, SOAR |
+**Examples:** 1922 (stratified 80/10/10)
+**Backbone:** Domain-adapted RoBERTa
+
+## Entities (13)
+
+| Entity | F1 | Examples |
+|--------|-----|----------|
+| CERTIFICATION | 100% | CISSP, OSCP, CEH |
+| SECURITY_ROLE | 100% | CISO, SOC Analyst |
+| SECURITY_TOOL | 100% | Splunk, Metasploit |
+| ATTACK_TECHNIQUE | 100% | SQL Injection, XSS |
+| FRAMEWORK | 100% | NIST CSF, ISO 27001 |
+| THREAT_TYPE | 100% | APT, ransomware |
+| AUDIT_TERM | 100% | Compliance, Audit |
+| CVE | 100% | CVE-2021-44228 |
+| SECURITY_DOMAIN | 99.10% | Cloud Security |
+| TECHNICAL_SKILL | 95.30% | Incident Response |
+| REGULATION | 94.44% | GDPR, HIPAA |
+| ACRONYM | 88.89% | SIEM, EDR |
+| CONTROL_ID | 0% | See hybrid approach |
 
 ## Performance
 
-**Overall Metrics:**
-- F1: **98.06%**
-- Precision: 97.82%
-- Recall: 98.30%
-- Inference Speed: ~68ms per document
+**Metrics:**
+- F1: 98.31%
+- Precision: 97.92%
+- Recall: 98.69%
+- Inference: ~60ms/doc
 
-**Improvements over v3:**
-- Overall F1: 69.4% → 98.06% (+28.6 points)
-- SECURITY_DOMAIN: 13% → 97.83% (+84.8 points)
-- Training examples: 1000 → 1922 (+92%)
+**v5 changes from v4:**
+- Tuned hyperparameters (dropout 0.25, L2 0.02)
+- Improved REGULATION (+6.64pp), ACRONYM (+22.22pp)
+- Overall +0.25pp F1
 
-## Training Data
+## CONTROL_ID Handling
 
-- **Source:** Prodigy 1.9.9 annotations (cybersecurity job descriptions, CVs, threat intelligence)
-- **Split:** Stratified by entity type (80% train, 10% dev, 10% test)
-- **Examples:** 1922 total
-- **Transformer:** Domain-adapted RoBERTa (roberta-cybersecurity)
+Model F1 for CONTROL_ID: 0% (insufficient training data: 25 examples).
 
-## Usage
+**Solution:** Hybrid approach - regex extraction for production use.
+
+Patterns: ISO 27001, NIST CSF, CIS Controls, SOC 2, PCI-DSS.
 
-### Installation
+See service implementation for details.
+
+## Usage
 
 ```bash
 pip install spacy>=3.7.0 spacy-transformers>=1.3.0
 ```
 
-### Load Model
-
 ```python
 import spacy
 
 nlp = spacy.load("pki/ner-cybersecurity")
-doc = nlp("CISO with CISSP certification, expert in Splunk and ISO 27001")
+doc = nlp("CISO with CISSP, expert in Splunk and ISO 27001")
 
 for ent in doc.ents:
     print(f"{ent.text:20} | {ent.label_}")
@@ -103,86 +101,45 @@ Splunk               | SECURITY_TOOL
 ISO 27001            | FRAMEWORK
 ```
 
-### FastAPI Service
-
-```python
-from fastapi import FastAPI
-import spacy
-
-app = FastAPI()
-nlp = spacy.load("pki/ner-cybersecurity")
-
-@app.post("/extract")
-async def extract_entities(text: str):
-    doc = nlp(text)
-    return {
-        "entities": [
-            {
-                "text": ent.text,
-                "label": ent.label_,
-                "start": ent.start_char,
-                "end": ent.end_char
-            }
-            for ent in doc.ents
-        ]
-    }
-```
-
 ## Use Cases
 
-- **Job Matching:** Extract skills, certifications, tools from CVs and job descriptions
-- **Threat Intelligence:** Identify attack techniques, threat types, CVEs in reports
-- **Compliance:** Extract regulatory frameworks, controls, audit terms
-- **Security Documentation:** Analyze security policies, procedures, assessments
+- Job/CV matching
+- Threat intelligence extraction
+- Compliance documentation parsing
+- Security policy analysis
 
-## Training Configuration
+## Training Config
 
 ```ini
-[training]
 max_steps = 8000
+dropout = 0.25
+L2 = 0.02
+learning_rate = 0.00003
 hidden_width = 128
-warmup_steps = 500
+maxout_pieces = 3
 batch_size = 128
-
-[transformer]
-name = roberta-cybersecurity (domain-adapted)
 ```
 
 ## Limitations
 
-- **ACRONYM entities:** Lower F1 (66.67%) due to limited training examples (46 total)
-- **Context dependency:** Some terms may be ambiguous without context (e.g., "SOAR" as tool vs role)
-- **Domain specificity:** Optimized for cybersecurity; may underperform on general text
-
-## Model Card Authors
-
-PKI Team
-
-## Citation
-
-```bibtex
-@misc{ner-cybersecurity-v4,
-  title={Cybersecurity NER Model v4},
-  author={PKI Team},
-  year={2025},
-  publisher={Hugging Face},
-  url={https://huggingface.co/pki/ner-cybersecurity}
-}
-```
+- ACRONYM: Lower F1 (88.89%) - limited examples (46)
+- CONTROL_ID: Requires hybrid regex approach
+- Domain-specific: Optimized for cybersecurity text
+- Context-dependent ambiguity on some terms
 
 ## License
 
-MIT License
+MIT
 
 ## Version History
 
-| Version | Date | F1 | Training Examples | Notes |
-|---------|------|-----|-------------------|-------|
-| v4 | 2025-12-29 | 98.06% | 1922 | spaCy 3.x, stratified split, domain RoBERTa |
+| Version | Date | F1 | Examples | Notes |
+|---------|------|-----|----------|-------|
+| v5 | 2025-12-29 | 98.31% | 1922 | Hyperparameter tuning |
+| v4 | 2025-12-29 | 98.06% | 1922 | Stratified split, domain RoBERTa |
 | v3 | 2025-01 | 69.4% | 1000 | spaCy 3.x migration |
-| v2 | 2024-12 | 99.5%* | 1805 | spaCy 2.x (*train accuracy, not F1) |
-| v1 | 2024-11 | N/A | N/A | Initial Prodigy training |
+| v2 | 2024-12 | 99.5%* | 1805 | spaCy 2.x (*train accuracy) |
 
 ## Contact
 
-For issues or questions, open an issue at the model repository.
+Issues: Model repository

config.cfg

@@ -33,7 +33,7 @@ update_with_oracle_cut_size = 100
 state_type = "ner"
 extra_state_tokens = false
 hidden_width = 128
-maxout_pieces = 2
+maxout_pieces = 3
 use_upper = true
 nO = null
 
@@ -88,9 +88,9 @@ dev_corpus = "corpora.dev"
 train_corpus = "corpora.train"
 seed = ${system.seed}
 gpu_allocator = ${system.gpu_allocator}
-dropout = 0.1
+dropout = 0.25
 accumulate_gradient = 3
-patience = 1600
+patience = 2000
 max_epochs = 0
 max_steps = 8000
 eval_frequency = 200
@@ -115,7 +115,7 @@ progress_bar = true
 beta1 = 0.9
 beta2 = 0.999
 L2_is_weight_decay = true
-L2 = 0.01
+L2 = 0.02
 grad_clip = 1.0
 use_averages = false
 eps = 0.00000001
@@ -124,7 +124,7 @@ eps = 0.00000001
 @schedules = "warmup_linear.v1"
 warmup_steps = 500
 total_steps = 8000
-initial_rate = 0.00005
+initial_rate = 0.00003
 
 [training.score_weights]
 ents_f = 1.0

meta.json

@@ -48,9 +48,9 @@
 
   ],
   "performance":{
-    "ents_f":0.9817232376,
-    "ents_p":0.9791666667,
-    "ents_r":0.9842931937,
+    "ents_f":0.9830508475,
+    "ents_p":0.9792207792,
+    "ents_r":0.9869109948,
     "ents_per_type":{
       "SECURITY_ROLE":{
         "p":1.0,
@@ -58,19 +58,19 @@
         "f":1.0
       },
       "SECURITY_TOOL":{
-        "p":0.9803921569,
+        "p":1.0,
         "r":1.0,
-        "f":0.9900990099
+        "f":1.0
       },
       "TECHNICAL_SKILL":{
-        "p":0.9333333333,
-        "r":0.9589041096,
-        "f":0.9459459459
+        "p":0.9342105263,
+        "r":0.9726027397,
+        "f":0.9530201342
       },
       "ATTACK_TECHNIQUE":{
-        "p":0.9411764706,
+        "p":1.0,
         "r":1.0,
-        "f":0.9696969697
+        "f":1.0
       },
       "FRAMEWORK":{
         "p":1.0,
@@ -88,9 +88,9 @@
         "f":1.0
       },
       "REGULATION":{
-        "p":0.9444444444,
+        "p":0.8947368421,
         "r":1.0,
-        "f":0.9714285714
+        "f":0.9444444444
       },
       "THREAT_TYPE":{
         "p":1.0,
@@ -98,9 +98,9 @@
         "f":1.0
       },
       "ACRONYM":{
-        "p":1.0,
+        "p":0.8,
         "r":1.0,
-        "f":1.0
+        "f":0.8888888889
       },
       "AUDIT_TERM":{
         "p":1.0,
@@ -118,7 +118,7 @@
         "f":1.0
       }
     },
-    "transformer_loss":100.855706131,
-    "ner_loss":153.2495727539
+    "transformer_loss":29.5171592474,
+    "ner_loss":20.9466311453
   }
 }

ner/model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cadbd57865a39db9e24519f868fa31dbf660371d19577b0f1695c9f6e12fd16f
-size 819891
+oid sha256:1d6aa3567ecbde04a9c944d91a90fab7d2e0561e228c667e66a3482fcefcfa94
+size 1018547

transformer/model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5beca8bcf2dd500d9ab6a64c86501ce9f8ea274a0e93b5e93ce1b2a3585b391a
-size 503478689
+oid sha256:09c07fd5b912da2969f0cecb91b599432839cd1b3faa7b80877203f886e82928
+size 503478228