Update README.md

README.md

@@ -52,60 +52,12 @@ Sample output:
 
 ## Training Data
 
-- **Source:** curated CTI corpus derived from the `dataset_20251024_1.jsonl` snapshot and published on the Hub as [`juanmcristobal/ner-ioc-dataset3`](https://huggingface.co/datasets/juanmcristobal/ner-ioc-dataset3) (earlier iterations remain available under `secureModernBert2`).
 - **Size:** 502,726 labelled text spans before filtering; 22 distinct entity classes in BIO format.
 - **Label distribution (spans):** `ORG` (~198k), `PRODUCT` (~79k), `MALWARE` (~67k), `PLATFORM` (~57k), `THREAT-ACTOR` (~49k), `SERVICE` (~46k), `CVE` (~41k), `LOC` (~38k), `SECTOR` (~34k), `TOOL` (~29k), plus indicator types such as `URL`, `IPV4`, `SHA256`, `MD5`, and `REGISTRY-KEYS`.
 - **Pre-processing:** JSONL articles were tokenised and converted to BIO tags; spans in conflict were resolved manually and via automated heuristics before upload.
 
 ## Label Mapping
 
-```
- 0 -> B-URL
- 1 -> I-URL
- 2 -> O
- 3 -> B-ORG
- 4 -> B-SERVICE
- 5 -> I-ORG
- 6 -> B-SECTOR
- 7 -> I-SECTOR
- 8 -> B-FILEPATH
- 9 -> I-FILEPATH
-10 -> I-DOMAIN
-11 -> B-PLATFORM
-12 -> I-SERVICE
-13 -> I-PLATFORM
-14 -> B-THREAT-ACTOR
-15 -> I-THREAT-ACTOR
-16 -> B-PRODUCT
-17 -> B-MALWARE
-18 -> I-MALWARE
-19 -> B-LOC
-20 -> B-CVE
-21 -> I-CVE
-22 -> B-TOOL
-23 -> I-PRODUCT
-24 -> B-IPV4
-25 -> I-IPV4
-26 -> B-MITRE-TACTIC
-27 -> I-MITRE-TACTIC
-28 -> B-DOMAIN
-29 -> I-TOOL
-30 -> B-MD5
-31 -> I-LOC
-32 -> B-CAMPAIGN
-33 -> I-CAMPAIGN
-34 -> B-SHA1
-35 -> B-SHA256
-36 -> B-EMAIL
-37 -> I-EMAIL
-38 -> B-IPV6
-39 -> I-IPV6
-40 -> B-REGISTRY-KEYS
-41 -> I-REGISTRY-KEYS
-```
-
-The `B-` prefix marks the first token of an entity span, while `I-` marks subsequent tokens within the same span. The base labels are described below.
-
 | Label | Description | Example mention |
 |-------|-------------|-----------------|
 | URL | Web address or obfuscated link used in campaigns. | `hxxp://185.222.202.55` |
@@ -133,9 +85,9 @@ The `B-` prefix marks the first token of an entity span, while `I-` marks subseq
 
 ## Training Procedure
 
-- **Project:** `autotrain-sec4` running inside a Hugging Face AutoTrain Space (local hardware mode).
 - **Base model:** [`answerdotai/ModernBERT-large`](https://huggingface.co/answerdotai/ModernBERT-large).
 - **Dataset configuration:** training and validation splits pulled from `juanmcristobal/ner-ioc-dataset3` with column mapping `tokens` → tokens, `tags` → labels.
+- **Hardware:** single Nvidia L40S instance (8 vCPU / 62 GB RAM / 48 GB VRAM).
 - **Optimisation setup:** mixed precision `fp16`, optimiser `adamw_torch`, cosine learning-rate scheduler, gradient accumulation `1`.
 - **Key hyperparameters:** learning rate `5e-5`, batch size `128`, epochs `5`, maximum sequence length `128`.
 - **Checkpoint:** best-performing checkpoint automatically pushed to the Hub as `juanmcristobal/autotrain-sec4`.
@@ -153,7 +105,7 @@ The `B-` prefix marks the first token of an entity span, while `I-` marks subseq
 
 ## Evaluation
 
-Validation metrics reported by AutoTrain on the held-out split:
+AutoTrain reports the following micro-averaged metrics on its validation split (seqeval entity scoring):
 
 | Metric     | Score  |
 |------------|--------|
@@ -162,8 +114,97 @@ Validation metrics reported by AutoTrain on the held-out split:
 | F1         | 0.8476 |
 | Accuracy   | 0.9589 |
 
+An independent re-evaluation against a consolidated CTI set (same taxonomy as this model) produced the label-level accuracy breakdown below. These scores are macro-averaged across labels and therefore are not numerically comparable to the micro metrics above, but they provide insight into class balance and span quality.
+
+| Label | Used | Accuracy |
+|-------|------|----------|
+| CAMPAIGN | 1,817 | 0.7980 |
+| CVE | 28,293 | 0.9995 |
+| DOMAIN | 12,182 | 0.8878 |
+| EMAIL | 731 | 0.8495 |
+| FILEPATH | 13,889 | 0.7957 |
+| IPV4 | 1,164 | 0.9631 |
+| IPV6 | 563 | 0.7425 |
+| LOC | 7,915 | 0.9557 |
+| MALWARE | 10,405 | 0.9087 |
+| MD5 | 389 | 0.9100 |
+| MITRE-TACTIC | 2,181 | 0.7093 |
+| ORG | 36,324 | 0.9301 |
+| PLATFORM | 8,036 | 0.8977 |
+| PRODUCT | 18,720 | 0.8432 |
+| REGISTRY-KEYS | 1,589 | 0.8490 |
+| SECTOR | 6,453 | 0.8309 |
+| SERVICE | 8,533 | 0.8179 |
+| SHA1 | 222 | 0.9189 |
+| SHA256 | 2,146 | 0.9874 |
+| THREAT-ACTOR | 9,532 | 0.9418 |
+| TOOL | 4,874 | 0.7895 |
+| URL | 7,470 | 0.9801 |
+
+- **Macro accuracy:** 0.8776
+
+Because micro vs macro averaging and dataset composition differ, expect numerical gaps between the two evaluations even though both describe the same checkpoint.
+
 These metrics were computed with the `seqeval` micro-average at the entity level.
 
+## External Benchmarks
+
+The following tables report detailed results on a shared CTI validation set. **Do not compare the per-label values across models directly:** each checkpoint uses a different taxonomy or remapping strategy, so accuracy percentages can be misleading when labels are aligned or collapsed differently. Use the per-model tables to understand performance within a single schema, and interpret macro-accuracy scores with caution.
+
+### PranavaKailash/CyNER-2.0-DeBERTa-v3-base
+
+| Label | Used | Accuracy |
+|-------|------|----------|
+| Indicator | 35,936 | 0.7878 |
+| Location | 7,895 | 0.0113 |
+| Malware | 12,125 | 0.7800 |
+| O | 2,896 | 0.7652 |
+| Organization | 42,537 | 0.6556 |
+| System | 35,063 | 0.7259 |
+| TOOL | 4,820 | 0.0000 |
+| Threat Group | 9,522 | 0.0000 |
+| Vulnerability | 27,673 | 0.1876 |
+
+- **Macro accuracy:** 0.4348
+
+### CyberPeace-Institute/SecureBERT-NER
+
+| Label | Used | Accuracy |
+|-------|------|----------|
+| ACT | 3,945 | 0.1706 |
+| APT | 9,518 | 0.5331 |
+| DOM | 10,694 | 0.0196 |
+| EMAIL | 731 | 0.0000 |
+| FILE | 31,864 | 0.0747 |
+| IP | 1,251 | 0.0088 |
+| LOC | 7,895 | 0.8711 |
+| MAL | 10,341 | 0.6076 |
+| MD5 | 354 | 0.8672 |
+| O | 16,275 | 0.4700 |
+| OS | 7,974 | 0.6598 |
+| SECTEAM | 36,083 | 0.3509 |
+| SHA1 | 191 | 0.0209 |
+| SHA2 | 1,647 | 0.9709 |
+| TOOL | 4,816 | 0.4043 |
+| URL | 6,997 | 0.0795 |
+| VULID | 27,586 | 0.3849 |
+
+- **Macro accuracy:** not reported (schema differs substantially from the others).
+
+### cisco-ai/SecureBERT2.0-NER
+
+| Label | Used | Accuracy |
+|-------|------|----------|
+| Indicator | 35,789 | 0.8854 |
+| Malware | 16,926 | 0.6204 |
+| O | 10,786 | 0.6813 |
+| Organization | 51,993 | 0.5579 |
+| System | 34,955 | 0.6600 |
+| Vulnerability | 27,525 | 0.2552 |
+
+- **Macro accuracy:** 0.6100
+
+
 ## Responsible Use
 
 - Confirm entity detections before acting on indicators (e.g., automated blocking).