File size: 2,289 Bytes
1fc0157 0ae4e58 47dd2d0 1fc0157 0ae4e58 1fc0157 47dd2d0 1fc0157 47dd2d0 1fc0157 0ae4e58 47dd2d0 0ae4e58 47dd2d0 0ae4e58 47dd2d0 0ae4e58 47dd2d0 0ae4e58 47dd2d0 0ae4e58 47dd2d0 0ae4e58 47dd2d0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | ---
license: mit
base_model: BAAI/bge-m3
base_model_relation: finetune
library_name: sentence-transformers
pipeline_tag: feature-extraction
language:
- en
- vi
pretty_name: sec-embedding (fine-tuned BGE-M3)
tags:
- sentence-transformers
- feature-extraction
- embedding
- dense-retrieval
- contrastive-learning
- cve
- cybersecurity
- qdrant
- secAI
---
# sec-embedding
**This is a fine-tuned version of [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3)** for CVE / cybersecurity dense retrieval.
It was trained on a **CVE investigation-trajectory dataset** with **hard-negative mining** from a **local Qdrant** collection (`cve_kb`, NVD/MITRE core chunks). It is not a raw copy of the base checkpoint.
## Training
From `notebooks/BGE_M3_Colab.ipynb`:
| | |
|---|---|
| Base | `BAAI/bge-m3` via Unsloth `FastSentenceTransformer` (`unsloth/bge-m3`) |
| Role | Bi-encoder / **dense retriever** (1024-d, same geometry as bge-m3) |
| Adapter | LoRA, `r=32`, modules `key`, `query`, `value`, `dense` |
| Loss | `CachedMultipleNegativesRankingLoss` (InfoNCE, in-batch hard negatives) |
| Engine | `sentence-transformers` `SentenceTransformerTrainer` |
| Max sequence length | 1024 |
| Learning rate | 2e-5, bf16 |
Each example is a `(query, positive)` pair:
- **Query** — CVE investigation trajectory (Vietnamese or English) over CVE-ID, CWE, product, severity, year, CAPEC / ATT&CK, filled from real KB metadata.
- **Positive** — matching CVE passage from local Qdrant `cve_kb`.
- **Hard negatives** — other CVE documents in the same mini-batch, all mined from that Qdrant index (near-miss CVEs: similar wording, wrong ID).
Dataset source field: `Qdrant cve_kb (NVD/MITRE)`. Split: 40k train / 5k validation.
## Usage
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("DuyTa/sec-embedding")
query_emb = model.encode("CVE-2021-44228 impact on log4j", normalize_embeddings=True)
doc_emb = model.encode(passage, normalize_embeddings=True)
```
Rebuild the Qdrant index with **this** checkpoint. Mixing vectors with vanilla `BAAI/bge-m3` drops recall.
## Attribution & license
Derived from [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) (**MIT License**). Credit for the base model belongs to the BAAI authors.
|