RAG for Cybersecurity: Augmenting LLMs with 85 Specialized Datasets

Author: Ayi NEDJIMI — Senior Offensive Cybersecurity & AI Consultant

Why RAG for Cybersecurity?

Fine-tuned models are powerful, but they have inherent limitations:

• Knowledge cutoff: They don't know about CVEs published after training
• Hallucination risk: They may generate plausible but incorrect security advice
• Source attribution: Users need to verify advice against authoritative sources
• Coverage gaps: No single model can memorize all 85 datasets perfectly

RAG (Retrieval-Augmented Generation) solves these issues by retrieving relevant context from a knowledge base before generating responses.

Architecture

Our RAG system combines fine-tuned cybersecurity LLMs with a vector database of 85 specialized datasets:

Components

1. Embedding Model: sentence-transformers/all-MiniLM-L6-v2

   • 384-dimensional embeddings
   • Fast inference on CPU
   • Good multilingual support (FR/EN)

2. Vector Store: FAISS (Facebook AI Similarity Search)

   • In-memory for fast retrieval
   • Cosine similarity search
   • Scales to millions of documents

3. Knowledge Base: 85 cybersecurity datasets

   • ~100,000+ instruction-response pairs
   • Bilingual (French + English)
   • Covering compliance, offensive, defensive, cloud, AI security

4. LLM: Fine-tuned Qwen 2.5 models

   • ISO27001-Expert-1.5B
   • RGPD-Expert-1.5B
   • CyberSec-Assistant-3B

Pipeline

User Query
    │
    ▼
[Embedding] → sentence-transformers
    │
    ▼
[Retrieval] → FAISS top-k (k=3)
    │
    ▼
[Context Injection] → System prompt + retrieved docs
    │
    ▼
[Generation] → Fine-tuned LLM with streaming
    │
    ▼
Response + Sources

Implementation

1. Building the Vector Index

from sentence_transformers import SentenceTransformer
from datasets import load_dataset
import numpy as np

# Load embedding model
embedder = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')

# Load all 85 datasets
documents = []
dataset_ids = [
    "AYI-NEDJIMI/iso27001",
    "AYI-NEDJIMI/rgpd-fr",
    "AYI-NEDJIMI/gdpr-en",
    "AYI-NEDJIMI/mitre-attack-fr",
    "AYI-NEDJIMI/owasp-top10-fr",
    "AYI-NEDJIMI/nis2-directive-fr",
    # ... 79 more datasets
]

for ds_id in dataset_ids:
    ds = load_dataset(ds_id, split="train")
    for item in ds:
        text = f"{item.get('instruction', '')}\n{item.get('output', '')}"
        documents.append({
            "text": text[:1000],
            "source": ds_id.split("/")[-1],
        })

# Create embeddings
texts = [doc["text"] for doc in documents]
embeddings = embedder.encode(texts, show_progress_bar=True)

2. Retrieval Function

def retrieve(query: str, top_k: int = 3) -> list[dict]:
    query_emb = embedder.encode([query])[0]
    similarities = np.dot(embeddings, query_emb)
    top_indices = np.argsort(similarities)[::-1][:top_k]

    results = []
    for idx in top_indices:
        results.append({
            "text": documents[idx]["text"],
            "source": documents[idx]["source"],
            "score": float(similarities[idx]),
        })
    return results

3. RAG-Enhanced Generation

def generate_with_rag(query, model, tokenizer):
    # Retrieve relevant context
    docs = retrieve(query, top_k=3)

    # Build enhanced prompt
    context = "\n".join([
        f"[Source: {d['source']}] {d['text'][:300]}"
        for d in docs
    ])

    messages = [
        {"role": "system", "content": f"You are a cybersecurity expert. "
         f"Use the following context to answer accurately:\n{context}"},
        {"role": "user", "content": query}
    ]

    # Generate with streaming
    input_text = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    # ... generate response

Results: With vs Without RAG

Example: "What are the GDPR notification requirements for data breaches?"

Without RAG:

"Under GDPR, data breaches must be notified to the supervisory authority within 72 hours. Data subjects must also be informed if there is a high risk to their rights."

With RAG (context from rgpd-fr + gdpr-en datasets):

"Under GDPR Article 33, controllers must notify the competent supervisory authority within 72 hours of becoming aware of a personal data breach, unless the breach is unlikely to result in a risk to rights and freedoms.

The notification must include:

1. Nature of the breach (categories and approximate number of data subjects)
2. Name and contact details of the DPO
3. Description of likely consequences
4. Description of measures taken or proposed

Per Article 34, data subjects must be informed without undue delay when the breach is likely to result in a high risk to their rights and freedoms. Exceptions apply when:

• Appropriate technical measures were in place (e.g., encryption)
• Subsequent measures ensure high risk is no longer likely
• Individual communication would involve disproportionate effort

[Sources: gdpr-en (score: 0.89), rgpd-fr (score: 0.85)]"

Key Improvements with RAG:

• More specific: Article numbers, exact procedures
• More complete: Exceptions, notification content
• Source attribution: Users can verify the information
• Fewer hallucinations: Grounded in authoritative data

Performance Metrics

Challenges & Solutions

Challenge 1: Multilingual Retrieval

Problem: French queries retrieving English documents (and vice versa)
Solution: all-MiniLM-L6-v2 has cross-lingual capabilities. French queries naturally retrieve both FR and EN documents, which is actually beneficial for comprehensive coverage.

Challenge 2: Memory on Free Spaces

Problem: Loading 85 datasets + embedding model + LLM exceeds 16GB RAM
Solution: Load only top 6-10 most relevant datasets based on the selected model. ISO27001-Expert loads ISO/compliance datasets; CyberSec-Assistant loads broader datasets.

Challenge 3: Retrieval Quality

Problem: Generic embeddings sometimes retrieve tangentially related content
Solution: Chunk documents at the instruction-response level (not arbitrary text chunks). Each "document" is a complete Q&A pair, ensuring semantic coherence.

Try It Live

The RAG-enhanced demo is available at: CyberSec Models Demo

Features:

• Toggle RAG on/off to compare results
• See retrieved sources with relevance scores
• Streaming responses (token-by-token)
• Compare all 3 models side-by-side

What's Next

1. Hybrid retrieval: Combine semantic search (embeddings) with keyword search (BM25)
2. Re-ranking: Use a cross-encoder to re-rank retrieved documents
3. Dynamic dataset loading: Load relevant datasets based on query topic
4. Fine-tuned embeddings: Train domain-specific embeddings on cybersecurity text
5. Knowledge graph: Link entities across datasets (CVEs ↔ techniques ↔ mitigations)

Links

• Interactive Demo: CyberSec Models Demo
• RAG Chat Space: CyberSec-Chat-RAG
• Full Portfolio: Collection (129 repos)

Ayi NEDJIMI — LinkedIn | Website