| --- |
| base_model: |
| - fdtn-ai/Foundation-Sec-8B |
| language: |
| - en |
| library_name: transformers |
| license: other |
| pipeline_tag: text-generation |
| tags: |
| - security |
| - llama |
| --- |
| |
| # Foundation-Sec-8B-Instruct - Model Card |
|
|
| ## Model Information |
|
|
| Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct (Foundation-Sec-8B-Instruct) is an open-weight, 8-billion parameter instruction-tuned language model specialized for cybersecurity applications. |
| It extends the Foundation-Sec-8B base model with instruction-following capabilities. |
| It leverages prior training to understand security concepts, terminology, and practices across multiple security domains. |
| Further instruction-tuning allows the model to interact with human users in a chat-like interface. |
| Foundation-Sec-8B-Instruct enables organizations to build AI-driven security tools that can be deployed locally, reducing dependency on cloud-based AI services while maintaining high performance on security-related tasks. |
|
|
| - **Model Name:** Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct (Foundation-Sec-8B-Instruct) |
| - **Model Developer:** Foundation AI at Cisco |
| - **Model Card Contact:** https://fdtn.ai/contact |
| - **Technical Report:** [Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report](https://huggingface.co/papers/2508.01059) |
| - **Model Release Date:** August 1st, 2025 |
| - **Supported Language(s):** English |
| - **Model Architecture:** Auto-regressive language model that uses an optimized transformer architecture (Meta Llama-3.1-8B backbone) |
| - **Training Objective:** Instruction following and alignment with human preferences |
| - **Training Data Status:** This is a static model trained on an offline dataset. Future versions of the tuned models will be released on updated data. |
| - **License:** See NOTICE.md |
| |
| |
| ## Intended Use |
|
|
| ### Intended Use Cases |
|
|
| Foundation-Sec-8B-Instruct is designed for security practitioners, researchers, and developers building AI-powered security workflows and applications. |
| Foundation-Sec-8B-Instruct is optimized for three core use case categories: |
|
|
| - **SOC Acceleration**: Automating triage, summarization, case note generation, and evidence collection. |
| - **Proactive Threat Defense**: Simulating attacks, prioritizing vulnerabilities, mapping TTPs, and modeling attacker behavior. |
| - **Engineering Enablement**: Providing security assistance, validating configurations, assessing compliance evidence, and improving security posture. |
|
|
| The model is intended for local deployment in environments prioritizing data security, regulatory compliance, and operational control. |
|
|
| ### Downstream Use |
|
|
| Foundation-Sec-8B-Instruct can be used directly for security-related chat use cases. Example downstream applications include: |
|
|
| - Summarization |
| - Summarizing detection playbooks and incident reports |
| - Consolidating fragmented analyst notes into structured case summaries |
| - Classification |
| - Mapping threats to MITRE ATT&CK techniques |
| - Prioritizing vulnerabilities based on contextual risk |
| - Classifying security-relevant emails and leaked file contents |
| - Named Entity Recognition |
| - Extracting compliance evidence from documents |
| - Building network behavior profiles from technical manuals |
| - Question & Answer |
| - Assisting SOC analysts with alert triage and investigation |
| - Responding to cloud security and software compliance queries |
| - Reasoning and Text Generation |
| - Generating red-team attack plans and threat models |
| - Predicting attacker next steps in active investigations |
| - Enriching vulnerability scan results with contextual insights |
|
|
| For questions or assistance with fine-tuning Foundation-Sec-8B-Instruct, please reach out to the team. |
|
|
| ### Out-of-Scope Use |
|
|
| The following uses are out-of-scope and are neither recommended nor intended use cases: |
|
|
| 1. **Generating harmful content** - The model should not be used to: |
| - Generate malware or other malicious code |
| - Create phishing content or social engineering scripts |
| - Develop attack plans targeting specific organizations |
| - Design exploitation techniques for vulnerabilities without legitimate security research purposes |
| 2. **Critical security decisions without human oversight** - The model should not be used for: |
| - Autonomous security decision-making without human review |
| - Critical infrastructure protection without expert supervision |
| - Final determination of security compliance without human verification |
| - Autonomous vulnerability remediation without testing |
| 3. **Legal or medical advice** - The model is not qualified to provide: |
| - Legal advice regarding security regulations, compliance requirements, or intellectual property disputes |
| - Legal advice regarding security issues that would reference legal statutes, precedents, or case law necessary to provide legal advice |
| - Medical advice regarding health impacts of security incidents |
| 4. **Non-security use cases** - The model is specifically optimized for cybersecurity and may not perform as well on general tasks as models trained for broader applications. |
| 5. **Violation of Laws or Regulations** - Any use that violates applicable laws or regulations. |
|
|
| ## How to Get Started with the Model |
|
|
| Use the code below to get started with the model. |
| [The cookbook](https://github.com/cisco-foundation-ai/cookbook) provides example use cases, code samples for adoption, and references. |
|
|
| ```python |
| # Import the required libraries |
| import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| # Load the model and tokenizer |
| tokenizer = AutoTokenizer.from_pretrained("fdtn-ai/Foundation-Sec-8B-Instruct") |
| model = AutoModelForCausalLM.from_pretrained("fdtn-ai/Foundation-Sec-8B-Instruct") |
| |
| prompt = "CVE-2015-10011 is a vulnerability about OpenDNS OpenResolve improper log output neutralization. What is the corresponding CWE?" |
| |
| messages = [ |
| {"role": "user", "content": prompt} |
| ] |
| |
| model_inputs = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(model_inputs, return_tensors="pt", add_special_tokens=False) |
| output = model.generate(**inputs, temperature=0.1, max_new_tokens=250) |
| resp = tokenizer.batch_decode(output)[0] |
| print(resp.replace(model_inputs, "")) |
| ``` |
|
|
| ## Training and Evaluation |
|
|
| ### Training Data |
|
|
| Foundation-Sec-8B-Instruct was trained on a wide variety of public and proprietary question answer/pairs for general and security-specific instruction-following. |
|
|
| **Data cutoff:** April 10th, 2025. |
|
|
| A more detailed description of the methodology is available in the technical report. |
|
|
| ### Training Setup |
|
|
| Foundation-Sec-8B-Instruct is based on the **Llama 3.1 8B** architecture. Training was performed on Cisco Foundation AI’s internal compute cluster. |
|
|
| Key training details: |
|
|
| - **Instruction fine-tuning** to follow human instructions |
| - **RLHF** to align model answers to human preferences |
| - **4096-token** sequence length |
| - **Optimizer:** AdamW |
|
|
| A more detailed description of the methodology is available in the technical report. |
|
|
| ### Evaluation |
|
|
| Foundation-Sec-8B-Instruct was benchmarked on cybersecurity and general reasoning tasks, using a standardized 0-shot instruction prompting setup (temperature = 0.3). |
|
|
| | **Benchmark** | **Foundation-sec-8B** | **Llama 3.1 8B** | **GPT-4o-mini** | |
| | --- | --- | --- | --- | |
| | CTI-MCQA | 0.644 | 0.617 | 0.672 | |
| | CTI-RCM | 0.692 | 0.558 | 0.655 | |
| | CTI-VSP | 0.802 | 0.815 | 0.792 | |
| | IF-Eval | 0.811 | 0.791 | 0.834 | |
| | Alpaca Eval 2 | 35.453 | 24.477 | 52.720 | |
|
|
| **Benchmark Overview:** |
|
|
| - **CTI-MCQA:** 2,500 multiple-choice questions testing cybersecurity knowledge across frameworks like MITRE ATT&CK, NIST, GDPR, and threat intelligence best practices. |
| - **CTI-RCM:** 1,000 vulnerability root cause mapping examples linking CVEs to CWE categories, assessing deep understanding of security weaknesses. |
| - **CTI-VSP:** A set of 1,000 CVE descriptions where models predict the CVSS v3 Base metrics and compute the overall score, with performance measured by the average absolute difference from the true scores. |
| - **IF-Eval:** 541 instruction-following prompts designed for automated, reproducible assessment of LLM instruction-following capabilities. |
| - **Alpaca Eval 2:** 805 single-turn prompts auto-scored by GPT-4 Turbo against a GPT-4 Turbo reference, validated with 20,000 human preference votes, and closely matching ChatBot Arena results. |
|
|
| **Key highlights:** |
|
|
| - **+3 to +11 point gains** over Llama-3.1-8B-Instruct across security-specific benchmarks. |
| - **Exceptional Instruction-Following capabilities** exceeding that of Llama-3.1-8B-Instruct. |
| - **Competitive against small Frontier Models** such as GPT-4o-mini on instruction-following capabilities and cybersecurity tasks. |
|
|
| For full benchmark details and evaluation methodology, please refer to the technical report. |
|
|
| ## Safety Alignment |
|
|
| Standard best practices were followed to align the model with general safety values. |
| Despite the alignment, however, safe out-of-the-box performance cannot be guaranteed. |
| Our evaluations show that while the model can achieve reasonable safety performance out-of-the-box, LlamaGuard provides much better protection against malicious requests. |
| It is recommended to deploy this model with additional safeguards (such as LlamaGuard) and human oversight. |
|
|
| | Model | HarmBench Performance | |
| |---|---| |
| | Llama-3.1-8b-Instruct | 72.43% | |
| | Foundation-Sec-8B-Instruct | 91.98% | |
| | **LlamaGuard** + Foundation-Sec-8B-Instruct | 99.25% | |
|
|
|
|
| ## Limitations |
|
|
| Foundation-Sec-8B-Instruct has several limitations that users should be aware of: |
|
|
| 1. **Domain-specific knowledge limitations**: |
| - Foundation-Sec-8B-Instruct may not be familiar with recent vulnerabilities, exploits, or novel attack vectors or security technologies released after its training cutoff date |
| - Knowledge of specialized or proprietary security systems or tools may be limited |
| 2. **Potential biases**: |
| - The model may reflect biases present in security literature and documentation |
| - The model may be trained on known attack patterns and have difficulty recognizing novel attack vectors |
| - Security practices and recommendations may be biased toward certain technological ecosystems |
| - Geographic and cultural biases in security approaches may be present |
| 3. **Security risks**: |
| - The model cannot verify the identity or intentions of users |
| - Adversarial prompting techniques might potentially bypass safety mechanisms |
| - The model may unintentionally provide information that could be misused if proper prompting guardrails are not implemented |
| 4. **Contextual blindness:** |
| - The model may struggle to understand the complex interrelationships between systems, users, and data in order to provide accurate context. |
| 5. **Technical limitations**: |
| - Performance varies based on how security concepts are described in prompts |
| - May not fully understand complex, multi-step security scenarios without clear explanation |
| - Cannot access external systems or actively scan environments |
| - Cannot independently verify factual accuracy of its outputs |
| 6. **Ethical considerations**: |
| - Dual-use nature of security knowledge requires careful consideration of appropriate use cases |
| |
| |
| ### Recommendations |
|
|
| To address the limitations of Foundation-Sec-8B-Instruct, we recommend: |
|
|
| 1. **Human oversight**: |
| - Always have qualified security professionals review model outputs before implementation |
| - Use the model as an assistive tool rather than a replacement for expert human judgment |
| - Implement a human-in-the-loop approach for security-critical applications |
| 2. **System design safeguards**: |
| - Implement additional validation layers for applications built with this model |
| - Consider architectural constraints that limit the model's ability to perform potentially harmful actions (excessive agency) |
| - Deploy the model in environments with appropriate access controls |
| 3. **Prompt engineering**: |
| - Use carefully designed prompts that encourage ethical security practices |
| - Include explicit instructions regarding responsible disclosure and ethical hacking principles |
| - Structure interactions to minimize the risk of inadvertently harmful outputs |
| 4. **Knowledge supplementation**: |
| - Supplement the model with up-to-date security feeds and databases |
| - Implement retrieval-augmented generation for current threat intelligence sources |
| 5. **Usage policies**: |
| - Develop and enforce clear acceptable use policies for applications using this model |
| - Implement monitoring and auditing for high-risk applications |
| - Create documentation for end users about the model's limitations |
