Foundation-Sec-8B-Instruct - Model Card

Model Information

• Model Name: Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct (Foundation-Sec-8B-Instruct)
• Model Developer: Foundation AI at Cisco
• Model Card Contact: https://fdtn.ai/contact
• Technical Report: Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report
• Model Release Date: August 1st, 2025
• Supported Language(s): English
• Model Architecture: Auto-regressive language model that uses an optimized transformer architecture (Meta Llama-3.1-8B backbone)
• Training Objective: Instruction following and alignment with human preferences
• Training Data Status: This is a static model trained on an offline dataset. Future versions of the tuned models will be released on updated data.
• License: See NOTICE.md

Intended Use

Intended Use Cases

• SOC Acceleration: Automating triage, summarization, case note generation, and evidence collection.
• Proactive Threat Defense: Simulating attacks, prioritizing vulnerabilities, mapping TTPs, and modeling attacker behavior.
• Engineering Enablement: Providing security assistance, validating configurations, assessing compliance evidence, and improving security posture.

Downstream Use

• Summarization
  • Summarizing detection playbooks and incident reports
  • Consolidating fragmented analyst notes into structured case summaries
• Classification
  • Mapping threats to MITRE ATT&CK techniques
  • Prioritizing vulnerabilities based on contextual risk
  • Classifying security-relevant emails and leaked file contents
• Named Entity Recognition
  • Extracting compliance evidence from documents
  • Building network behavior profiles from technical manuals
• Question & Answer
  • Assisting SOC analysts with alert triage and investigation
  • Responding to cloud security and software compliance queries
• Reasoning and Text Generation
  • Generating red-team attack plans and threat models
  • Predicting attacker next steps in active investigations
  • Enriching vulnerability scan results with contextual insights

Out-of-Scope Use

1. Generating harmful content - The model should not be used to:
   • Generate malware or other malicious code
   • Create phishing content or social engineering scripts
   • Develop attack plans targeting specific organizations
   • Design exploitation techniques for vulnerabilities without legitimate security research purposes
2. Critical security decisions without human oversight - The model should not be used for:
   • Autonomous security decision-making without human review
   • Critical infrastructure protection without expert supervision
   • Final determination of security compliance without human verification
   • Autonomous vulnerability remediation without testing
3. Legal or medical advice - The model is not qualified to provide:
   • Legal advice regarding security regulations, compliance requirements, or intellectual property disputes
   • Legal advice regarding security issues that would reference legal statutes, precedents, or case law necessary to provide legal advice
   • Medical advice regarding health impacts of security incidents
4. Non-security use cases - The model is specifically optimized for cybersecurity and may not perform as well on general tasks as models trained for broader applications.
5. Violation of Laws or Regulations - Any use that violates applicable laws or regulations.

How to Get Started with the Model

Training and Evaluation

Training Data

Training Setup

• Instruction fine-tuning to follow human instructions
• RLHF to align model answers to human preferences
• 4096-token sequence length
• Optimizer: AdamW

Evaluation

• CTI-MCQA: 2,500 multiple-choice questions testing cybersecurity knowledge across frameworks like MITRE ATT&CK, NIST, GDPR, and threat intelligence best practices.
• CTI-RCM: 1,000 vulnerability root cause mapping examples linking CVEs to CWE categories, assessing deep understanding of security weaknesses.
• CTI-VSP: A set of 1,000 CVE descriptions where models predict the CVSS v3 Base metrics and compute the overall score, with performance measured by the average absolute difference from the true scores.
• IF-Eval: 541 instruction-following prompts designed for automated, reproducible assessment of LLM instruction-following capabilities.
• Alpaca Eval 2: 805 single-turn prompts auto-scored by GPT-4 Turbo against a GPT-4 Turbo reference, validated with 20,000 human preference votes, and closely matching ChatBot Arena results.

• +3 to +11 point gains over Llama-3.1-8B-Instruct across security-specific benchmarks.
• Exceptional Instruction-Following capabilities exceeding that of Llama-3.1-8B-Instruct.
• Competitive against small Frontier Models such as GPT-4o-mini on instruction-following capabilities and cybersecurity tasks.

Safety Alignment

Limitations

1. Domain-specific knowledge limitations:
   • Foundation-Sec-8B-Instruct may not be familiar with recent vulnerabilities, exploits, or novel attack vectors or security technologies released after its training cutoff date
   • Knowledge of specialized or proprietary security systems or tools may be limited
2. Potential biases:
   • The model may reflect biases present in security literature and documentation
   • The model may be trained on known attack patterns and have difficulty recognizing novel attack vectors
   • Security practices and recommendations may be biased toward certain technological ecosystems
   • Geographic and cultural biases in security approaches may be present
3. Security risks:
   • The model cannot verify the identity or intentions of users
   • Adversarial prompting techniques might potentially bypass safety mechanisms
   • The model may unintentionally provide information that could be misused if proper prompting guardrails are not implemented
4. Contextual blindness:
   • The model may struggle to understand the complex interrelationships between systems, users, and data in order to provide accurate context.
5. Technical limitations:
   • Performance varies based on how security concepts are described in prompts
   • May not fully understand complex, multi-step security scenarios without clear explanation
   • Cannot access external systems or actively scan environments
   • Cannot independently verify factual accuracy of its outputs
6. Ethical considerations:
   • Dual-use nature of security knowledge requires careful consideration of appropriate use cases

Recommendations

1. Human oversight:
   • Always have qualified security professionals review model outputs before implementation
   • Use the model as an assistive tool rather than a replacement for expert human judgment
   • Implement a human-in-the-loop approach for security-critical applications
2. System design safeguards:
   • Implement additional validation layers for applications built with this model
   • Consider architectural constraints that limit the model's ability to perform potentially harmful actions (excessive agency)
   • Deploy the model in environments with appropriate access controls
3. Prompt engineering:
   • Use carefully designed prompts that encourage ethical security practices
   • Include explicit instructions regarding responsible disclosure and ethical hacking principles
   • Structure interactions to minimize the risk of inadvertently harmful outputs
4. Knowledge supplementation:
   • Supplement the model with up-to-date security feeds and databases
   • Implement retrieval-augmented generation for current threat intelligence sources
5. Usage policies:
   • Develop and enforce clear acceptable use policies for applications using this model
   • Implement monitoring and auditing for high-risk applications
   • Create documentation for end users about the model's limitations