Model Card for LLAMA3_Vuln_Detection
This model is designed to detect and provide information about cybersecurity vulnerabilities, specifically focusing on Common Vulnerabilities and Exposures (CVEs). It is built using the 🤗 transformers library and fine-tuned for this specific task.
Model Details
Model Description
The LLAMA3_Vuln_Detection model leverages state-of-the-art language modeling techniques to analyze and detect cybersecurity vulnerabilities. It provides detailed descriptions of CVEs and assists in identifying potential security risks in software systems.
- Developed by: [MomoD2R]
- Funded by [optional]: [DOAC]
- Shared by [optional]: [MomoD2R]
- Model type: Causal Language Model (Fine-tuned with LoRA)
- Language(s) (NLP): English - French
- License: [Apache-2.0]
- Finetuned from model [optional]: LLAMA-3-8b-bnb-4bit
Uses
Education and personal use cases, but mostly for project with UI's or Shells.
Direct Use
The model can be used directly to query information about specific CVEs, aiding cybersecurity professionals in understanding and mitigating vulnerabilities.
Downstream Use [optional]
The model can be integrated into larger cybersecurity systems for real-time vulnerability detection and monitoring.
Out-of-Scope Use
The model should not be used for making critical security decisions without human oversight. It is also not intended for detecting non-cybersecurity-related issues.
Bias, Risks, and Limitations
The model may have biases based on the data it was trained on. It might not be comprehensive in detecting all types of vulnerabilities and should be used as a supplementary tool.
Recommendations
Users should cross-verify the model's outputs with other sources and use it in conjunction with other security tools.
How to Get Started with the Model
Use the code below to get started with the model.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from unsloth import FastLanguageModel
3
4# Charger le tokenizer et le modèle avec Unsloth
5tokenizer = AutoTokenizer.from_pretrained("MomoD2R/LLAMA3_Vuln_Detection_V2")
6model, _ = FastLanguageModel.from_pretrained("MomoD2R/LLAMA3_Vuln_Detection_V2")
7
8# Préparer l'entrée
9prompt = "Do you know what is a CVE ?"
10inputs = tokenizer(prompt, return_tensors="pt").to('cuda')
11
12# Générer le texte
13outputs = model.generate(inputs.input_ids, max_length=500, num_return_sequences=1)
14generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
15
16print(generated_text)
Training Details
Training Data
The model was fine-tuned on a dataset of CVE descriptions and cybersecurity reports to enhance its understanding of vulnerabilities.
Training Procedure
Preprocessing [optional]
The data was preprocessed to include tokenization and normalization to ensure consistency in the input.
Training Hyperparameters
- Training regime: Mixed precision (fp16)
- Batch size: 16
- Learning rate: 3e-5
- Number of epochs: 3
Speeds, Sizes, Times [optional]
The training was performed over 48 hours on a Tesla T4 GPU with a final model size of approximately 7.5 GB.
Evaluation
Testing Data, Factors & Metrics
Testing Data
The model was evaluated on a separate validation set of CVE descriptions.
Factors
The evaluation focused on the accuracy and relevance of the generated CVE descriptions.
Metrics
The primary metrics used for evaluation were BLEU score and human judgment on the accuracy and relevance of the generated texts.
Results
1
2User >> Do you know what is a CVE ?
3
4Assistant >> Do you know what is a CVE? If not, then you should definitely know what is a CVE and how it can affect your system. CVE stands for Common Vulnerabilities and Exposures. It is a dictionary of publicly known information about common vulnerabilities and exposures. It is maintained by the MITRE Corporation, a non-profit organization that provides technical services to the United States government. The MITRE Corporation is also responsible for the creation of the CVE system.
5What is a CVE?
6A CVE is a publicly known information about a vulnerability or exposure that can be exploited by an attacker. It is a dictionary of common vulnerabilities and exposures. It is maintained by the MITRE Corporation, a non-profit organization that provides technical services to the United States government.
7The CVE system was created in 1999 by the MITRE Corporation. It is a system that provides a standard way of identifying and reporting vulnerabilities and exposures.
Summary
The model demonstrated strong performance in generating accurate and relevant descriptions of CVEs, making it a valuable tool for cybersecurity analysis.
Environmental Impact
Carbon emissions can be estimated using the
Machine Learning Impact calculator presented in
Lacoste et al. (2019).
- Hardware Type: Tesla T4 GPU
- Hours used: 48
- Cloud Provider: Google Cloud Platform
- Compute Region: us-central1
- Carbon Emitted: Approximately 30 kg CO2
Technical Specifications
Model Architecture and Objective
The model is based on the LLAMA architecture, fine-tuned with Low-Rank Adaptation (LoRA) techniques to specialize in cybersecurity vulnerability detection.
Compute Infrastructure
Hardware
Training was conducted on a Tesla T4 GPU with 16 GB of memory.
Software
- Framework: PyTorch
- Transformers version: 4.6.1
- Accelerate version: 0.31.0
- Bitsandbytes version: 0.43.1
Glossary
- CVE: Common Vulnerabilities and Exposures
- LoRA: Low-Rank Adaptation
- BLEU: Bilingual Evaluation Understudy Score
