Model Card for Model ID
This is a Llama 3 model finetuned on execution logs to be used for a sock-shop app anomaly detections.
Model Details
Model Description
This model was finetuned on a variety of system logs of a sock shop app. Given a log chunk of 10 messages, it generates the next log message according to normal execution.
- Developed by: Luís Almeida, Diego Pedroso, Lucas Pulcinelli, William Aisawa, Sarita Bruschi, Inês Dutra
- Model type: Text Generation
- Language(s) (NLP): [English]
- License: [llama3]
- Finetuned from model: Llama 3 8b
Model Sources
- Repository: https://github.com/lasdpc-icmc/maia
- Paper [optional]: [More Information Needed]
Uses
Since the model was finetuned on execution logs of the sock-shop app, it is intended to be used to generate logs of said app. To adapt it to another system, it should be
finetuned on a sample of execution logs of the new system.
Direct Use
Direct plugin to the sock-shop app.
Out-of-Scope Use
The usage of this model on execution logs that it hasn't been finetuned on may yield bad results.
Recommendations
We recommend users to finetune this model on logs of their app before they use it.
How to Get Started with the Model
Please refer to
https://github.com/lasdpc-icmc/maia/apps/llm for the code files that were developed for this model. The file "eval_llm.py" provides code to detect system
anomalies.
Training Details
Training Data
Training Procedure
The model was finetuned using the SFTTrainer from the transformer's library in an autoregressive way.
Training Hyperparameters
- Training regime: 10 epochs, AdamW optimizer, 1e-4 learning rate, bf16, weight decay 0.01, max gradient norm 0.3, cosine learning rate scheduler
Evaluation
Testing Data, Factors & Metrics
Testing Data
Metrics
The model was used to detect anomalies on a small sample of execution logs, achieving a precision of 0.77 and a recall of 1. Precision and recall metrics were used since
they allow for the accurate assessment of model behavior in regards to false positives and false negatives.
Results
Precision 0.77
Recall 1
Citation [optional]
BibTeX:
[More Information Needed]
