|
--- |
|
license: llama3 |
|
language: |
|
- en |
|
metrics: |
|
- perplexity |
|
base_model: |
|
- meta-llama/Meta-Llama-3-8B |
|
pipeline_tag: text-generation |
|
tags: |
|
- log |
|
- anomaly |
|
- detection |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
This is a Llama 3 model finetuned on execution logs to be used for a sock-shop app anomaly detections. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
This model was finetuned on a variety of system logs of a sock shop app. Given a log chunk of 10 messages, it generates the next log message according to normal execution. |
|
|
|
- **Developed by:** Luís Almeida, Diego Pedroso, Lucas Pulcinelli, William Aisawa, Sarita Bruschi, Inês Dutra |
|
- **Model type:** Text Generation |
|
- **Language(s) (NLP):** [English] |
|
- **License:** [llama3] |
|
- **Finetuned from model:** Llama 3 8b |
|
|
|
### Model Sources |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** https://github.com/lasdpc-icmc/maia |
|
- **Paper [optional]:** [More Information Needed] |
|
|
|
## Uses |
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
Since the model was finetuned on execution logs of the sock-shop app, it is intended to be used to generate logs of said app. To adapt it to another system, it should be |
|
finetuned on a sample of execution logs of the new system. |
|
|
|
### Direct Use |
|
|
|
Direct plugin to the sock-shop app. |
|
|
|
### Out-of-Scope Use |
|
|
|
The usage of this model on execution logs that it hasn't been finetuned on may yield bad results. |
|
|
|
### Recommendations |
|
|
|
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. --> |
|
|
|
We recommend users to finetune this model on logs of their app before they use it. |
|
|
|
## How to Get Started with the Model |
|
|
|
Please refer to https://github.com/lasdpc-icmc/maia/apps/llm for the code files that were developed for this model. The file "eval_llm.py" provides code to detect system |
|
anomalies. |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
https://huggingface.co/datasets/lmma25/sock-shop-logs-train |
|
|
|
### Training Procedure |
|
|
|
The model was finetuned using the SFTTrainer from the transformer's library in an autoregressive way. |
|
|
|
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
- **Training regime:** 10 epochs, AdamW optimizer, 1e-4 learning rate, bf16, weight decay 0.01, max gradient norm 0.3, cosine learning rate scheduler |
|
|
|
## Evaluation |
|
|
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
|
|
#### Testing Data |
|
|
|
https://huggingface.co/datasets/lmma25/sock-shop-logs-test |
|
|
|
#### Metrics |
|
|
|
The model was used to detect anomalies on a small sample of execution logs, achieving a precision of 0.77 and a recall of 1. Precision and recall metrics were used since |
|
they allow for the accurate assessment of model behavior in regards to false positives and false negatives. |
|
|
|
### Results |
|
|
|
Precision 0.77 |
|
Recall 1 |
|
|
|
|
|
## Citation [optional] |
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
**BibTeX:** |
|
|
|
[More Information Needed] |
