File size: 3,573 Bytes
1538836 9a39249 1538836 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 |
---
language: en
library_name: LogClassifier
tags:
- log-classification
- log feature
- log-similarity
- transformers
- AIOps
pipeline_tag: text-classification
---
# s2-log-classifier-BERT-v1
This model is a transformers classification model, trained using BERTForSequenceClassification designed for use in network and device log mining tasks.
Developed by [Selector AI](https://www.selector.ai/)
## Model Usage
```python
from transformers import BertForSequenceClassification, BertTokenizer
# Step 1: Load the model and tokenizer from Hugging Face
model = BertForSequenceClassification.from_pretrained("SelectorAI/s2-log-classifier-BERT-v1")
tokenizer = BertTokenizer.from_pretrained("SelectorAI/s2-log-classifier-BERT-v1")
import torch
model.eval()
# Step 2: Prepare the input data (Example log text)
log_text = "Error occurred while accessing the database."
# Tokenize the input data
inputs = tokenizer(log_text, return_tensors="pt", padding=True, truncation=True, max_length=128)
# Step 3: Make predictions
with torch.no_grad():
outputs = model(**inputs)
logits = outputs.logits
# Step 4: Get the predicted class (the class with the highest score)
predicted_class = torch.argmax(logits, dim=1).item()
# label mapping (can load from JSON file in repo or config)
label_mapping = model.config.id2label
# Step 5: Get the event name
predicted_event = label_mapping[predicted_class]
print(f"Predicted Event: {predicted_event}")
```
## Background
The model focuses on structured and semi-structured log data, outputing around 60 different event categories. It is highly effective
for real-time log analysis, anomaly detection, and operational monitoring, helping organizations manage
large-scale network data by automatically classifying logs into predefined categories, facilitating faster
and more accurate diagnosis of network issues.
## Intended uses
Our model is intended to be used as classifier. Given an input text (a log coming from a network/device/router), it outputs a corresponding event most associated with the log.
The possible events that can be classified are shown in [encoder-main.json](https://huggingface.co/rahulm-selector/log-classifier-BERT-v1/blob/main/encoder-main.json)
## Training Details
### Data
The model was trained on a variety of network events and system logs, focusing on monitoring and analyzing state changes,
protocol behaviors, and hardware interactions across infrastructure components. This included tracking routing issues,
protocol neighbor state changes, link stability, and security events, ensuring that the model could recognize and
classify critical patterns in device communications, network health, and configuration activities.
### Train/Test Split
- **Train Data Size**: `~80K Logs`
- **Test Data Size**: `~20K Logs`
#### Hyper Parameters
The following hyperparameters were used during training to optimize the model's performance:
- **Batch Size**: `32`
- **Learning Rate**: `.001`
- **Optimizer**: `Adam`
- **Epochs**: `10`
- **Dropout Rate**: N/A
- **LSTM Hidden Dimension**: `384`
- **Embedding Dimension**: `384`
## Credits
This project was developed by a collaborative team at [Selector AI](https://www.selector.ai/). Below are the key contributors:
### Authors
- **Rahul Muthuswamy**
Role: Project Lead and Model Developer
Email: [[email protected]]
- **Alex Lau**
Role: Mentor
Email: [[email protected]]
- **Sebastian Reyes**
Role: Mentor
Email: [[email protected]]
- **Surya Nimmagadda**
Role: Mentor
Email: [[email protected]]
|