ModernLogBERT

Overview

• dataprocess.py: Preprocesses HDFS log files, extracts log messages, groups them by block ID, and tokenizes sequences for model input.
• train.py: Trains the LogBERT model using Masked Log Word Prediction (MLWP) and KL divergence tasks for anomaly detection.

Requirements

• Python 3.8+
• PyTorch
• Transformers (huggingface)
• Pandas
• NumPy
• Scikit-learn

Setup

1. Clone the Repository:
   bash

   1git clone https://huggingface.co/MingtongDu/ModernLogBERT 2cd ModernLogBERT
2. Prepare Data:
   • Place your HDFS log file (e.g., HDFS.log) and anomaly label file (e.g., anomaly_label.csv) in an accessible directory.
   • Update the file paths in dataprocess.py and train.py to match your environment.
3. Tokenizer:
   • The custom tokenizer is saved in the output directory during preprocessing. Ensure the tokenizer path is correctly specified in train.py.
4. Pretrained Model:
   • The model uses a pretrained checkpoint (ep165.pt). Update the CHECKPOINT_PATH in train.py to point to your checkpoint file.

Usage

1. Data Preprocessing

• Parses HDFS.log to extract log messages and block IDs.
• Loads labels from anomaly_label.csv.
• Groups logs by block ID and splits into train (normal sequences only), validation, and test sets.
• Tokenizes sequences and saves them as train.pt, val.pt, and test.pt in the specified output directory.

2. Model Training

• Loads preprocessed datasets and the custom tokenizer.
• Initializes the LogBERT model with a pretrained ModernBERT backbone.
• Trains the model using MLWP and KL divergence losses.
• Evaluates performance on the test set and saves the trained model as EP3_froze21.pt.

Hyperparameters

• dataprocess.py: MAX_LENGTH, TRAIN_RATIO, VAL_RATIO, TEST_RATIO, PRINT_SAMPLES, RANDOM_SEED.
• train.py: BATCH_SIZE, NUM_EPOCHS, LEARNING_RATE, NUM_LAYERS_TO_FREEZE, DROPOUT_RATE, MASK_PROB, ALPHA, G, R.

Files

• dataprocess.py: Script for log parsing, label loading, grouping, tokenization, and data saving.
• train.py: Script for loading data, defining the LogBERT model, and training with anomaly detection metrics.
• output/: Directory containing processed datasets (train.pt, val.pt, test.pt) and the saved tokenizer.
• saved_model/: Directory for saving the trained model checkpoint.

Notes

• The model is optimized for English log data and may require fine-tuning for other languages.
• Ensure sufficient GPU memory for training, especially with long sequences (up to 512 tokens).
• For improved efficiency, install Flash Attention 2 if your GPU supports it.

Citation

ModernLogBERT: Mingtong Du and Haoran Li. https://huggingface.co/MingtongDu/ModernLogBERT

License