Training uses Asymmetric Focal Loss, which emphasizes hard negatives while keeping positive weighting mild. This helps balance recall and false positive rate.

Gamma_pos: 0.0 (minimal emphasis on positives)
Gamma_neg: 4.0 (stronger emphasis on negatives)
Clip: 0.05 (stability for probabilities)

Validation is performed every 5000 steps, with early stopping to prevent overfitting.

Usage

python

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline
from peft import PeftModel

# Base ModernBERT model
base_model_name = "answerdotai/ModernBERT-base"

# LoRA adapter checkpoint
adapter_model_name = "AINovice2005/ModernBERT-base-lora-cicflow-1m-r16"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_name)

# Load base masked language model
base_model = AutoModelForMaskedLM.from_pretrained(base_model_name)

# Attach LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_model_name)

# Move to device
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

# Build fill-mask pipeline
fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer,
    device=0 if device == "cuda" else -1
)

# Example usage
text = "The network traffic shows a [MASK] pattern."
outputs = fill_mask(text)

for o in outputs:
    print(f"Token: {o['token_str']}, Score: {o['score']:.4f}")

Intended Use

Binary classification tasks where recall is critical.
Efficient fine‑tuning scenarios with limited compute resources.
Research and experimentation with parameter‑efficient methods.

Artifacts:

LoRA adapter
Training configuration and evaluation logs