GEM-ModernBERT HQ Legal is a ModernBERT-base model pre-trained from scratch on a strategically curated 21GB corpus of Greek legal, parliamentary, and governmental text. This model leverages ModernBERT's cutting-edge architectural innovations including Flash Attention 2, StableAdamW optimizer, 1024-token context length, and advanced memory optimization techniques to deliver superior performance on Greek legal document understanding tasks.

Building upon our proven quality-based data repetition strategy, this model incorporates ModernBERT's state-of-the-art training methodology with 30% masking probability, trapezoidal learning rate scheduling, and optimized batch sizing for enhanced convergence and performance. The model is specifically designed to handle longer legal documents with its extended 1024-token context window while maintaining computational efficiency through advanced optimization techniques.

This model represents the culmination of our Greek legal language modeling research, combining domain expertise with the latest architectural advances in transformer-based language models. It has been optimized for downstream tasks such as Named Entity Recognition (NER), Text Classification, and Question Answering within the complex legal domain.

How to Get Started

You can use this model directly with the fill-mask pipeline:

python

from transformers import pipeline

# Load the model
fill_mask = pipeline(
    "fill-mask",
    model="novelcore/gem-modernbert-hq-legal",
    tokenizer="novelcore/gem-modernbert-hq-legal"
)

# Example from a legal context with longer sequence support
text = "Σύμφωνα με το άρθρο 15 του Συντάγματος, η <mask> των δικαιωμάτων του ανθρώπου αποτελεί βασική υποχρέωση του κράτους στο πλαίσιο της δημοκρατικής πολιτείας."

# Get predictions
predictions = fill_mask(text)
print(predictions)

For downstream tasks:

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# For legal document classification with extended context
tokenizer = AutoTokenizer.from_pretrained("novelcore/gem-modernbert-hq-legal")
model = AutoModelForSequenceClassification.from_pretrained("novelcore/gem-modernbert-hq-legal")

# The model supports up to 1024 tokens for longer legal documents

Training Data

The model was pre-trained on the same comprehensive corpus of Greek text used in our previous models, employing our proven quality-based data repetition strategy that increases exposure to higher-quality legal content. The original 16.75GB corpus was expanded to 21.12GB through strategic repetition, now processed with 1024-token sequences for enhanced context understanding.

Quality-Based Data Repetition Strategy

Dataset	Original Size (GB)	Quality Level	Repetition Factor	Effective Size (GB)
Raptarchis Legal Dictionary	0.35	Best	4x	1.40
Political Reports of the Supreme Court	1.20	Medium-Best	3x	3.60
Eur-Lex (Greek Content)	0.92	Medium	2x	1.84
FEK - Greek Government Gazette	11.00	Low	1x	11.00
Greek Parliament Proceedings	2.90	Low-Medium	1x	2.90
Europarl (Greek Content)	0.38	Low	1x	0.38
TOTAL	16.75 GB	-	-	21.12 GB

Enhanced Context Processing

With 1024-token sequences, this model can process:

Complete legal articles without truncation
Full court decisions with extended reasoning
Complex legislative texts with multiple references
Parliamentary debates with comprehensive context

Training Procedure

Model Architecture

The model uses the ModernBERT-base architecture with the following configuration:

Hidden Size: 768
Attention Heads: 12
Hidden Layers: 12
Parameters: ~139M
Max Position Embeddings: 1024
Vocabulary Size: 50,373
Flash Attention 2: Enabled
Context Length: 1024 tokens (2x longer than previous models)

Key Architectural Advantages

ModernBERT's innovations provide significant benefits for legal text processing:

Flash Attention 2: Memory-efficient attention computation for longer sequences
Extended Context: 1024-token sequences capture complete legal documents
StableAdamW Optimizer: Enhanced training stability and convergence
Optimized MLM: 30% masking probability for improved representation learning
Advanced Memory Management: Optimized CUDA memory allocation for large batches

Preprocessing

The text was processed into 1024-token chunks using ModernBERT's tokenizer (vocabulary: 50,373 tokens), providing excellent coverage of Greek legal terminology while maintaining compatibility with the base architecture.

Higher-quality sources were strategically repeated during the data preparation phase, with sequences now capturing much more context per training example.

Pre-training

The model was pre-trained from scratch for 150,000 steps on 8x NVIDIA H100 80GB GPUs, using BFloat16 (bf16) mixed-precision with advanced optimization techniques. The training took approximately 97 hours and 9 minutes to complete.

Key Training Optimizations

Batch Size Optimization:

Per-device batch size: 16 (optimized for H100 memory)
Gradient accumulation steps: 8
Effective batch size: 1,024 (16 × 8 × 8 GPUs)
Context length: 1024 tokens per sequence

StableAdamW Configuration:

Learning Rate: 0.0002 (conservative for stable convergence)
Weight Decay: 0.1
Adam Beta1: 0.9
Adam Beta2: 0.95
Adam Epsilon: 1e-08
Gradient Clipping: 1.0
Epsilon Mode: element_wise

Advanced Learning Rate Schedule:

Schedule Type: Polynomial decay with trapezoidal warmup
Warmup Steps: 9,000
Decay Power: 0.5 (square-root decay)
Max Steps: 150,000

ModernBERT Specifications:

MLM Probability: 0.30 (higher than traditional 15%)
Max Sequence Length: 1024
Flash Attention 2: Enabled with optimizations
Memory Optimization: Advanced CUDA allocation strategies

Training Results

The model achieved excellent performance metrics:

Final Training Loss: 0.7648
Final Evaluation Loss: 0.7751
Training Infrastructure: 8x NVIDIA H100 80GB GPUs
Total Training Steps: 150,000
Total Training Time: 97 hours 9 minutes
Train/Validation Split: 90%/10%
Effective Training Data: 21.12GB (with quality-based repetition)
Context Length: 1024 tokens per sequence

Advanced Training Infrastructure

The model was trained with cutting-edge optimizations:

Flash Attention 2 Optimizations:

yaml

FLASH_ATTENTION_FORCE_FP16: "0"        # Use bfloat16
FLASH_ATTENTION_SKIP_RESHAPE: "1"      # Skip unnecessary reshapes
FLASH_ATTENTION_CAUSAL: "0"            # Non-causal for BERT
FORCE_FLASH_ATTENTION: "1"             # Force Flash Attention usage

Memory Optimization:

PYTORCH_CUDA_ALLOC_CONF: "max_split_size_mb:256,roundup_power2_divisions:16,expandable_segments:True,garbage_collection_threshold:0.8"

Distributed Training:

Backend: NCCL with extended timeout configurations
Mixed Precision: BFloat16 for optimal H100 performance
Evaluation Frequency: Every 5,000 steps
Checkpointing: Every 5,000 steps
Logging: Every 250 steps

Key Innovations

ModernBERT Architecture Benefits

Extended Context Window: 1024 tokens vs 512 in previous models
Flash Attention 2: Memory-efficient attention for longer sequences
StableAdamW Optimizer: Enhanced training stability and convergence
Higher MLM Probability: 30% masking for improved representation learning
Trapezoidal LR Schedule: Optimized learning rate progression

Quality-Based Data Repetition

Consistent with our previous models:

Highest quality sources (legal dictionaries) repeated 4x
Medium-high quality sources (court reports) repeated 3x
Medium quality sources (EU legal texts) repeated 2x
Lower quality sources used once for diversity