A DistilBERT model fully fine-tuned on the EURLEX subset of the Pile of Law dataset, achieving 87.54% perplexity improvement in legal language understanding.

📊 Model Performance

Quantitative Results

Metric	Base Model	Fine-tuned Model	Improvement
Validation Loss	2.4934	0.4106	83.53%
Perplexity	12.1026	1.5077	87.54%

Training Progress

Epoch	Training Loss	Validation Loss
1	0.6224	0.5546
2	0.5209	0.4721
3	0.4833	0.4408
4	0.4725	0.4217
5	0.4477	0.4146

🏗️ Model Details

Architecture

Base Model: distilbert-base-uncased
Parameters: 66 million (fully fine-tuned)
Fine-tuning Type: Full parameter updates
Objective: Masked Language Modeling (MLM)
Max Sequence Length: 512 tokens

Training Configuration

python

learning_rate = 5e-5
batch_size = 16
epochs = 5
mlm_probability = 0.15
weight_decay = 0.01
warmup_steps = 500

Technical Information

Model Specifications

Architecture: DistilBERT (6 layers, 768 hidden size, 12 attention heads)
Parameters: 66,000,000

-Vocabulary Size: 30,522

-Training Time: ~2.5 hours (5 epochs)

Training Data

Dataset Statistics:

Source: EURLEX subset of Pile of Law
Total Examples: 106,387 documents
Training Set: 85,109 examples
Validation Set: 10,639 examples
Test Set: 10,639 example

Text Characteristics:

Domain: European Union Legal Documents
Average Length: 13,327 characters
Min Length: 80 characters
Max Length: 1,415,135 characters

Usage :

python

pip install transformers torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model = AutoModelForMaskedLM.from_pretrained("Nahla-yasmine/legal-distilbert")
tokenizer = AutoTokenizer.from_pretrained("Nahla-yasmine/legal-distilbert")

# Masked language prediction
text = "The court found the defendant [MASK] of all charges."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

python

def predict_masked_text(text, model, tokenizer, num_predictions=5):
    inputs = tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)
    
    mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
    logits = outputs.logits[0, mask_token_index, :]
    top_tokens = torch.topk(logits, num_predictions, dim=1).indices[0].tolist()
    
    return [tokenizer.decode([token]) for token in top_tokens]

# Example usage
predictions = predict_masked_text(
    "The contract was declared [MASK] due to fraudulent activities.",
    model, tokenizer
)
print(predictions)  # ['void', 'invalid', 'bankrupt', 'terminated', 'cancelled']

Performance Analysis:

87.54% perplexity improvement demonstrates superior legal language understanding
83.53% loss reduction indicates effective domain adaptation
Consistent training progression shows stable learning across epochs
The fine-tuned model significantly outperforms the base DistilBERT on legal text, showing:

(Better understanding of legal terminology,More appropriate contextual predicti ,Enhanced domain-specific knowledge)

Intended Use Cases:

- Legal document analysis and processing

- Legal text completion and generation

- Law education and research tools

- Legal AI assistants

- Document review automation

- Legal terminology understanding

Technical Constraints:

- English language only

- Maximum sequence length: 512 tokens

- Training data up to 2022

Nahla-yasmine/legal-DistilBERT-fullFN-eurlex

Legal-DistilBERT: Fine-tuned on EU Legal Documents

📊 Model Performance

Quantitative Results

Training Progress

🏗️ Model Details

Architecture

Training Configuration

Technical Information

Model Specifications

Training Data

Dataset Statistics:

Text Characteristics:

Usage :

Performance Analysis:

Intended Use Cases:

Technical Constraints:

Nahla-yasmine/
legal-DistilBERT-fullFN-eurlex