Views
No views yet
| Metric | Score |
|---|---|
| Micro-F1 | 87.3% |
| Macro-F1 | 79.3% |
| Parameters | 110M |
| Loss | Weighted Cross-Entropy |
| Model | Micro-F1 | Macro-F1 | Params | Notes |
|---|---|---|---|---|
| Rule-based baseline | 20.9% | 16.0% | — | Keyword matching |
| BERT-base (paper) | ~83% | — | 110M | ContractNLI paper reference |
| contractnli-legalbert-nda-weighted | 87.3% | 79.3% | 110M | highest accuracy ← this model |
| contractnli-legalbert-nda-standard | 86.7% | 77.0% | 110M | |
| contractnli-bert-nda-standard | 86.9% | 76.7% | 110M | paper reproduction |
| contractnli-bert-nda-weighted | 86.3% | 77.9% | 110M | |
| contractnli-distilbert-nda | 86.0% | 76.2% | 66M | fastest inference, recommended for production |
| ID | Provision |
|---|---|
| nda-1 | Explicit identification |
| nda-2 | Non-inclusion of non-technical information |
| nda-3 | Inclusion of verbally conveyed information |
| nda-4 | Limited use |
| nda-5 | Sharing with employees |
| nda-7 | Sharing with third-parties |
| nda-8 | Notice on compelled disclosure |
| nda-10 | Confidentiality of Agreement |
| nda-11 | No reverse engineering |
| nda-12 | Permissible development of similar information |
| nda-13 | Permissible acquirement of similar information |
| nda-15 | No licensing |
| nda-16 | Return of confidential information |
| nda-17 | Permissible copy |
| nda-18 | No solicitation |
| nda-19 | Survival of obligations |
| nda-20 | Permissible post-agreement possession |
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
3
4model_name = "Agreemind/contractnli-legalbert-nda-weighted"
5tokenizer = AutoTokenizer.from_pretrained(model_name)
6model = AutoModelForSequenceClassification.from_pretrained(model_name)
7
8hypothesis = "All Confidential Information shall be expressly identified by the Disclosing Party."
9premise = "Section 2.1: Any information disclosed must be marked as Confidential..."
10
11inputs = tokenizer(hypothesis, premise, return_tensors="pt", truncation=True, max_length=512)
12with torch.no_grad():
13 logits = model(**inputs).logits
14 probs = torch.softmax(logits, dim=-1)
15 pred = ["Entailment", "Contradiction", "NotMentioned"][probs.argmax()]
16 print(f"Prediction: {pred} (confidence: {probs.max():.3f})")nlpaueb/legal-bert-base-uncased1@inproceedings{koreeda-manning-2021-contractnli,
2 title = "ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts",
3 author = "Koreeda, Yuta and Manning, Christopher",
4 booktitle = "Findings of EMNLP 2021",
5 year = "2021",
6}