Legal Risk Classifier — Mistral 7B + LoRA
A LoRA fine-tuned adapter for Mistral 7B Instruct v0.3 that classifies commercial
contract clauses by type and risk level, and generates a plain-English explanation —
all in a single structured JSON output.
Model Description
This model takes a contract clause as input and returns:
- Clause type — one of 41 legal categories (e.g. Indemnification, Governing Law,
Limitation of Liability)
- Risk level — High, Medium, or Low
- Summary — a one-sentence plain-English explanation
- Reason — why the clause carries that risk level
It was fine-tuned using LoRA (Low-Rank Adaptation) on top of the frozen
Mistral 7B Instruct v0.3 base model, trained on the CUAD (Contract Understanding
Atticus Dataset) legal clause dataset.
Why a decoder model instead of an encoder (e.g. BERT)?
Encoder-only models output a single label. This task requires three simultaneous
outputs — classification, risk assessment, and natural-language explanation — in one
coherent response. A decoder-based generative model produces all three in a single
forward pass. Mistral 7B was additionally pre-trained on a broad corpus that includes
legal text, giving it strong zero-shot legal reasoning before any fine-tuning was applied.
Example Output
1{
2 "clause_type": "Indemnification",
3 "risk_level": "High",
4 "summary": "You are responsible for all losses the other party suffers.",
5 "reason": "Broad unlimited indemnification — no liability cap specified."
6}
Training Details
Training Data
CUAD — an expert-annotated
legal dataset produced by the Atticus Project. 510 real commercial contracts annotated
by practising lawyers and supervised law students across 41 clause categories.
- Raw examples: 84,000+ question–answer pairs
- After filtering to positive (non-empty) examples: ~4,000 genuine clause instances
- Train / test split: 95% / 5%, shuffled with seed 42 for reproducibility
Risk-level labels (High / Medium / Low) are not present in CUAD and were added as
part of this project, based on legal exposure principles:
| Risk Level | Example Clause Types |
|---|
| High | Indemnification, Unlimited Liability, IP Ownership, Termination for Convenience, Non-Compete |
| Medium | Limitation of Liability, Governing Law, Payment Terms, Warranty |
| Low | Confidentiality, Notice Period, Audit Rights |
Unmapped clause types default to Medium as a conservative measure.
Training Procedure
- Base model: mistralai/Mistral-7B-Instruct-v0.3
- Method: LoRA via HuggingFace PEFT
- Hardware: NVIDIA DGX Spark (GB10, Blackwell architecture, 128GB unified memory)
- Precision: BF16 (full precision — no quantisation)
- LoRA rank (r): 32
- LoRA alpha: 64
- Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- Epochs: 3
- Batch size: 4 (gradient accumulation 4 → effective batch size 16)
- Learning rate: 2e-4
- Trainable parameters: ~85M of 7.2B total (1.2%)
- Adapter size: ~150MB
Evaluation
Evaluated on 454 held-out test examples never seen during training
(High: 78, Medium: 325, Low: 51).
Results — Before vs After Fine-Tuning
| Metric | Zero-Shot (Before) | Fine-Tuned (After) | Improvement |
|---|
| Overall F1 (weighted) | 67.36% | 88.71% | +21.35 pts |
| Accuracy | 67.62% | 88.99% | +21.37 pts |
| High Risk F1 | 47.29% | 76.81% | +29.52 pts |
| High Risk Recall | 61.54% | 67.95% | +6.41 pts |
| Medium Risk F1 | 78.06% | 92.63% | +14.57 pts |
| Medium Risk Recall | 76.62% | 94.77% | +18.15 pts |
| Low Risk F1 | 29.85% | 81.90% | +52.05 pts |
| Low Risk Recall | 19.61% | 84.31% | +64.70 pts (+330% relative) |
| JSON Parse Success | 99.34% | 100.00% | Perfect |
| Inference Failures | 3 | 0 | Zero failures |
| Avg Latency / clause | 13.85s | 11.33s | Faster |
Comparison With Published Baselines
| Method | Score | Source |
|---|
| RoBERTa-base | 8.2% P@80R | CUAD paper (Hendrycks et al. 2021) |
| DeBERTa-xlarge | 44.0% P@80R | CUAD paper (Hendrycks et al. 2021) |
| Best zero-shot LLM | ~52% F1 | ContractEval 2025 |
| Mistral 7B zero-shot (this model, pre-tuning) | 67.36% F1 | This project |
| Mistral 7B + LoRA (this model) | 88.71% F1 | This project |
Key Findings
- Low Risk Recall improved by 330% relative (19.6% → 84.3%) — the most dramatic
improvement. Before fine-tuning the base model rarely identified genuinely low-risk
clauses, which in production creates alert fatigue. After fine-tuning it can
confidently clear low-risk clauses without unnecessary human review.
- JSON reliability reached 100% with zero inference failures, critical for any
production API integration.
- Zero-shot Mistral 7B already exceeded the best published zero-shot result in
ContractEval 2025 (~52% F1), suggesting strong innate legal reasoning from
pre-training alone.
- High Risk Recall remains the hardest category (67.95%), reflecting class
imbalance in CUAD (78 High Risk vs 325 Medium Risk test examples). In production,
all High Risk classifications should be routed to mandatory human review regardless
of model confidence.
How to Use
1from peft import PeftModel
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4base_model = AutoModelForCausalLM.from_pretrained(
5 "mistralai/Mistral-7B-Instruct-v0.3",
6 torch_dtype="bfloat16",
7 device_map="auto"
8)
9model = PeftModel.from_pretrained(base_model, "Govardhan12345/legal-risk-classifier-lora")
10tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
11
12prompt = """<s>[INST] You are a legal risk analyst specialising in commercial contracts.
13
14Analyse the following contract clause:
15The contractor shall indemnify and hold harmless the employer from all claims.
16
17Respond ONLY with a valid JSON object:
18{
19 "clause_type": "...",
20 "risk_level": "High or Medium or Low",
21 "summary": "...",
22 "reason": "..."
23} [/INST]"""
24
25inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
26output = model.generate(**inputs, max_new_tokens=300, temperature=0.1)
27print(tokenizer.decode(output[0], skip_special_tokens=True))
Intended Use
This model is designed as a decision-support tool for identifying and summarising
contract risk — it is not a substitute for legal advice. High Risk classifications and
edge cases should always be reviewed by a qualified professional before any contractual
decision is made.
Limitations
- Trained on CUAD, which consists of US commercial contracts — coverage of other
jurisdictions or specialised contract types (e.g. UK construction subcontracts) may
be limited without further domain-specific fine-tuning.
- High Risk Recall (67.95%) is lower than Medium and Low Risk due to class imbalance
in the training data.
- Outputs should be treated as a first-pass risk flag, not a final legal determination.
Framework Versions
- PEFT 0.12.0
- Transformers
- BF16 training on NVIDIA DGX Spark (GB10, Blackwell)
