Supervised instruction-tuning of Qwen2.5 7B for contract clause identification and extraction across 41 legally significant clause types from the CUAD benchmark.
Task
Dataset
Models
Domain
Overview
Models are fine-tuned to produce a structured response: a binary Yes/No decision on clause presence, followed by the verbatim quotation of the relevant clause. This reframes CUAD's span-extraction task as an instruction-following problem, allowing the model to leverage parametric legal knowledge while remaining grounded in the provided contract excerpt.
BERTScore uses Legal-BERT as the scoring backbone with IDF weighting and baseline rescaling.
Key finding: QLoRA achieves near-parity with full-precision LoRA at significantly lower GPU memory cost — confirming that 4-bit base weight compression does not meaningfully degrade clause identification quality at the 7B scale.
Fine-tuning gain vs zero-shot
Overall score
Qwen2.5 7B (zero-shot)
0.479
Qwen2.5 7B LoRA (fine-tuned)
0.760
Gain
+0.281
At the 7B scale, SFT refines behaviour qualitatively — teaching the model to ground responses in the specific contract text rather than generalised legal knowledge, producing more faithful and legally precise extractions.
Training setup
Parameter
Value
LoRA rank / alpha
r=16, α=16
LoRA target
All linear layers
Context windowing
10,000-char answer-aware window
Data split
95 / 2.5 / 2.5 (train / val / test)
Attention
Flash Attention 2
Kernel optimisation
Liger kernel fusion
Memory
Gradient checkpointing + sequence packing
QLoRA quantisation
4-bit + double quantisation (bfloat16 adapters)
Known Limitations
Temporal / date clauses
Subword tokenisation fragments date strings, causing models to identify the correct sentence but predict a generic label (e.g. "Effective Date") rather than the precise date token. This is a tokeniser-level limitation that fine-tuning alone cannot resolve.
Autoregressive repetition loop
Party name extraction can trap the model in a token-repetition loop until the context limit is reached, particularly when the reference contains multiple semicolon-separated entity names.
Exposure bias / cascade errors
A wrong Yes/No gate token propagates forward with no corrective signal — the model can confabulate a plausible but incorrect clause extraction (e.g. a hallucinated date) with full confidence.
Evaluation notes
Length bias is present in all fine-tuned models (negative Pearson correlation between reference length and scores). The 7B QLoRA large-batch variant shows the weakest bias (r = −0.133 / −0.119).
BERTScore floor: a rescaled score near zero indicates semantic orthogonality, not a random prediction. A corpus-specific CUAD baseline would provide a more appropriate rescaling anchor.
Batch size effect: larger batch sizes produce a weaker 7B QLoRA model, suggesting gains are driven by the number of gradient updates rather than update quality.
