Legal pretrained model using Replaced Token Detection (RTD) task, trained on Pile-of-Law dataset with 4096 tokens as context windows.

Legal pretrained model using ELECTRA objective task, trained on Pile-of-Law dataset with 4096 tokens as context windows.

The model can directly be used to generate embeddings for example for similarity search. It likely works best on US focused legal data.

The model can be finetuned for any NLU task or when coupled with a decoder also for generative tasks. In our experiments on summarization with the BillSum dataset, we found that random initialization of the decoder improved performance.

This model will likely work worse on non-legal text in non-English languages originating from outside the US.

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.

As described in the dataset card, the internal variation allows contextual privacy rules to be learned. If robust mechanisms for this are developed they can applied more broadly. As discussed in ``On the Opportunities and Risks of Foundation Models'', legal language models can help improve access to justice in various ways. But they can also be used in potentially harmful ways. While such models are not ready for most production environments and are the subject of significant research, we ask that model users and model creators using this model, particularly when creating generative models (e.g. attaching a decoder), consider the impacts of their model and make a good faith effort to weigh the benefits against the harms of their method. As our license, the training dataset license also restricts commercial usage.

The data reflects the biases of governments and courts. As discussed in their work Pile of Law, these can be significant, though more recent text will likely be less overtly toxic. Please consult the above statement and keep it in mind in the use and/or any modification of this model, implementing responsible use.

As with any large LM there is the risk of it producing biased or unfair output. Researchers using the model should put into place respective safeguards to identify biased and/or toxic language.

The diverse model was trained on caselaw (“Court Listener Opinions” & “Court Listener Docket Entry Documents”), legislation (“US Code”, “State Codes” & “EURLEX”) and contracts (“Atticus Contracts” & “EDGAR Contracts”) from public Pile-of-Law dataset. To balance the training data, we limited the number of documents to 500K (this affects Court Listener Opinions, Court Listener Docket Entry Documents and EDGAR Contracts).

More information needed

More information needed

We tested the model on the BillSum and PubMed summarization datasets achieving SotA Rouge scores for the respective parameter sizes in August 2022.

More information needed

We followed the standard in research on summarization datasets and used Rouge 1, 2 and L.

More information needed

Carbon emissions estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

We used a Longformer attention window of 256 as generator and discriminator. The generator model was three times smaller than the discriminator model. In particular, the generator’s depth (number of hidden layers) instead of its width (embedding size, hidden size and intermediate size). We used a MLM probability of 25% for the generator.

Amazon SageMaker Notebooks.

4 x 16GB NVIDIA V100

transformers

@misc{niklaus2022budgetlongformer, title={BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch?}, author={Joel Niklaus and Daniele Giofré}, year={2022}, eprint={2211.17135}, archivePrefix={arXiv}, primaryClass={cs.CL} }

Joel Niklaus, Daniele Giofré