UK Immigration Rulings Outcome Classifier
This deberta-v3-large model is fine-tuned to classify the outcomes of the rulings of the UK Upper Tribunal (Immigration and Asylum Chamber) using hand-coded text data from this court.
The model predicts whether an input text sequence indicates appeal success, failure or neither.
Out-of-sample final test metrics:
accuracy: 0.9289
matthews_correlation: 0.8927
We have not tested how well the classifier generalizes to any other court.
Fine-tuning parameters
The base model is deberta-v3-large. It was fine-tuned using the following parameters on a single A100 80GB GPU:
learning_rate=2e-5,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
num_train_epochs=8,
weight_decay=0.01,
evaluation_strategy="epoch"
Data
The training data comes from the website of the UK Upper Tribunal (Immigration and Asylum Chamber):
https://tribunalsdecisions.service.gov.uk/utiac/Court decisions were first split into paragraphs. In the second step, paragraphs likely containing statement of outcome were identified using regular expressions in an over-inclusive manner.
Paragraphs (split if longer than 700 characters) were subsequently hand-coded by a single researcher for indicating appeal success, failure or neither.
Appeal success means that the Upper Tribunal sets aside (in full or in part) the decision of the First-tier Tribunal (FtT).
Appeal failure means that the Upper Tribunal upholds the FtT decision. neither indicates an irrelevant or inconclusive paragraph.
For example, the following paragraph is coded as success:
44. The determination of the First-tier Tribunal is set aside. The following decision is substituted: the appeal, as originally brought to the FtT, is allowed.
The hand-coded data was split 80-10-10 into a training, evaluation and test set.
Training set size: 3929
Evaluation set size: 491
Test set size: 492
