This repository contains model files for the deberta-v3-large variant of RAGulator. Code can be found here.
Key Points
RAGulator predicts whether a sentence is out-of-context (OOC) from retrieved text documents in a RAG setting.
We preprocess a combination of summarisation and semantic textual similarity datasets (STS) to construct training data using minimal
resources.
We demonstrate 2 types of trained models: tree-based meta-models trained on features engineered on preprocessed text, and BERT-based classifiers fine-tuned directly on original text.
We find that fine-tuned DeBERTa is not only the best-performing model under this pipeline, but it is also fast and does not require additional text preprocessing or feature engineering.
Model Details
Dataset
Training data for RAGulator is adapted from a combination of summarisation and STS datasets to simulate RAG:
The datasets were transformed before concatenation into the final dataset. Each row of the final dataset consists [sentence, context, OOC label].
For summarisation datasets, transformation was done by randomly pairing summary abstracts with unrelated articles to create OOC pairs, then sentencizing
the abstracts to create one example for each abstract sentence.
For STS datasets, transformation was done by inserting random sentences from the datasets to one of the sentences in the pair to simulate a long "context". The original labels were mapped to our OOC definition. If the original pair was indicated as dissimilar, we consider the pair as OOC.
To enable training of BERT-based classifiers, each training example was split into sub-sequences of maximum 512 tokens. The OOC label for each sub-sequence was derived through a generative labelling process with Llama-3.1-70b-Instruct.
Model Training
RAGulator is fine-tuned from microsoft/deberta-v3-large (He et al., 2023).
Model Performance
We compare our models to LLM-as-a-judge (Llama-3.1-70b-Instruct) as a baseline. We evaluate on both a held-out data split of our simulated RAG dataset, as well as an out-of-distribution collection of private enterprise data, which consists of RAG responses from a real use case.
The deberta-v3-large variant is our best-performing model, showing a 19% increase in AUROC and a 17% increase in F1 score despite being significantly smaller than Llama-3.1.
Basic Usage
python
1import torch
2from transformers import DebertaV2Tokenizer, DebertaV2ForSequenceClassification
34model_path ="./ragulator-deberta-v3-large"# assuming model folder located here5tokenizer = DebertaV2Tokenizer.from_pretrained(model_path)6model = DebertaV2ForSequenceClassification.from_pretrained(7 model_path,8 num_labels=29)10model.eval()1112# input13sentences =["This is the first sentence","This is the second sentence"]14contexts =["This is the first context","This is the second context"]15inputs = tokenizer(16 sentences,17 contexts,18 add_special_tokens=True,19 return_token_type_ids=True,20 return_attention_mask=True,21 padding='max_length',22 max_length=512,23 truncation='longest_first',24 return_tensors='pt'25)2627# forward pass28with torch.no_grad():29 outputs = self.model(**inputs)3031# OOC score32fn = torch.nn.Softmax(dim=-1)33ooc_scores = fn(outputs.logits).cpu().numpy()[:,1]
Usage - batch and long-context inference
We provide a simple wrapper to demonstrate batch inference and accommodation for long-context examples. First, install the package:
1from ragulator import RAGulator
23model = RAGulator(4 model_name='deberta-v3-large',# only value supported for now5 batch_size=32,6 device='cpu'7)89# input10sentences =["This is the first sentence","This is the second sentence"]11contexts =["This is the first context","This is the second context"]1213# batch inference14model.infer_batch(15 sentences,16 contexts,17 return_probas=True# True for OOC probabilities, False for binary labels18)
Citation
@misc{poey2024ragulatorlightweightoutofcontextdetectors,
title={RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generation},
author={Ian Poey and Jiajun Liu and Qishuai Zhong and Adrien Chenailler},
year={2024},
eprint={2411.03920},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2411.03920},
}
