DIPPER is a large model, built from google/t5-efficient-xxl and finetuned on 6.3M datapoints.
I am proposing a lightweight, non-context equivalent for lower-cost usage.
This model is built from google/t5-large-nl32 and finetuned on 100,000 datapoints.
Notably, the datapoints are all non-context. Refer to the original paper if you wish for further understanding on this topic.
The dataset used to finetune this model is available here: Dataset
Model Details
Model Description
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
The model is intended to be used for paraphrasing with notions of control.
The dataset used encourages lexical (word) and order (paragraph structure) parameters, which control the degree of strength in paraphrasing.
See the example code usage for a further understanding.
Direct Use
The model is entirely usable from the uploaded state. No further finetuning is required, although possible.
Downstream Use [optional]
This model was finetuned from a T5 checkpoint.
It is possible to further finetune this model, if desired.
If you plan for transfer learning, I would simply recommend starting from the initial checkpoint model: google/t5-large-nl32.
Recommendations
In terms of recommendation, if you have the capacity, I would recommend using the more powerful model: DIPPER
Otherwise, this model is sufficiently strong.
It outperforms the sentence-based paraphraser ChatGPT Paraphraser when it comes to perplexity scores - when both models are compared using the facebook/opt-2.7b model.
How to Get Started with the Model
Use the code below to get started with the model.
Training Details
Training Data
As mentioned, the training data is here: kpar3-no-ctx
Pre-processing simply contains tokenisation through the google/t5-efficient-large-nl32 tokenizer.
The data is classic paraphrase pairs. However, the first element in the pair has terms "lexical = x" and "order = y".
The values x and y are in the set {0, 20, 40, 60, 80, 100} and denote the strength with which the model should paraphrase.
In particular, a sentence with "lexical = 0" should change as many words as possible, while maintaining the original meaning.
Meanwhile, a sentence with "order = 0" should restructure the paragraph to the model's greatest extent.
The dataset only contains parameter values in increments of 20.
Finetuning on 100,000 datapoints, this took around 14 GPU hours using a GTX 3090.
Example Usage
python
1import torch
2from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
34device = torch.device("cuda"if torch.cuda.is_available()else"cpu")56tokenizer = AutoTokenizer.from_pretrained("google/t5-efficient-large-nl32")78model = AutoModelForSeq2SeqLM.from_pretrained("SamSJackson/paraphrase-dipper-no-ctx")9model = model.to(device)1011text ="Each Wednesdsay, I take my dog for a walk in Central Park."1213lexical =2014order =401516prompt =f"lexical = {lexical}, order = {order}{text}"1718input_ids = tokenizer(19 prompt,20 return_tensors='pt',21 padding="longest",22 max_length=1000,23 truncation=True,24).to(device)2526outputs = model.generate(27**input_ids,28 top_p=0.75,29 do_sample=True,30 max_new_tokens=300,31)3233response = tokenizer.batch_decode(outputs, skip_special_tokens=True)34response =f"{' '.join(response)}"3536print(response)
Citation [optional]
BibTeX:
@misc{krishna2023paraphrasing,
title={Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense},
author={Kalpesh Krishna and Yixiao Song and Marzena Karpinska and John Wieting and Mohit Iyyer},
year={2023},
eprint={2303.13408},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
Model Card Contact
Contact me through huggingface if you have any questions.
