This model is a fine-tuned T5 model on the Synthetic GEC dataset and filtered CommonCrawl data in English Language. The Base Model(T5) is Pre-trained on C4(Colossal Clean Crawled Corpus) dataset and works with numerous downstrem tasks.
Our Model specifically is fine-tuned on a single downstream task of context correction on the above mentioned two datasets.

Model description

This Model has the same architecture as its base model, thus having 220 Million Parameters while consisting of 12 encoder blocks and 12 decoder blocks with an input embedding size of 32128. Please refer to this link to know more about the model details.

Intended Use & Limitations

As the model is intented to correct the context of the given sentence, all you have to do is pass the non-contextually correct sentence and get the corrected response back.
Based on Multiple experiments performed as part of the training, we observe that the model works best when the the total number of tokens in the input is less than 256.
So, if you have a long paragraph that needs to be context corrected, we suggest to first sentence tokenize the paragraph and run the context corrector for each sentence separately to obtain best results.

Note that the model is primarily trained on general publicly available corpus, so it maynot work well for Medical Contexts.

Usage

You can use this model directly with a pipeline for Text to Text Generation:

python

from transformers import pipeline


ctx_corr = pipeline("text2text-generation", model='DeathReaper0965/t5-context-corrector')
ctx_corr("Do you even know why I always need changed our checking account number")

###########OUTPUT###########
# [{'generated_text': 'Do you even know why I always need to change our checking account number?'}]

Or you can also use the model to get the features for a given text:

python

from nltk import sent_tokenize

from transformers import T5ForConditionalGeneration, T5Tokenizer


# Load model and tokenizer
cc_tokenizer = T5Tokenizer.from_pretrained("DeathReaper0965/t5-context-corrector")
cc_model = T5ForConditionalGeneration.from_pretrained("DeathReaper0965/t5-context-corrector")

# Utility function to correct context
def correct_context(input_text, temperature=0.5):
    # tokenize
    batch = cc_tokenizer(input_text,
                         truncation=True,
                         padding='max_length',
                         max_length=256, 
                         return_tensors="pt")

    # forward pass
    results = cc_model.generate(**batch,
                                max_length=256,
                                num_beams=3,
                                no_repeat_ngram_size=2,
                                repetition_penalty=2.5,
                                temperature=temperature,
                                do_sample=True)
    
    return results

# Utility function to split the paragraph into multiple sentences
def split_and_correct_context(sent):
    sents = sent_tokenize(sent)
    
    final_sents = cc_tokenizer.batch_decode(correct_context(sents), 
                                            clean_up_tokenization_spaces=True, 
                                            skip_special_tokens=True)
    
    final_sents = " ".join([final_sents[i].strip() for i in range(len(final_sents))])
    
    return final_sents


split_and_correct_context("Do you even know why I always need changed our checking account number. Because of the securty purpos.")

###########OUTPUT###########
# 'Do you even know why I always need to change our checking account number? Because of the security purpose.'

Designed and Developed with ♥ by Praneet | LinkedIn | GitHub