You can use this model directly with a pipeline for masked language modeling:
python
1from transformers import pipeline
2# 1. load the model with the huggingface `pipeline`3genius = pipeline("text2text-generation", model='beyond/genius-large', device=0)4# 2. provide a sketch (joint by <mask> tokens)5sketch ="<mask> Conference on Empirical Methods <mask> submission of research papers <mask> Deep Learning <mask>"6# 3. here we go!7generated_text = genius(sketch, num_beams=3, do_sample=True, max_length=200)[0]['generated_text']8print(generated_text)
If you find our paper/code/demo useful, please cite our paper:
@article{guo2022genius,
title={GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation},
author={Guo, Biyang and Gong, Yeyun and Shen, Yelong and Han, Songqiao and Huang, Hailiang and Duan, Nan and Chen, Weizhu},
journal={arXiv preprint arXiv:2211.10330},
year={2022}
}
