The model is trained on bi-modal data (documents & code) of
CodeSearchNetThis model is initialized with Roberta-base and trained with MLM+RTD objective (cf. the paper).
Please see
the official repository for scripts that support "code search" and "code-to-document generation".
1@misc{feng2020codebert,
2 title={CodeBERT: A Pre-Trained Model for Programming and Natural Languages},
3 author={Zhangyin Feng and Daya Guo and Duyu Tang and Nan Duan and Xiaocheng Feng and Ming Gong and Linjun Shou and Bing Qin and Ting Liu and Daxin Jiang and Ming Zhou},
4 year={2020},
5 eprint={2002.08155},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}
