Codebert with injection of vocabulary

• Bio.tools

J. Ison, K. Rapacki, H. Ménager, M. Kalas, E. Rydza, P. Chmura, C. Anthon, N. Beard, K. Berka, D. Bolser, T. Booth, A. Bretaudeau, J. Brezovsky, R. Casadio, G. Cesareni, F. Coppens, M. Cornell, G. Cuccuru, and et al. Tools and data services registry: a community effort to document bioinformatics resources. Nucleic Acids Research,44(D1):D38–D47, Jan. 2016. ISSN 0305-1048. doi: 10.1093/nar/gkv1116. URL https://doi.org/10.1093/nar/gkv1116.

• Bioconda

B. Grüning, R. Dale, A. Sjödin, B. A. Chapman, J. Rowe, C. H. Tomkins-Tinch, R. Valieris, and J. Köster. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nature Methods, 15 (7):475–476, July 2018. ISSN 1548-7105. doi: 10.1038/s41592-018-0046-7. URL https://www.nature.com/articles/s41592-018-0046-7.

• Biocontainers

F. da Veiga Leprevost, B. A. Grüning, S. Alves Aflitos, H. L. Röst, J. Uszkoreit, H. Barsnes, M. Vaudel, and et al. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics, 33 (16):2580–2582, Aug. 2017. ISSN 1367-4803. doi: 10.1093/bioinformatics/btx192. URL https://doi.org/10.1093/bioinformatics/btx192.

• and Bioweb (Knowledge base manually curated at Institut Pasteur, France)

Vocabulary Injection Strategy

• Tokenizer Update: new words are added to the existing tokenizer of CodeBert
• Embedding Initialization: embeddings for newly introduced words are initialized as the average of the embeddings of their constituent existing words

Model Adaptation

• Option 1: Fine-Tuning (Recommended): the language model is fine-tuned on domain-specific corpora to better integrate the new vocabulary into its internal representations
• Option 2: No Fine-Tuning: the extended model is used directly without further training

Use in

• https://github.com/ClemenceS/nlstruct
• Other experiments are presented in https://gitlab.liris.cnrs.fr/sharefair/copalink-experiments