The HIV-BERT model was trained as a refinement of the
ProtBert-BFD model for HIV centric tasks. It was refined with whole viral genomes from the
Los Alamos HIV Sequence Database. This pretraining is important for HIV related tasks as the original BFD database contains few viral proteins making it sub-optimal when used as the basis for transfer learning tasks. This model and other related HIV prediction tasks have been published (link).
Like the original
ProtBert-BFD model, this model encodes each amino acid as an individual token. This model was trained using Masked Language Modeling: a process in which a random set of tokens are masked with the model trained on their prediction. This model was trained using the damlab/hiv-flt dataset with 256 amino acid chunks and a 15% mask rate.
As a masked language model this tool can be used to predict expected mutations using a masking approach. This could be used to identify highly mutated sequences, sequencing artifacts, or other contexts. As a BERT model, this tool can also be used as the base for transfer learning. This pretrained model could be used as the base when developing HIV-specific classification tasks.
As this is a BERT-style Masked Language learner, it can be used to determine the most likely amino acid at a masked position.
1from transformers import pipeline
2
3unmasker = pipeline("fill-mask", model="damlab/HIV_FLT")
4
5unmasker(f"C T R P N [MASK] N T R K S I R I Q R G P G R A F V T I G K I G N M R Q A H C")
6
7[
8 {
9 "score": 0.9581968188285828,
10 "token": 17,
11 "token_str": "N",
12 "sequence": "C T R P N N N T R K S I R I Q R G P G R A F V T I G K I G N M R Q A H C"
13 },
14 {
15 "score": 0.022986575961112976,
16 "token": 12,
17 "token_str": "K",
18 "sequence": "C T R P N K N T R K S I R I Q R G P G R A F V T I G K I G N M R Q A H C"
19 },
20 {
21 "score": 0.003997281193733215,
22 "token": 14,
23 "token_str": "D",
24 "sequence": "C T R P N D N T R K S I R I Q R G P G R A F V T I G K I G N M R Q A H C"
25 },
26 {
27 "score": 0.003636382520198822,
28 "token": 15,
29 "token_str": "T",
30 "sequence": "C T R P N T N T R K S I R I Q R G P G R A F V T I G K I G N M R Q A H C"
31 },
32 {
33 "score": 0.002701344434171915,
34 "token": 10,
35 "token_str": "S",
36 "sequence": "C T R P N S N T R K S I R I Q R G P G R A F V T I G K I G N M R Q A H C"
37 }
38]
39
The dataset
damlab/HIV_FLT was used to refine the original
rostlab/Prot-bert-bfd. This dataset contains 1790 full HIV genomes from across the globe. When translated, these genomes contain approximately 3.9 million amino-acid tokens.
As with the
rostlab/Prot-bert-bfd model, the rare amino acids U, Z, O, and B were converted to X and spaces were added between each amino acid. All strings were concatenated and chunked into 256 token chunks for training. A random 20% of chunks were held for validation.
Training was performed with the HuggingFace training module using the MaskedLM data loader with a 15% masking rate. The learning rate was set at E-5, 50K warm-up steps, and a cosine_with_restarts learning rate schedule and continued until 3 consecutive epochs did not improve the loss on the held-out dataset.
