MsAlEhR/

MetaBERTa-bigbird-gene

• Base Model: BigBird transformer architecture
• Tokenizer: Custom K-mer Tokenizer with k-mer length of 6 and overlapping tokens
• Training: Trained on a diverse dataset of 497 housekeeping genes from 2000 bacterial and archaeal genomes
• Embeddings: Generates sequence embeddings using mean pooling of hidden states

1. Install KmerTokenizer:
2. pip install git+https://github.com/MsAlEhR/KmerTokenizer.git
3. Example Code:
   python

   1 from KmerTokenizer import KmerTokenizer 2 from transformers import AutoModel 3 import torch 4 5 # Example gene sequence 6 seq = "ATTTTTTTTTTTCCCCCCCCCCCGGGGGGGGATCGATGC" 7 8 # Initialize the tokenizer 9 tokenizer = KmerTokenizer(kmerlen=6, overlapping=True, maxlen=4096) 10 tokenized_output = tokenizer.kmer_tokenize(seq) 11 pad_token_id = 2 # Set pad token ID 12 13 # Create attention mask (1 for tokens, 0 for padding) 14 attention_mask = torch.tensor([1 if token != pad_token_id else 0 for token in tokenized_output], dtype=torch.long).unsqueeze(0) 15 16 # Convert tokenized output to LongTensor and add batch dimension 17 inputs = torch.tensor([tokenized_output], dtype=torch.long) 18 19 # Load the pre-trained BigBird model 20 model = AutoModel.from_pretrained("MsAlEhR/MetaBERTa-bigbird-gene", output_hidden_states=True) 21 22 # Generate hidden states 23 outputs = model(input_ids=inputs, attention_mask=attention_mask) 24 25 # Get embeddings from the last hidden state 26 embeddings = outputs.hidden_states[-1] 27 28 # Expand attention mask to match the embedding dimensions 29 expanded_attention_mask = attention_mask.unsqueeze(-1) 30 31 # Compute mean sequence embeddings 32 mean_sequence_embeddings = torch.sum(expanded_attention_mask * embeddings, dim=1) / torch.sum(expanded_attention_mask, dim=1) 33

Refahi, M.S., Sokhansanj, B.A., & Rosen, G.L. (2023). Leveraging Large Language Models for Metagenomic Analysis. IEEE SPMB.

Refahi, M., Sokhansanj, B.A., Mell, J.C., Brown, J., Yoo, H., Hearne, G. and Rosen, G., 2025. Enhancing nucleotide sequence representations in genomic analysis with contrastive optimization Communications Biology, Nature.