
architecture	BERT, 24 layers, 768 hidden, 12 heads
parameters	~430M
input	1000 bp DNA sequence (ACGT)
output	768-dim embedding per position
pretraining	metagenomic contigs + microbial genomes

Usage

python

import tensorflow as tf
from huggingface_hub import hf_hub_download

# download model
model_path = hf_hub_download(
    repo_id="genomenet/bert-metagenome",
    filename="bert_1k_3.h5"
)

# load with custom objects (if needed)
model = tf.keras.models.load_model(model_path, compile=False)

# get embeddings from layer 21
embedding_model = tf.keras.Model(
    inputs=model.input,
    outputs=model.get_layer("layer_transformer_block_21").output
)

# input: one-hot encoded DNA (batch, 1000, 4)
# output: embeddings (batch, 1000, 768)

Tokenization

DNA sequences are one-hot encoded:

A = [1, 0, 0, 0]
C = [0, 1, 0, 0]
G = [0, 0, 1, 0]
T = [0, 0, 0, 1]
N = [0.25, 0.25, 0.25, 0.25]

Input shape: (batch_size, 1000, 4)

Applications

CRISPR array detection (fine-tuned version: genomenet/crispr-array-detection)
Sequence classification
Metagenome binning
Functional annotation
Sequence similarity search via embeddings

Acknowledgements

BMBF de.NBI / GenomeNet
DFG SPP 2141
Helmholtz Centre for Infection Research (HZI)