LongSafari/hyenadna-tiny-1k-seqlen-hf, a 436K-parameter Hyena (implicit long-convolution) genomic foundation model, fine-tuned for binary classification: is a 251bp human DNA sequence a non-TATA promoter, or a non-promoter fragment from a gene body.

This is a learning/portfolio project, not a validated diagnostic or production model. Full methodology, code, and figures: github.com/Arun0364/genomic-finetune.

katarinagresova/Genomic_Benchmarks_human_nontata_promoters, part of the GenomicBenchmarks suite (Gresova et al. 2023). Positive sequences span -200bp to +50bp around a transcription start site lacking a TATA box; negatives are random 251bp fragments from human gene regions after the first exon.

The official train/test split is a plain row-level random split with no chromosome or locus grouping. An audit found the negative class is about 94% near-duplicated across train/test (overlapping genomic windows from the same gene land on both sides), which would inflate eval metrics without the model learning real promoter biology. This checkpoint was fine-tuned on a rebuilt, leakage-free split instead: sequences clustered by shared 50-mers via union-find, then whole clusters assigned to train or test so no overlapping window crosses the split (26,648 train / 9,483 test, 54.4% positive in both, 0% exact or near-duplicate overlap verified). Because of this, results here are not directly comparable to published GenomicBenchmarks leaderboard numbers on this dataset.

HuggingFace Trainer, batch size 32, learning rate 5e-5, EarlyStoppingCallback(early_stopping_patience=1) on validation ROC-AUC. Training stopped itself after 3 epochs; the epoch-2 checkpoint was restored via load_best_model_at_end.

On the held-out, leakage-corrected test set:

A 6-mer frequency + LogisticRegressionCV baseline on the identical split scores accuracy 0.799, precision 0.894, recall 0.715, F1 0.794, ROC-AUC 0.870. This model beats the baseline on accuracy, recall, F1, and ROC-AUC, but the margin is modest, not dramatic, on a compositionally-driven task like this a simple k-mer baseline is a reasonably strong competitor.

The model has no attention mechanism, so interpretation used in-silico mutagenesis (ISM) first. Naive ISM turned out to be dominated by two structural confounds: proximity to the model's last-token pooling position, and a class-asymmetric sensitivity to mutation not fully explained by softmax saturation. A different method, applying the model's own classification head at every position instead of only the pooled last token (a position-wise adaptation of the "logit lens" technique), resolved this: it found that class-relevant evidence accumulates over roughly positions 20-61 of the 251bp window (upstream of the TSS), then plateaus for the rest of the sequence, including through the region ISM had misleadingly flagged as most important. A JASPAR motif scan of that region found SP1 (the canonical GC-box regulator of non-TATA, CpG-island promoters) enriched above chance with strong match quality, a plausible if exploratory biological explanation for the signal.

Full writeup with figures: github.com/Arun0364/genomic-finetune.

If referencing this checkpoint, cite the underlying HyenaDNA paper (Nguyen et al. 2023, arXiv:2306.15794) and the GenomicBenchmarks paper (Gresova et al. 2023, BMC Genomic Data), listed in full at github.com/Arun0364/genomic-finetune.