This model is an unintended but valuable research artifact. It was produced across multiple interrupted training runs with known bugs. Its behavior is unpredictable and it should only be used for research into hierarchical LoRA architectures.

Architecture: Hierarchical Stacked LoRA (HSL) — "Harmonics"

3 sequential LoRA levels (rank=16 each) stacked on Omni-DNA-20M:

h = W_base(x)                    # frozen base model
h = h + B₀A₀(x)                  # Level 0: weight-space (fundamental)
h = h + B₁A₁(h)                  # Level 1: activation-space (1st harmonic)
h = h + B₂A₂(h)                  # Level 2: activation-space (2nd harmonic)

Effective rank: 16³ = 4,096 via sequential composition (not additive)
Level 0 finds the fundamental representation (converges first)
Level 1 refines on top of Level 0 output (mid-frequency)
Level 2 captures residual signal (highest frequency / rarest patterns)
Each level receives the accumulated output of all previous levels

Key Finding: Per-Layer Gradient Handling

This is a heterogeneous architecture — each level operates in a different representational space. Global gradient clipping destroys learning by compressing all layers through a single bottleneck. Per-layer gradient norms must be tracked independently.

Training History (The Messy Truth)

Run	Steps	Data Batches	Why It Died
1	0→4356	17,424	OOM (eval memory leak + long sequence attention)
2	4356→5742	5,544	Killed to deploy log timestamps
3	5742→7950	8,832	OOM (same eval leak pattern)
4	7920→9999	8,316	Killed at completion

Total: ~39,516 batches ≈ 5.0 effective epochs (but across 4 interrupted runs with reshuffled data due to broken epoch tracking — each restart reshuffles from epoch 0)

Known Bugs in Training Code

Epoch tracking broken on resume: always starts epoch loop at 0. Fix: start_epoch = resume_step // steps_per_epoch
Memory leak in eval: ~6MB leaked per eval from CUDA allocator fragmentation. gc.collect + empty_cache slows but does not stop it.
VRAM log shows PyTorch-allocated only (~72MB), not real nvidia-smi total (~2.3GB). Gap is CUDA allocator fragmentation.
expandable_segments:True must be set via tmux, not export VAR=val && nohup (env var is lost)

Performance at Step 9999

Token	AUC	F1	Samples (pos)
HGT_prok_host	0.7855	0.0000	11
HGT_arch_euk	0.6417	0.0000	13
HGT_detection	0.6290	0.0000	97
HGT_fungi_euk	0.5118	0.0000	81
HGT_bac_euk	0.4557	0.0149	133
HGT_prot_euk	0.4393	0.0625	62
Overall	0.5140	—	—

Best overall AUC: 0.5745 at step 1287 (early training before snap-collapse oscillations)

What Worked

Ranking (AUC) learned well — prok_host at 0.79 with only 11 positive samples
Hierarchical emergence observed: detection → prok_host → fungi_euk → arch_euk (rarest last)
Level 0 converges first, then resumes learning when upper levels create new signal
Per-layer gradient handling kept training alive where global clipping failed
101 successful evals before OOM

What Did Not Work

F1 never stabilized — logit rescaling phase never completed across 5 effective epochs
Decision thresholds never calibrated — model ranks well but cannot classify
Memory leak capped training length at ~8000 steps per uninterrupted run

Task Tokens

Token ID	Token	Type
4096	`0`	Label
4097	`1`	Label
4098	`2`	Label
4117	`HGT_detection`	Binary HGT detection
4119	`HGT_bac_euk`	Bacteria→Eukaryote transfer
4120	`HGT_arch_euk`	Archaea→Eukaryote transfer
4121	`HGT_fungi_euk`	Fungi→Eukaryote transfer
4122	`HGT_prot_euk`	Protist→Eukaryote transfer
4123	`HGT_prok_host`	Prokaryotic host transfer

Configuration

python

BASE_MODEL = "zehui127/Omni-DNA-20M"  # expanded to omni-dna-20m-hgt
NUM_LEVELS = 3
LORA_R = 16
LORA_ALPHA = 32
TARGET_MODULES = ["att_proj", "attn_out", "ff_proj", "ff_out"]
LEARNING_RATE = 2e-4
GRAD_ACCUM = 4
MAX_LENGTH = 4096
MAX_TOKENS_PER_BATCH = 4096  # 1 sequence per batch
TRAINABLE_PARAMS = 2,693,312  # 20.41% of model

Datasets

IslandViewer4: Genomic island detection (binary)
euHGT: Cross-domain transfer type classification
DeepHGT: Large-scale HGT dataset

Total: 95,344 train / 23,837 eval

Code

Training script and full eval history available at: https://github.com/NamelessHoodie/dnabert-mut-exp/tree/dev/sad-omni-hyena/harmonics/

Citation

If you use this architecture or findings, please reference:

Architecture: "Harmonics" — Hierarchical Stacked LoRA with sequential composition
Base model: Omni-DNA by Zehui et al.
Datasets: IslandViewer4, euHGT, DeepHGT

Nhoodie/omni-dna-20m-hgt-stacked-lora

Omni-DNA-20M HGT Stacked LoRA — Harmonics Architecture

⚠️ Research Artifact — NOT Recommended for Production Use