Parameter	Value
LoRA Rank	64
LoRA Alpha	128
Target Modules	att_proj, attn_out, ff_proj, ff_out
LoRA Dropout	0.05
Quantization	4-bit NF4 (QLoRA)
Trainable Params	~46.1M
Learning Rate	2e-4 (cosine)
Batch Size	8 (grad accum 4)

Best Performance (Step 1300, Epoch 3.72)

Metric	Value
AUC	0.8736
AP	0.8735
F1	0.6857
Accuracy	0.736

AUC Trajectory

Step	Epoch	AUC	F1
100	0.29	0.6146	0.6967
200	0.57	0.7620	0.7399
300	0.86	0.8101	0.7069
400	1.14	0.8244	0.7172
500	1.43	0.8359	0.6844
600	1.72	0.8381	0.6697
700	2.00	0.8166	0.5854
800	2.29	0.8015	0.3593
900	2.57	0.8224	0.7445
1000	2.86	0.8213	0.5407
1100	3.14	0.8227	0.7000
1200	3.43	0.7214	0.3933
1300	3.72	0.8736	0.6857 BEST
1400	4.00	0.8267	0.7701
1500	4.29	0.8389	0.7795
1600	4.57	0.8048	0.5260
1700	4.86	0.7646	0.4451
1800	5.14	0.7948	0.4070
1900	5.43	0.8115	0.5714
2000	5.72	0.7920	0.5168
2100	6.00	0.7985	0.5528
2200	6.29	0.7355	0.3631
2300	6.57	0.7689	0.3873

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained('yahmaachi/omni-dna-multitask-1b')
tokenizer = AutoTokenizer.from_pretrained('yahmaachi/omni-dna-multitask-1b')
tokenizer.add_tokens(['HGT_detection'])
base.resize_token_embeddings(len(tokenizer))

model = PeftModel.from_pretrained(base, 'Nhoodie/omni-dna-hgt-lora-best')
model.eval()

dna = 'ATGCGATCGATCGATCGATC...'  # your sequence
inputs = tokenizer(dna + 'HGT_detection', return_tensors='pt')
with torch.no_grad():
    logits = model(**inputs).logits[:, -1, :]
    prob = torch.softmax(logits, dim=-1)
    # token 4097 = 1 (HGT), token 4096 = 0 (not HGT)
    hgt_prob = prob[0, 4097].item()
print(f'HGT probability: {hgt_prob:.4f}')

Training Notes

The training exhibited an interesting collapse-recovery pattern: after initial overfitting around epoch 2-3.4, the novel token (HGT_detection, #4117) underwent a representational reorganization, leading to a new best AUC of 0.8736 at step 1300.

Steps 1300 and 1500 are sibling models - different representational equilibria rather than descendant relationships. Step 1300 optimizes ranking (AUC), step 1500 optimizes classification (F1).

License

MIT