Evaluation

We evaluate on PubMedQA, BioASQ (Yes/No), and MedQA (USMLE 4-option). MedQA is not included in the SFT data — the model is run on MedQA in a transfer (non-in-distribution) setting, which is why this family is named biomedqa rather than medqa.

Loading

The checkpoint is saved in mamba-ssm native format, load with mamba-ssm:

python

from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel
from transformers import AutoTokenizer

model = MambaLMHeadModel.from_pretrained("zmzfpc/biomamba-biomedqa-sft-780m", device="cuda", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("zmzfpc/biomamba-biomedqa-sft-780m")

AutoModelForCausalLM.from_pretrained will not work on this config.

Intended use

Biomedical Yes/No/Maybe QA and transfer to MedQA-style multiple-choice. Not intended for clinical decision-making and not validated on any patient-facing task.

Citation

Paper: BioMamba: Domain-Adaptive Biomedical Language Models (arXiv:2408.02600). Code: https://github.com/LeoYML/BioMamba

bibtex

@article{yue2024biomamba,
  title   = {{BioMamba}: Domain-Adaptive Biomedical Language Models},
  author  = {Yue, Ling and Zhu, Mingzhi and Xing, Sixue and Pan, Shaowu and
             Chenthamarakshan, Vijil and Wang, Yanbo and Cao, Yunning and
             Das, Payel and Fu, Tianfan},
  journal = {arXiv preprint arXiv:2408.02600},
  year    = {2024}
}