Democratizing Medical LLMs For Much More Languages

🌈 Update

• [2024.10.15] ApolloMoE repo is published！🎉

Languages Coverage

Click to view the Languages Coverage

Architecture

Click to view the MoE routing image

Results

Dense

Click to view the Dense Models Results

Post-MoE

Click to view the Post-MoE Models Results

Usage Format

Apollo2

• 0.5B, 1.5B, 7B: User:{query}\nAssistant:{response}<|endoftext|>
• 2B, 9B: User:{query}\nAssistant:{response}<eos>
• 3.8B: <|user|>\n{query}<|end|><|assisitant|>\n{response}<|end|>

Apollo-MoE

• 0.5B, 1.5B, 7B: User:{query}\nAssistant:{response}<|endoftext|>

Dataset & Evaluation

• Dataset 🤗 ApolloMoEDataset
  Click to expand

  

  • Data category
• Evaluation 🤗 ApolloMoEBench
  Click to expand

  • EN:
    • MedQA-USMLE
    • MedMCQA
    • PubMedQA: Because the results fluctuated too much, they were not used in the paper.
    • MMLU-Medical
      • Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
  • ZH:
    • MedQA-MCMLE
    • CMB-single: Not used in the paper
      • Randomly sample 2,000 multiple-choice questions with single answer.
    • CMMLU-Medical
      • Anatomy, Clinical_knowledge, College_medicine, Genetics, Nutrition, Traditional_chinese_medicine, Virology
    • CExam: Not used in the paper
      • Randomly sample 2,000 multiple-choice questions
  • ES: Head_qa
  • FR:
    • Frenchmedmcqa
    • [MMLU_FR]
      • Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
  • HI: MMLU_HI
    • Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
  • AR: MMLU_AR
    • Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
  • JA: IgakuQA
  • KO: KorMedMCQA
  • IT:
    • MedExpQA
    • [MMLU_IT]
      • Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
  • DE: BioInstructQA: German part
  • PT: BioInstructQA: Portuguese part
  • RU: RuMedBench

Model Download and Inference

1. Login Huggingface
   huggingface-cli login --token $HUGGINGFACE_TOKEN
2. Download model to local dir
   python

   1from huggingface_hub import snapshot_download 2import os 3 4local_model_dir=os.path.join('/path/to/models/dir','Apollo-MoE-0.5B') 5snapshot_download(repo_id="FreedomIntelligence/Apollo-MoE-0.5B", local_dir=local_model_dir)
3. Inference Example
   python

   1from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig 2import os 3 4local_model_dir=os.path.join('/path/to/models/dir','Apollo-MoE-0.5B') 5 6model=AutoModelForCausalLM.from_pretrained(local_model_dir,trust_remote_code=True) 7tokenizer = AutoTokenizer.from_pretrained(local_model_dir,trust_remote_code=True) 8generation_config = GenerationConfig.from_pretrained(local_model_dir, pad_token_id=tokenizer.pad_token_id, num_return_sequences=1, max_new_tokens=7, min_new_tokens=2, do_sample=False, temperature=1.0, top_k=50, top_p=1.0) 9 10inputs = tokenizer('Answer direclty.\nThe capital of Mongolia is Ulaanbaatar.\nThe capital of Iceland is Reykjavik.\nThe capital of Australia is', return_tensors='pt') 11inputs = inputs.to(model.device) 12pred = model.generate(**inputs,generation_config=generation_config) 13print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

Results reproduction

Citation

@misc{zheng2024efficientlydemocratizingmedicalllms,
      title={Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts}, 
      author={Guorui Zheng and Xidong Wang and Juhao Liang and Nuo Chen and Yuping Zheng and Benyou Wang},
      year={2024},
      eprint={2410.10626},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.10626}, 
}