You can deploy this model and start earning money today!

TsinghuaC3I/
Llama-3.1-8B-UltraMedical

Name: Llama-3.1-8B-UltraMedical
Availability: InStock

safetensors

llama

TsinghuaC3I/UltraMedical

TsinghuaC3I/UltraMedical-Preference

2406.03949

meta-llama/Llama-3.1-8B-Instruct

finetune

llama3

Views

No views yet

UltraMedical: Building Specialized Generalists in Biomedicine.

SFT Dataset • Pref Dataset • Collection • GitHub Code • Paper

Llama-3.1-8B-UltraMedical is an open-access large language model (LLM) specialized in biomedicine. Developed by the Tsinghua C3I Lab, this model aims to enhance medical examination access, literature comprehension, and clinical knowledge.

Building on the foundation of Meta's Llama-3.1-8B, Llama-3.1-8B-UltraMedical is trained on our UltraMedical collection with supervised fine-tuning (SFT), iterative preference learning (like DPO and KTO). The UltraMedical collection is a large-scale, high-quality dataset of biomedical instructions, comprising 410,000 synthetic and manually curated samples, along with more than 100,000 preference data.

Evaluations

Model	MultiMedQA	MMLU	GPQA	Alpaca Eval 2 (LC/WR)
Llama-3.1-8B-Instruct	71.38	73.00	30.40	20.90/21.80
Llama-3.1-8B-UltraMedical	76.82	72.96	34.82	41.22/48.20

Inference with vLLM

python

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

llm = LLM(model="TsinghuaC3I/Llama-3.1-8B-UltraMedical", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("TsinghuaC3I/Llama-3.1-8B-UltraMedical")
sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=1024, stop=["<|eot_id|>"])

messages = [
    {"role": "user", "content": """The question format used in the above input examples。"""},
]
prompts = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(prompts[0])
"""
<|begin_of_text|><|start_header_id|>user<|end_header_id|>

{question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

"""

outputs = llm.generate(prompts=prompts, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

Citation

@misc{zhang2024ultramedical,
      title={UltraMedical: Building Specialized Generalists in Biomedicine}, 
      author={Kaiyan Zhang and Sihang Zeng and Ermo Hua and Ning Ding and Zhang-Ren Chen and Zhiyuan Ma and Haoxin Li and Ganqu Cui and Biqing Qi and Xuekai Zhu and Xingtai Lv and Hu Jinfang and Zhiyuan Liu and Bowen Zhou},
      year={2024},
      eprint={2406.03949},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}