PubMedUL2 and MedUL2 are a family of domain-specific UL2/T5-style encoder–decoder language models pretrained on large-scale biomedical and medical corpora using the UL2 (Mixture-of-Denoisers) objective.

PubMedUL2 models are pretrained on 25 million PubMed abstracts
MedUL2 models are pretrained on PubMed abstracts + clinical notes + additional medical documents
All models use a T5-efficient architecture, inspired by Google’s efficient T5 variants

These checkpoints are pretraining-only models and must be fine-tuned before use on downstream tasks.

Pretraining Objective: UL2 (Mixture-of-Denoisers)

These models were pretrained using UL2, a unified framework that formulates language modeling objectives as denoising tasks.

UL2 introduces a Mixture-of-Denoisers (MoD) approach that samples from multiple denoising paradigms during pretraining.

Denoising Tasks

UL2 pretraining uses a mixture of three denoising tasks:

R-denoising (Regular Span Corruption)
- Equivalent to standard T5 span corruption
- Optimized for language understanding tasks
X-denoising (Extreme Span Corruption)
- Uses very large masked spans
- Encourages long-form generation and abstraction
S-denoising (Sequential / PrefixLM)
- Prefix language modeling similar to causal LM
- Suitable for sequence-to-sequence and generative tasks

Paradigm Tokens (Mode Switching)

During pretraining, a paradigm token is inserted at the beginning of each input:

Token	Mode	Recommended Use
`[NLU]`	R-denoising	Classification, QA, retrieval
`[NLG]`	X-denoising	Mixed understanding & generation
`[S2S]`	S-denoising	Generative / causal tasks

Important:
For best performance, the same token should be prepended during fine-tuning and inference.

Architecture

Encoder–decoder Transformer (T5-style)
Uses T5-efficient architecture
Compatible with Hugging Face T5ForConditionalGeneration

Intended Uses

These models are intended to be fine-tuned for:

Biomedical and clinical text classification
Question answering
Summarization of medical literature or clinical notes
Text generation in medical contexts

Limitations

❌ Not instruction-tuned
❌ No supervised training
❌ Not suitable for zero-shot use

These checkpoints are self-supervised pretraining models only and require task-specific fine-tuning.

Fine-Tuning Recommendations

Avoid mixed precision (fp16 / bf16) initially
- Fine-tuning is more stable in fp32
Always prepend one of [NLU], [NLG], or [S2S] to input text
Suggested defaults:
- Classification / QA → [NLU]
- Causal or generative tasks → [S2S]
- Mixed tasks → [NLG]

Model Parameter Summary

Model Name	Parameter Count	Description	Access
`pubmedul2-tiny-nl6`	19.26M	Tiny UL2-style model with 6 layers	Open
`pubmedul2-mini-nl8`	50.12M	Mini UL2 with 8 layers	Open
`pubmedul2-small`	60.52M	Small UL2 variant	Open
`pubmedul2-small-nl24`	192.73M	Small UL2 with 24 layers	Open
`medul2-base`	222.93M	Base UL2/T5-style model	Open
`pubmedul2-base`	222.93M	Base UL2/T5-style model	Open
`medul2-base-nl36`	619.44M	Base UL2 with 36 layers	Gated commercial
`pubmedul2-base-nl36`	619.44M	Base UL2 with 36 layers	Gated commercial
`medul2-large`	737.72M	Large UL2/T5-style model	Gated non-commercial
`pubmedul2-large`	737.72M	Large UL2/T5-style model	Gated non-commercial
`medul2-large-nl36`	1090.14M	Very large UL2 with 36 layers	Access on Request

Named Entity Recognition (NER) Evaluation

We evaluate PubMedUL2 and MedUL2 models on a biomedical Named Entity Recognition (NER) task using multiple matching criteria to better capture boundary-level performance.

The evaluation reports entity-level F1 scores across different biomedical entity types and model sizes.

Exact Match F1

An entity prediction is considered correct only if both the entity span and label exactly match the gold annotation.

entity_type	medul2-base	pubmedul2-base	pubmedul2-mini-nl8	pubmedul2-small	pubmedul2-tiny-nl6
cell_line	0.42	0.43	0.44	0.43	0.35
cell_type	0.59	0.58	0.59	0.58	0.52
chemical	0.76	0.75	0.72	0.72	0.56
disease	0.7	0.73	0.7	0.68	0.63
dna	0.59	0.55	0.54	0.55	0.45
gene	0.62	0.59	0.6	0.59	0.55
protein	0.59	0.58	0.58	0.59	0.55
rna	0.6	0.56	0.55	0.6	0.56
species	0.66	0.67	0.58	0.63	0.54

Partial Match F1

A prediction is counted as correct if it partially overlaps with a gold entity of the same type.

entity_type	medul2-base	pubmedul2-base	pubmedul2-mini-nl8	pubmedul2-small	pubmedul2-tiny-nl6
cell_line	0.48	0.49	0.48	0.48	0.41
cell_type	0.66	0.64	0.66	0.65	0.59
chemical	0.79	0.78	0.76	0.75	0.6
disease	0.82	0.84	0.8	0.79	0.74
dna	0.65	0.61	0.6	0.61	0.53
gene	0.76	0.74	0.74	0.73	0.68
protein	0.66	0.66	0.66	0.67	0.64
rna	0.68	0.63	0.64	0.66	0.65
species	0.68	0.7	0.61	0.65	0.56

IoU Match F1

Predictions are evaluated using Intersection-over-Union (IoU) overlap between predicted and gold spans, providing a softer boundary-based metric.

entity_type	medul2-base	pubmedul2-base	pubmedul2-mini-nl8	pubmedul2-small	pubmedul2-tiny-nl6
cell_line	0.5	0.5	0.5	0.5	0.42
cell_type	0.67	0.66	0.68	0.67	0.62
chemical	0.83	0.83	0.82	0.82	0.72
disease	0.85	0.86	0.86	0.85	0.82
dna	0.65	0.62	0.62	0.62	0.55
gene	0.76	0.75	0.75	0.74	0.71
protein	0.67	0.66	0.67	0.67	0.66
rna	0.68	0.65	0.66	0.67	0.67
species	0.72	0.74	0.65	0.69	0.58

Observations

MedUL2 models generally outperform PubMedUL2 on clinical-heavy entity types such as disease and chemical
Performance improves consistently from tiny → base models
Boundary-sensitive metrics (Partial / IoU) show significantly higher scores than Exact Match, highlighting boundary ambiguity in biomedical NER

Acknowledgements

This project would not have been possible without compute generously provided by Google TPU Research Cloud.

Thanks to:

The Finnish-NLP authors for releasing the UL2 objective code, task definitions, and guidance
Yeb Havinga for help getting started with the t5x framework

License

Please refer to the individual model repositories for license and access details, which may vary depending on training data sources.

Siddharth63/medul2-base

PubMedUL2 & MedUL2

Model Description