README.md

---
language:
- en
license: apache-2.0
tags:
- biomedical
- clinical
- ul2
- t5
- encoder-decoder
- pretraining
- text2text-generation
- medical
---

# PubMedUL2 & MedUL2

## Model Description

**PubMedUL2** and **MedUL2** are a family of **domain-specific UL2/T5-style encoder–decoder language models** pretrained on large-scale biomedical and medical corpora using the **UL2 (Mixture-of-Denoisers)** objective.

- **PubMedUL2** models are pretrained on **25 million PubMed abstracts**
- **MedUL2** models are pretrained on **PubMed abstracts + clinical notes + additional medical documents**
- All models use a **T5-efficient architecture**, inspired by Google’s efficient T5 variants

These checkpoints are **pretraining-only models** and **must be fine-tuned** before use on downstream tasks.

---

## Pretraining Objective: UL2 (Mixture-of-Denoisers)

These models were pretrained using **UL2**, a unified framework that formulates language modeling objectives as **denoising tasks**.

UL2 introduces a **Mixture-of-Denoisers (MoD)** approach that samples from multiple denoising paradigms during pretraining.

### Denoising Tasks

UL2 pretraining uses a mixture of three denoising tasks:

1. **R-denoising (Regular Span Corruption)**  
   - Equivalent to standard T5 span corruption  
   - Optimized for language understanding tasks

2. **X-denoising (Extreme Span Corruption)**  
   - Uses very large masked spans  
   - Encourages long-form generation and abstraction

3. **S-denoising (Sequential / PrefixLM)**  
   - Prefix language modeling similar to causal LM  
   - Suitable for sequence-to-sequence and generative tasks

### Paradigm Tokens (Mode Switching)

During pretraining, a **paradigm token** is inserted at the beginning of each input:

| Token  | Mode | Recommended Use |
|------|------|------------------|
| `[NLU]` | R-denoising | Classification, QA, retrieval |
| `[NLG]` | X-denoising | Mixed understanding & generation |
| `[S2S]` | S-denoising | Generative / causal tasks |

**Important:**  
For best performance, the same token should be **prepended during fine-tuning and inference**.

---

## Architecture

- Encoder–decoder Transformer (T5-style)
- Uses **T5-efficient architecture**
- Compatible with Hugging Face `T5ForConditionalGeneration`

---

## Intended Uses

These models are intended to be **fine-tuned** for:

- Biomedical and clinical **text classification**
- **Question answering**
- **Summarization** of medical literature or clinical notes
- **Text generation** in medical contexts

---

## Limitations

- ❌ Not instruction-tuned  
- ❌ No supervised training  
- ❌ Not suitable for zero-shot use  

These checkpoints are **self-supervised pretraining models only** and require task-specific fine-tuning.

---

## Fine-Tuning Recommendations

- **Avoid mixed precision** (fp16 / bf16) initially  
  - Fine-tuning is more stable in **fp32**
- Always prepend one of `[NLU]`, `[NLG]`, or `[S2S]` to input text
- Suggested defaults:
  - Classification / QA → `[NLU]`
  - Causal or generative tasks → `[S2S]`
  - Mixed tasks → `[NLG]`

---

## Model Parameter Summary

| Model Name | Parameter Count | Description | Access
|-----------|----------------|------------|------------|
| `pubmedul2-tiny-nl6` | **19.26M** | Tiny UL2-style model with 6 layers | Open
| `pubmedul2-mini-nl8` | **50.12M** | Mini UL2 with 8 layers | Open
| `pubmedul2-small` | **60.52M** | Small UL2 variant | Open
| `pubmedul2-small-nl24` | **192.73M** | Small UL2 with 24 layers | Open
| `medul2-base` | **222.93M** | Base UL2/T5-style model | Open
| `pubmedul2-base` | **222.93M** | Base UL2/T5-style model | Open
| `medul2-base-nl36` | **619.44M** | Base UL2 with 36 layers | Gated commercial
| `pubmedul2-base-nl36` | **619.44M** | Base UL2 with 36 layers | Gated commercial
| `medul2-large` | **737.72M** | Large UL2/T5-style model | Gated non-commercial
| `pubmedul2-large` | **737.72M** | Large UL2/T5-style model | Gated non-commercial
| `medul2-large-nl36` | **1090.14M** | Very large UL2 with 36 layers | Access on Request

---

## Named Entity Recognition (NER) Evaluation

We evaluate PubMedUL2 and MedUL2 models on a biomedical **Named Entity Recognition (NER)** task using multiple matching criteria to better capture boundary-level performance.

The evaluation reports **entity-level F1 scores** across different biomedical entity types and model sizes.

### Exact Match F1

An entity prediction is considered correct only if both the **entity span and label exactly match** the gold annotation.

| entity_type   |   medul2-base |   pubmedul2-base |   pubmedul2-mini-nl8 |   pubmedul2-small |   pubmedul2-tiny-nl6 |
|:--------------|--------------:|-----------------:|---------------------:|------------------:|---------------------:|
| cell_line     |          0.42 |             0.43 |                 0.44 |              0.43 |                 0.35 |
| cell_type     |          0.59 |             0.58 |                 0.59 |              0.58 |                 0.52 |
| chemical      |          0.76 |             0.75 |                 0.72 |              0.72 |                 0.56 |
| disease       |          0.7  |             0.73 |                 0.7  |              0.68 |                 0.63 |
| dna           |          0.59 |             0.55 |                 0.54 |              0.55 |                 0.45 |
| gene          |          0.62 |             0.59 |                 0.6  |              0.59 |                 0.55 |
| protein       |          0.59 |             0.58 |                 0.58 |              0.59 |                 0.55 |
| rna           |          0.6  |             0.56 |                 0.55 |              0.6  |                 0.56 |
| species       |          0.66 |             0.67 |                 0.58 |              0.63 |                 0.54 |

---

### Partial Match F1

A prediction is counted as correct if it **partially overlaps** with a gold entity of the same type.

| entity_type   |   medul2-base |   pubmedul2-base |   pubmedul2-mini-nl8 |   pubmedul2-small |   pubmedul2-tiny-nl6 |
|:--------------|--------------:|-----------------:|---------------------:|------------------:|---------------------:|
| cell_line     |          0.48 |             0.49 |                 0.48 |              0.48 |                 0.41 |
| cell_type     |          0.66 |             0.64 |                 0.66 |              0.65 |                 0.59 |
| chemical      |          0.79 |             0.78 |                 0.76 |              0.75 |                 0.6  |
| disease       |          0.82 |             0.84 |                 0.8  |              0.79 |                 0.74 |
| dna           |          0.65 |             0.61 |                 0.6  |              0.61 |                 0.53 |
| gene          |          0.76 |             0.74 |                 0.74 |              0.73 |                 0.68 |
| protein       |          0.66 |             0.66 |                 0.66 |              0.67 |                 0.64 |
| rna           |          0.68 |             0.63 |                 0.64 |              0.66 |                 0.65 |
| species       |          0.68 |             0.7  |                 0.61 |              0.65 |                 0.56 |

---

### IoU Match F1

Predictions are evaluated using **Intersection-over-Union (IoU)** overlap between predicted and gold spans, providing a softer boundary-based metric.

| entity_type   |   medul2-base |   pubmedul2-base |   pubmedul2-mini-nl8 |   pubmedul2-small |   pubmedul2-tiny-nl6 |
|:--------------|--------------:|-----------------:|---------------------:|------------------:|---------------------:|
| cell_line     |          0.5  |             0.5  |                 0.5  |              0.5  |                 0.42 |
| cell_type     |          0.67 |             0.66 |                 0.68 |              0.67 |                 0.62 |
| chemical      |          0.83 |             0.83 |                 0.82 |              0.82 |                 0.72 |
| disease       |          0.85 |             0.86 |                 0.86 |              0.85 |                 0.82 |
| dna           |          0.65 |             0.62 |                 0.62 |              0.62 |                 0.55 |
| gene          |          0.76 |             0.75 |                 0.75 |              0.74 |                 0.71 |
| protein       |          0.67 |             0.66 |                 0.67 |              0.67 |                 0.66 |
| rna           |          0.68 |             0.65 |                 0.66 |              0.67 |                 0.67 |
| species       |          0.72 |             0.74 |                 0.65 |              0.69 |                 0.58 |

---

### Observations

- **MedUL2 models** generally outperform PubMedUL2 on clinical-heavy entity types such as *disease* and *chemical*
- Performance improves consistently from **tiny → base models**
- Boundary-sensitive metrics (Partial / IoU) show significantly higher scores than Exact Match, highlighting boundary ambiguity in biomedical NER

--- 

## Acknowledgements

This project would not have been possible without compute generously provided by **Google TPU Research Cloud**.

Thanks to:
- The **Finnish-NLP** authors for releasing the UL2 objective code, task definitions, and guidance
- **Yeb Havinga** for help getting started with the **t5x** framework

---

## License

Please refer to the individual model repositories for **license and access details**, which may vary depending on training data sources.