README.md

---
library_name: transformers
tags: []
---


<div align="center">


<h1 align="center">

MuSViT: A Foundation Vision Model for Sheet Music Representation

</h1>

<p align="center">

Accepted at European Conference on Computer Vision (ECCV'26)

</p>

[![repo](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/OMR-PRAIG-UA-ES/MuSViT)
[![web](https://img.shields.io/badge/Project-Page-blue)](https://grfia.dlsi.ua.es/musvit/)
[![paper](https://img.shields.io/badge/arXiv-MuSViT-red?logo=arxiv)](https://arxiv.org/abs/2606.31811)
</div>

# MuSViT

<!-- Provide a quick summary of what the model is/does. -->
MuSViT (**Mu**sic **S**core **Vi**sion **T**ransformer) is a foundation vision encoder for music score pages. The model is a ViT pre-trained following Masked Autoencoders (MAE) on 9.7M sheet music images from the IMSLP. The embeddings produced by MuSViT are task agnostic, so they can be used for any downstream task.

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

- **Developed by:** Pattern Recognition and Artificial Intelligence Group (PRAIG), University of Alicante, Spain
- **Model type:** MAE
- **Paper:** MuSViT: A Foundation Vision Model for Sheet Music Representation
- **General License:** CC BY-NC-SA 4.0

## How to use

### Installation

MuSViT does not requiere to clone any repository! Only to have installed transformers library.

### Usage on music score pages

```python
import torch
from transformers import ViTModel
from PIL import Image
from torchvision import transforms as T


image_path = 'path/image.png'
image = Image.open(image_path).convert("RGB")
processor = T.Compose([
    T.Resize([1024, 1024]),
    T.ToTensor()
])
images = processor(image).unsqueeze(0) # shape: B, C, H, W

model = ViTModel.from_pretrained('PRAIG/musvit', trust_remote_code=True)

out = model(images).last_hidden_state

print(out.shape) #shape: B, 4097, 768. Note it has CLS token
```

### Usage in systems (non-pages)

For system-level images whose reshape to 1024x1024 px would distort too much its aspect, there are two options:

- **Padding**

Pad the image to fit input size. Recommended for zero-shot configuration

```python
import torch
from transformers import ViTModel
from PIL import Image
from torchvision import transforms as T


image_path = 'path/staff_image.png'
image = Image.open(image_path).convert("RGB")
image.resize((1024, 64)) # (W, H)

background = Image.new("RGB", (1024, 1024), color=(255, 255, 255))
background.paste(image, (0, 0))
image = background # You might check image aspect with image.save('img.png')

processor = T.Compose([ # It already has 1024x1024 shape
    T.ToTensor()
])
images = processor(image).unsqueeze(0) # shape: B, C, H, W

model = ViTModel.from_pretrained('PRAIG/musvit', trust_remote_code=True)

out = model(images).last_hidden_state
out = out[:, 1:, :] # Skip CLS token
out = out.reshape(out.shape[0], 64, 64, -1) # shape: B, Rows, Columns, Dim
out = out[:, :4, :, :] # take 64/16=4 first rows
out = out.flatten(1, 2)

print(out.shape) #shape: B, 256, 768
```

- **Interpolate positional encoding**

If you don't want to pad, you can interpolate positional encoding of the model. In zero-shot, this configuration downgrades embeddings quality. However, for fine-tuning MuSViT this configuration reports good performance.

```python
import torch
from transformers import ViTModel
from PIL import Image
from torchvision import transforms as T


image_path = 'path/staff_image.png'
image = Image.open(image_path).convert("RGB")
processor = T.Compose([
    T.ToTensor()
])
images = processor(image).unsqueeze(0) # shape: B, C, H, W

model = ViTModel.from_pretrained('PRAIG/musvit', trust_remote_code=True)

out = model(images, interpolate_pos_encoding=True).last_hidden_state

print(out.shape) #shape: B, Len, Dim. Note it has CLS token
```

### Usage of pre-trained MAE model

```python
import torch
from transformers import ViTMAEForPreTraining
from PIL import Image
from torchvision import transforms as T

image_path = 'path/image.png'
image = Image.open(image_path).convert("RGB")
processor = T.Compose([
    T.Resize([1024, 1024]),
    T.ToTensor()
])
images = processor(image).unsqueeze(0) # shape: B, C, H, W

model = ViTMAEForPreTraining.from_pretrained('PRAIG/musvit', trust_remote_code=True)

out = model(images)

print(out.loss) # Reconstruction loss
print(out.logits.shape) #shape: B, 4096, 768. This 768 comes from 3*16*16 px to reconstruct per patch
```

### ⚠️ Warning ⚠️:

Loading with AutoModel loads the model with ViTMAEModel. This model returns the patches with the 70% masked out and shuffled. If you want all the patches, set masking to 0. Moreover, for avoiding shuffled patches set 'noise' parameter in forward with contiguous positions.

```python
import torch
from transformers import AutoModel
from PIL import Image
from torchvision import transforms as T


image_path = 'path/image.png'
image = Image.open(image_path).convert("RGB")
processor = T.Compose([
    T.Resize([1024, 1024]),
    T.ToTensor()
])
images = processor(image).unsqueeze(0) # shape: B, C, H, W

model = AutoModel.from_pretrained('PRAIG/musvit', trust_remote_code=True)
model.config.mask_ratio = 0.

noise = torch.arange(4096).expand(images.shape[0], 4096)
out = model(images, noise=noise).last_hidden_state

print(out.shape) #shape: B, 4097, 768. Note it has CLS token
```

To avoid all these inconveniences, we recommend loading the model with ViTModel. See code of sections above.


## Citation

```bibtex
@inproceedings{penarrubia2026musvit,
  title     = {MuSViT: A Foundation Vision Model for Sheet Music Representation},
  author    = {Penarrubia, Carlos and Rios-Vila, Antonio and Fuentes-Martinez, Eliseo and Martinez-Sevilla, Juan C. and Castellanos, Francisco J. and Alfaro-Contreras, Maria and Calvo-Zaragoza, Jorge},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}
```