---
license: apache-2.0
library_name: transformers
pipeline_tag: fill-mask
tags:
  - music
  - symbolic-music
  - chords
  - chord-embeddings
  - deberta-v2
  - masked-language-modeling
  - feature-extraction
---

# ChordBERT

ChordBERT is a compact DeBERTa-v2 masked language model for symbolic chord
sequences. It can predict masked chords or act as an encoder that maps a chord
progression to a 256-dimensional embedding.


Model developed by [Lameusiwe](https://huggingface.co/Lameusiwe).

## Model details

| Property | Value |
|---|---|
| Architecture | DeBERTa-v2 masked language model |
| Transformer layers | 4 |
| Attention heads | 4 |
| Hidden size | 256 |
| Intermediate size | 1,024 |
| Vocabulary size | 3,230 |
| Parameters | 4,713,886 |
| Recommended maximum length | 256 tokens including boundary tokens |
| Training objective | Masked language modeling |
| Training domain | Chordonomicon chord sequences |
| License | Apache-2.0 |

This repository contains the original Chordonomicon-pretrained ChordBERT
checkpoint. It is not the separately adapted `ChordBERT + WIR` checkpoint.

## Intended uses

ChordBERT is intended for research with harmonic sequences, including:

- chord and progression embeddings;
- harmonic similarity and retrieval;
- clustering and visualization of musical works;
- masked-chord prediction;
- feature extraction for downstream music-information-retrieval models.

The model was not designed for audio transcription, music generation,
copyright detection, composer attribution, or high-stakes cultural and
historical judgments.

## Input format

Input must be a whitespace-separated sequence of tokens from the supplied
tokenizer vocabulary:

```text
C G Amin F
Cmaj7 Amin7 Dmin7 G7
Bb F Gmin Eb
```

Chord spelling follows the Chordonomicon convention:

- major triads use only the root, such as `C`;
- sharps use `s`, such as `Csmin` for C-sharp minor;
- flats use `b`, such as `Bb`;
- common suffixes include `min`, `7`, `maj7`, `min7`, `dim`, `dim7`, and
  `aug`;
- some structural markers, such as `<verse_1>`, are present in the vocabulary.

Unknown or differently formatted symbols may become `<unk>`. Check token
coverage before embedding a new corpus:

```python
from transformers import AutoTokenizer

model_id = "YOUR_USERNAME/YOUR_MODEL_NAME"
tokenizer = AutoTokenizer.from_pretrained(model_id)

tokens = "C G Amin F".split()
unknown = [token for token in tokens if token not in tokenizer.get_vocab()]
print("Unknown tokens:", unknown)
```

## Masked-chord prediction

```python
from transformers import pipeline

model_id = "YOUR_USERNAME/YOUR_MODEL_NAME"
fill_mask = pipeline("fill-mask", model=model_id, tokenizer=model_id)

predictions = fill_mask("C G <mask> F", top_k=5)
for prediction in predictions:
    print(prediction["token_str"], prediction["score"])
```

## Generate a progression embedding

The checkpoint does not include a trained sentence-pooling head. A practical
default is mean pooling over non-special tokens:

```python
import torch
from transformers import AutoModel, AutoTokenizer

model_id = "YOUR_USERNAME/YOUR_MODEL_NAME"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()

progressions = [
    "C G Amin F",
    "Dmin7 G7 Cmaj7",
]

batch = tokenizer(
    progressions,
    padding=True,
    truncation=True,
    max_length=256,
    return_tensors="pt",
)

with torch.inference_mode():
    hidden = model(**batch).last_hidden_state

pool_mask = batch["attention_mask"].bool()
for special_id in tokenizer.all_special_ids:
    pool_mask &= batch["input_ids"] != special_id

weights = pool_mask.unsqueeze(-1).to(hidden.dtype)
embeddings = (hidden * weights).sum(dim=1) / weights.sum(dim=1).clamp(min=1)

print(embeddings.shape)  # torch.Size([2, 256])
```

The resulting vectors are not normalized. Apply L2 normalization if cosine
similarity is the downstream comparison:

```python
embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
```

## Long sequences

For progressions longer than 254 chord tokens:

1. split the sequence into chunks of at most 254 tokens;
2. embed each chunk with the pooling procedure above;
3. combine chunk embeddings using the number of chord tokens as weights.

This leaves room for the two boundary tokens added by the tokenizer and matches
the evaluation setup used for this checkpoint.



## Training information

The supplied checkpoint is recorded in the project artifacts as a
Chordonomicon-pretrained masked language model. The original detailed training
hyperparameters and a complete training-data statement are not included with
this checkpoint; they should not be inferred from the later evaluation and
adaptation scripts.

The model configuration identifies the architecture as
`DebertaV2ForMaskedLM`. Weights are stored in `safetensors` format.

## Reproducibility

Recommended dependencies:

```text
torch
transformers
safetensors
```

For deterministic comparisons, keep the same token conversion, maximum length,
special-token handling, pooling, chunk weighting, and vector normalization
across all corpora.

## Citation

```bibtex
@misc{he2026modelingstylisticcoevolutionsymbolic,
      title={Modeling Stylistic Co-evolution in Symbolic Music Heritage Collections}, 
      author={Yulong He and Ivan Smirnov and Yanming Li},
      year={2026},
      eprint={2607.23957},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2607.23957}, 
}
```