Upload folder using huggingface_hub

README.md

@@ -1,3 +1,122 @@
----
-license: apache-2.0
----
+---
+language:
+  - en
+license: apache-2.0
+library_name: transformers
+pipeline_tag: fill-mask
+tags:
+  - music
+  - lilypond
+  - mlm
+  - music-information-retrieval
+base_model: microsoft/codebert-base
+datasets:
+  - custom
+model-index:
+  - name: lilyBERT
+    results:
+      - task:
+          type: text-classification
+          name: Composer Classification (Linear Probe)
+        dataset:
+          type: custom
+          name: Mutopia (out-of-domain)
+        metrics:
+          - type: accuracy
+            value: 84.3
+            name: Composer Accuracy
+          - type: accuracy
+            value: 82.9
+            name: Style Accuracy
+---
+
+# lilyBERT
+
+**lilyBERT** is a masked language model for [LilyPond](https://lilypond.org/) music notation, built by adapting [CodeBERT](https://huggingface.co/microsoft/codebert-base) to the musical domain.
+
+LilyPond is a text-based music engraving language with formal grammar, block structure, and backslash commands — making it structurally similar to a programming language. lilyBERT leverages this by extending CodeBERT's vocabulary with 115 domain-specific tokens (e.g. `\trill`, `\fermata`, `\mordent`, `\staccato`) and performing MLM pre-training on curated Baroque music scores.
+
+## Training
+
+This checkpoint was trained in two stages:
+
+1. **Stage 1 — PDMX pre-training**: CodeBERT fine-tuned on the [PDMX](https://github.com/fosfrancesco/pdmx) corpus of automatically converted LilyPond files.
+2. **Stage 2 — BMdataset fine-tuning**: Further fine-tuned on the [BMdataset](https://doi.org/10.5281/zenodo.18723290), a musicologically curated collection of ~470 Baroque scores in LilyPond format (~90M tokens).
+
+| Hyperparameter | Value |
+|---|---|
+| Architecture | RobertaForMaskedLM (12 layers, 768 hidden, 12 heads) |
+| Vocab size | 50,380 (50,265 base + 115 music tokens) |
+| Max sequence length | 512 |
+| MLM probability | 0.15 |
+| Batch size | 72 × 2 GPUs × 2 grad. accum. = 288 |
+| Learning rate | 2e-4 (cosine schedule) |
+| Warmup | 10% |
+| Epochs | 10 (early stopping, patience 5) |
+| Precision | bf16 |
+| Optimizer | AdamW (fused) |
+
+## Results
+
+Linear probing on the out-of-domain Mutopia corpus (layer 6, 5-fold CV):
+
+| Model | Composer Acc. | Style Acc. |
+|-------|:---:|:---:|
+| CB + PDMX_full (15B tokens) | 80.8 | 82.6 |
+| CB + BMdataset (90M tokens) | 82.9 | **83.7** |
+| CB + PDMX_90M (90M tokens) | 81.7 | 82.3 |
+| **CB + PDMX → BM (this model)** | **84.3** | 82.9 |
+
+90M tokens of expertly curated data outperform 15B tokens of automatically converted data. Combining broad pre-training with domain-specific fine-tuning yields the best overall composer accuracy.
+
+## Usage
+
+```python
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("csc-unipd/lilybert")
+model = AutoModelForMaskedLM.from_pretrained("csc-unipd/lilybert")
+```
+
+### Fill-mask example
+
+```python
+from transformers import pipeline
+
+filler = pipeline("fill-mask", model="csc-unipd/lilybert")
+filler("\\relative c' { c4 d <mask> f | g2 g }")
+```
+
+### Feature extraction
+
+```python
+import torch
+
+inputs = tokenizer("\\relative c' { c4 d e f | g2 g }", return_tensors="pt")
+with torch.no_grad():
+    outputs = model(**inputs, output_hidden_states=True)
+
+# Layer 6 embeddings (best for linear probing)
+embeddings = outputs.hidden_states[6]
+```
+
+## Citation
+
+```bibtex
+@misc{spanio2026bmdataset,
+  title     = {BMdataset: A Musicologically Curated LilyPond Dataset},
+  author    = {Spanio, Matteo and Guler, Ilay and Roda, Antonio},
+  year      = {2026},
+  publisher = {Under review},
+}
+```
+
+## Links
+
+- **Paper**: SMC 2026 (to appear)
+- **Dataset**: [Zenodo (doi:10.5281/zenodo.18723290)](https://doi.org/10.5281/zenodo.18723290)
+- **Code**: [GitHub (CSCPadova/lilybert)](https://github.com/CSCPadova/lilybert)
+
+## License
+
+Apache-2.0

config.json

@@ -0,0 +1,29 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "RobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": false,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 1,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.2.0",
+  "type_vocab_size": 1,
+  "use_cache": false,
+  "vocab_size": 50380
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7740960ce5b8a6feadc9a567464544ab29471b3db4b4055c0849e8e2bac147db
+size 499167664

tokenizer_config.json

@@ -0,0 +1,16 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "errors": "replace",
+  "is_local": true,
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "TokenizersBackend",
+  "trim_offsets": true,
+  "unk_token": "<unk>"
+}

training_config.json

@@ -0,0 +1,48 @@
+{
+  "data_dir": "/nfsd/voce/machine_learning/experiments/artifacts/processed",
+  "tokenizer_path": "/nfsd/voce/machine_learning/experiments/artifacts/tokenizer",
+  "output_dir": "/nfsd/voce/machine_learning/experiments/cb-pdmx-bm",
+  "model_architecture": "microsoft/codebert-base",
+  "random_init": false,
+  "hidden_size": 768,
+  "num_hidden_layers": 12,
+  "num_attention_heads": 12,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 514,
+  "max_length": 512,
+  "mlm_probability": 0.15,
+  "per_device_train_batch_size": 72,
+  "per_device_eval_batch_size": 72,
+  "num_train_epochs": 10,
+  "learning_rate": 0.0002,
+  "lr_scheduler_type": "cosine",
+  "max_grad_norm": 1.0,
+  "weight_decay": 0.01,
+  "warmup_ratio": 0.1,
+  "max_steps": -1,
+  "logging_steps": 50,
+  "eval_steps": 1000,
+  "save_steps": 1000,
+  "seed": 42,
+  "pretokenized_shards_dir": "/nfsd/voce/machine_learning/experiments/codebert/baroque-music-shards/mlm",
+  "resume_from_checkpoint": null,
+  "dataloader_num_workers": 8,
+  "early_stopping": true,
+  "early_stopping_patience": 5,
+  "early_stopping_threshold": 0.0,
+  "bf16": true,
+  "gradient_accumulation_steps": 2,
+  "optim": "adamw_torch_fused",
+  "dataloader_pin_memory": true,
+  "dataloader_prefetch_factor": 4,
+  "save_total_limit": 3,
+  "torch_compile": false,
+  "ddp_find_unused_parameters": false,
+  "wandb_enabled": true,
+  "wandb_project": "lilybert",
+  "wandb_entity": null,
+  "wandb_mode": "online",
+  "wandb_run_name": "cb-pdmx-baroquemusic",
+  "tensorboard_enabled": true,
+  "tensorboard_log_dir": "/nfsd/voce/machine_learning/experiments/cb-pdmx-bm/tensorboard"
+}