Instructions to use rokotmidi/rokot-omr-2b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use rokotmidi/rokot-omr-2b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="rokotmidi/rokot-omr-2b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("rokotmidi/rokot-omr-2b")
model = AutoModelForMultimodalLM.from_pretrained("rokotmidi/rokot-omr-2b", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use rokotmidi/rokot-omr-2b with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf rokotmidi/rokot-omr-2b:F16
# Run inference directly in the terminal:
llama cli -hf rokotmidi/rokot-omr-2b:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf rokotmidi/rokot-omr-2b:F16
# Run inference directly in the terminal:
llama cli -hf rokotmidi/rokot-omr-2b:F16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf rokotmidi/rokot-omr-2b:F16
# Run inference directly in the terminal:
./llama-cli -hf rokotmidi/rokot-omr-2b:F16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf rokotmidi/rokot-omr-2b:F16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf rokotmidi/rokot-omr-2b:F16

Use Docker

docker model run hf.co/rokotmidi/rokot-omr-2b:F16

LM Studio
Jan

vLLM

How to use rokotmidi/rokot-omr-2b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "rokotmidi/rokot-omr-2b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rokotmidi/rokot-omr-2b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/rokotmidi/rokot-omr-2b:F16

SGLang

How to use rokotmidi/rokot-omr-2b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "rokotmidi/rokot-omr-2b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rokotmidi/rokot-omr-2b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "rokotmidi/rokot-omr-2b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rokotmidi/rokot-omr-2b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Ollama
How to use rokotmidi/rokot-omr-2b with Ollama:
```
ollama run hf.co/rokotmidi/rokot-omr-2b:F16
```

Unsloth Studio

How to use rokotmidi/rokot-omr-2b with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for rokotmidi/rokot-omr-2b to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for rokotmidi/rokot-omr-2b to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for rokotmidi/rokot-omr-2b to start chatting

How to use rokotmidi/rokot-omr-2b with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rokotmidi/rokot-omr-2b:F16

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "rokotmidi/rokot-omr-2b:F16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use rokotmidi/rokot-omr-2b with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rokotmidi/rokot-omr-2b:F16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default rokotmidi/rokot-omr-2b:F16

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use rokotmidi/rokot-omr-2b with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rokotmidi/rokot-omr-2b:F16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "rokotmidi/rokot-omr-2b:F16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use rokotmidi/rokot-omr-2b with Docker Model Runner:
```
docker model run hf.co/rokotmidi/rokot-omr-2b:F16
```

Lemonade

How to use rokotmidi/rokot-omr-2b with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull rokotmidi/rokot-omr-2b:F16

Run and chat with the model

lemonade run user.rokot-omr-2b-F16

List all available models

lemonade list

rokot-omr-2b — sheet music in, MusicXML out

✅ v2 release (17 Jul 2026): the completed second training run — 47,532 steps over 1.5M samples spanning engraved scores, lead sheets with chords and lyrics, and pianoform music. Reads piano / grand-staff scores, chord symbols (incl. European conventions like Em written "em" and German H), and multi-verse lyrics. Ranked #1 on the IMSLP Piano Scores leaderboard at 0.4409 OMR-NED (20 Jul 2026), with less than half the error of GPT-5 and Gemini 2.5 Pro on the same pages. Handwritten-manuscript support is the next training run.

rokot-omr-2b turns images of printed sheet music — scans, photos, pages out of a PDF — into machine-readable notation. Show it a system of music and it writes rokot-ABC, a strict, canonical profile of ABC notation that converts losslessly to MusicXML. From there your score opens directly in MuseScore, Finale, Dorico, or anything else that speaks MusicXML — ready to edit, transpose, and play back.

Because the output is plain ABC text, the model is small enough and simple enough to run locally on your own machine — including in LM Studio on a Mac: load the model, drop in a cropped image of a staff, and copy the ABC out. Standard ABC tooling (abc2xml, EasyABC, abcjs, …) reads rokot-ABC as-is, since it is exactly one deterministic spelling of ordinary ABC.

Note the unit of work: the model reads one system at a time (one line of music, cropped to the staff). Turning a whole page or PDF into a single score additionally needs page segmentation and measure-wise joining — that's a pipeline around the model, not the model itself. LM Studio is great for trying it out and for transcribing lines you crop yourself.

It is a full fine-tune of Qwen/Qwen3-VL-2B-Instruct (2.1 B parameters, Apache-2.0 base), trained on a large synthetic corpus of engraved scores rendered with realistic page degradation — so it is built for the sheet music people actually have: photocopies, songbook scans, lead sheets with chords and lyrics, not just clean digital engravings.

What it can read

Single staff (treble or bass) and grand-staff systems, up to 2 voices per staff
Pitches, accidentals, rests, chords, tuplets, dotted values from breve to 1/64
Key/meter/clef changes mid-score, repeats, voltas, ties, slurs, grace notes
Ornaments, articulations, dynamics and hairpins
Chord symbols (root/quality/bass), tempo marks
Lyrics with multiple verses, melisma and syllable continuation

Out of scope (for now): more than one part, percussion/TAB, nested tuplets, and handwritten manuscripts — that last one is actively being worked on.

How it scores

On the public IMSLP Piano Scores leaderboard — 32 real scanned piano pages from IMSLP, scored by OMR-NED (a normalized edit distance over the full MusicXML; lower is better):

System	OMR-NED ↓
rokot-omr-2b (this model) — #1	0.4409
homr	0.9050
Gemini 2.5 Pro	0.9403
GPT-5	0.9438

Standings as of 20 Jul 2026. The score is produced by our full-page pipeline: staff-system detection, per-system transcription with this model, and measure-wise joining into one MusicXML per page — the same per-system transcription you get from the quick-start below, run over every system of a page. A 2.1 B model you can run on a laptop, leading the board.

Quick start in LM Studio (Mac & PC)

GGUF builds are included in this repo for local use with LM Studio or llama.cpp:

File	What it is	Size
`rokot-omr-2b-Q8_0.gguf`	Recommended — 8-bit, verified output matches the full model	~1.8 GB
`rokot-omr-2b-f16.gguf`	Full-precision text model	~3.4 GB
`mmproj-rokot-omr-2b-f16.gguf`	Vision projector — required alongside either model	~0.8 GB

Search for rokotmidi/rokot-omr-2b in LM Studio's Discover tab (it fetches the vision projector automatically), drop in an image of one system (a single line of music cropped to the staff — not a full page), and ask: "Transcribe this sheet music to rokot-ABC."

A note on quantization: we deliberately do not publish 4-bit builds. This model writes a strict notation language where a single wrong token corrupts the score, and in our tests Q4 visibly degraded it. Q8_0 matched the full-precision model exactly on our eval samples — use that.

Usage with transformers

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "rokotmidi/rokot-omr-2b"
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": "staff.png"},
        {"type": "text", "text": "Transcribe this staff to rokot-ABC."},
    ],
}]

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

out = model.generate(**inputs, max_new_tokens=1600, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Input images should be one system (one line of music) cropped to the staff, ideally around 1400 px wide. The output is a complete, self-contained ABC tune with fixed header order (X, M, L, K, …) — one deterministic spelling for everything, so downstream tooling can rely on it.

Support the project ❤️

This model is free for personal and research use, and it exists because of RokoTMIDI — a compact MIDI controller with the button layout of a chromatic button accordion. If this model saves you hours of transcription work and you'd like to give something back, the nicest way to do it is to grab a controller (or tell an accordionist friend about it):

👉 rokotmidi.com

Every controller sold funds more GPU hours, better datasets, and future versions of this model — including the handwritten-manuscript support that is currently in training. Thank you! 🪗

License

Released under CC BY-NC 4.0 — free to use, share and build on for non-commercial purposes with attribution. For commercial licensing, get in touch via rokotmidi.com.

Limitations

One part / max two staves per system; complex piano textures with more than two voices per staff are simplified.
Handwritten manuscripts are not yet supported (next training run).
Lyrics in non-Latin scripts remain less reliable than notation.
Very low-resolution or heavily skewed crops degrade accuracy — deskew and crop per system for best results.

Downloads last month: 661

Safetensors

Model size

2B params

Tensor type

BF16

Model tree for rokotmidi/rokot-omr-2b

Base model

Qwen/Qwen3-VL-2B-Instruct

Quantized

(84)

this model

Quantizations

1 model