Rokot Omr 2b

rokot-omr-2b — sheet music in, MusicXML out

RokoTMIDI controller

✅ v2 release (17 Jul 2026): the completed second training run — 47,532 steps over 1.5M samples spanning engraved scores, lead sheets with chords and lyrics, and pianoform music. Reads piano / grand-staff scores, chord symbols (incl. European conventions like Em written "em" and German H), and multi-verse lyrics. Ranked 2nd on the IMSLP Piano Scores leaderboard among all systems including GPT-5 and Gemini, behind only a specialist research model. Handwritten-manuscript support is the next training run.

rokot-omr-2b turns images of printed sheet music — scans, photos, pages out of a PDF — into machine-readable notation. Show it a system of music and it writes rokot-ABC, a strict, canonical profile of ABC notation that converts losslessly to MusicXML. From there your score opens directly in MuseScore, Finale, Dorico, or anything else that speaks MusicXML — ready to edit, transpose, and play back.

Because the output is plain ABC text, the model is small enough and simple enough to run locally on your own machine — including in LM Studio on a Mac: load the model, drop in a cropped image of a staff, and copy the ABC out. Standard ABC tooling (abc2xml, EasyABC, abcjs, ...) reads rokot-ABC as-is, since it is exactly one deterministic spelling of ordinary ABC.

It is a full fine-tune of Qwen/Qwen3-VL-2B-Instruct (2.1 B parameters, Apache-2.0 base), trained on a large synthetic corpus of engraved scores rendered with realistic page degradation — so it is built for the sheet music people actually have: photocopies, songbook scans, lead sheets with chords and lyrics, not just clean digital engravings.

What it can read

• Single staff (treble or bass) and grand-staff systems, up to 2 voices per staff
• Pitches, accidentals, rests, chords, tuplets, dotted values from breve to 1/64
• Key/meter/clef changes mid-score, repeats, voltas, ties, slurs, grace notes
• Ornaments, articulations, dynamics and hairpins
• Chord symbols (root/quality/bass), tempo marks
• Lyrics with multiple verses, melisma and syllable continuation

Out of scope (for now): more than one part, percussion/TAB, nested tuplets, and handwritten manuscripts — that last one is actively being worked on.

Quick start in LM Studio (Mac & PC)

GGUF builds are included in this repo for local use with LM Studio or llama.cpp:

File	What it is	Size
rokot-omr-2b-Q8_0.gguf	Recommended — 8-bit, verified output matches the full model	~1.8 GB
rokot-omr-2b-f16.gguf	Full-precision text model	~3.4 GB
mmproj-rokot-omr-2b-f16.gguf	Vision projector — required alongside either model	~0.8 GB

Search for rokotmidi/rokot-omr-2b in LM Studio's Discover tab (it fetches the vision projector automatically), drop in an image of a staff, and ask: "Transcribe this sheet music to rokot-ABC."

A note on quantization: we deliberately do not publish 4-bit builds. This model writes a strict notation language where a single wrong token corrupts the score, and in our tests Q4 visibly degraded it. Q8_0 matched the full-precision model exactly on our eval samples — use that.

Usage with transformers

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "rokotmidi/rokot-omr-2b"
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": "staff.png"},
        {"type": "text", "text": "Transcribe this staff to rokot-ABC."},
    ],
}]

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

out = model.generate(**inputs, max_new_tokens=1600, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Input images should be one system (one line of music) cropped to the staff, ideally around 1400 px wide. The output is a complete, self-contained ABC tune with fixed header order (X, M, L, K, ...) — one deterministic spelling for everything, so downstream tooling can rely on it.

Support the project ❤️

This model is free for personal and research use, and it exists because of RokoTMIDI — a compact MIDI controller with the button layout of a chromatic button accordion. If this model saves you hours of transcription work and you'd like to give something back, the nicest way to do it is to grab a controller (or tell an accordionist friend about it):

👉 rokotmidi.com

Every controller sold funds more GPU hours, better datasets, and future versions of this model — including the handwritten-manuscript support that is currently in training. Thank you! 🪗

License

Released under CC BY-NC 4.0 — free to use, share and build on for non-commercial purposes with attribution. For commercial licensing, get in touch via rokotmidi.com.

Limitations

• One part / max two staves per system; complex piano textures with more than two voices per staff are simplified.
• Handwritten manuscripts are not yet supported (next training run).
• Lyrics in non-Latin scripts remain less reliable than notation.
• Very low-resolution or heavily skewed crops degrade accuracy — deskew and crop per system for best results.