MusicBERT large is a 24-layer BERT-style masked language model trained on REMI+BPE symbolic music sequences extracted from the GigaMIDI corpus. It is tailored for symbolic music understanding, fill-mask style infilling, and as a backbone for downstream generative tasks.

Checkpoint: 120000 steps
Hidden size: 1024
Parameters: ~330M
Validation loss: 1.19900381565094

Training Configuration

Objective: Masked language modeling with span-aware masking
Dataset: GigaMIDI (REMI tokens → BPE, vocab size 50000)
Sequence length: 1024
Max events per MIDI: 2048

Inference Example

Using with MIDI files

python

import torch
from transformers import BertForMaskedLM
from miditok import MusicTokenizer

# Load model and tokenizer
model = BertForMaskedLM.from_pretrained("manoskary/musicbert-large")
tokenizer = MusicTokenizer.from_pretrained("manoskary/miditok-REMI")

# Convert MIDI to BPE tokens (MIDI → REMI → BPE pipeline)
midi_path = "path/to/your/file.mid"
tok_seq = tokenizer(midi_path)
bpe_ids = tok_seq.ids

# Mask some tokens for prediction
import random
mask_token_id = 3  # MASK_None token
input_ids = bpe_ids.copy()
mask_positions = random.sample(range(1, len(input_ids)-1), k=5)
for pos in mask_positions:
    input_ids[pos] = mask_token_id

# Run inference
input_tensor = torch.tensor([input_ids])
with torch.no_grad():
    outputs = model(input_tensor)
    predictions = outputs.logits[0, mask_positions, :].argmax(dim=-1)

print("Predicted token IDs:", predictions.tolist())

Limitations and Risks

Model is trained purely on symbolic data; it does not produce audio directly.
The GigaMIDI dataset is biased towards Western tonal music.
Long-form structure beyond 1024 tokens requires chunking or iterative decoding.
Generated continuations may need post-processing to ensure musical coherence.

Citation

If you use this checkpoint, please cite the original MusicBERT introduction and the GigaMIDI dataset.