Musicroll-50M
Musicroll-50M is a small Qwen3-style causal language model over a 259-token
byte-level music vocabulary. It was used as the MusicRoll negative-control model
in the replication artifact for "Many Next-Token Predictors are In-Context
Learners".
The checkpoint is provided as model.safetensors with a compact tokenizer:
- vocabulary size: 259
- hidden size: 512
- layers: 16
- attention heads: 4
- key/value heads: 2
- context length: 4096
- BOS/EOS/PAD token ids: 256/257/258
The bitstring replication harness encodes tasks as piano-roll-like text and
loads this checkpoint through the local MLX model loader.
Intended Use
This upload is primarily for reproducing the MusicRoll negative-control runs in
the ICLManyReplication artifact. General music-generation behavior has not been
characterized here.
