Note on padding and training: This dataset contains some sequences that are already padded to a fixed length. If you train a causal language model and simply copy input_ids to labels without masking, the model will also learn to predict padding tokens, which can lead to artificially low loss values and misleading training results. To avoid this, you should mask padding tokens in the labels by setting them to -100 (so they are ignored by the loss function). For example:
def… See the full description on the dataset page:
https://huggingface.co/datasets/TylerHMC/GigaMIDI-starts-set.
