units-surprisal: English data and pipeline intermediates
Everything needed to reproduce the English GAMM results of the units-surprisal
paper (code). The full multilingual
MECO data is in the companion dataset
samuki-hf/meco-char.
Contents (place under src/units_surprisal/data/ in the code repo)
Folder
Pipeline stage
Description
meco_source/
stage 0
MECO source CSVs (passage_data_version1.3.csv, sentences.csv, supp_texts.csv), copied to the data root by… See the full description on the dataset page: https://huggingface.co/datasets/samuki-hf/units-surprisal.
