Per-token degeneration variant of the instruct generations. chunk_summary is a list of {token_index, degenerating, repetition} records, where repetition is the average bigram-repetition score over all length-256 sliding windows that contain the token, and degenerating is True when that average is >= 0.8.
The rows are sourced from /Users/lucae/Desktop/degeneration/degeneration-probe/outputs/hf_datasets/instruct_concatenated/data… See the full description on the dataset page:
https://huggingface.co/datasets/luca-sartori/degeneration-probe-instruct-token-level.
