lorenzo0312/

degeneration-probe-instruct-token-level-balanced

Downsampled (1:3 positive:negative) variant of luca-sartori/degeneration-probe-instruct-token-level. An example is considered positive if it contains at least one token with repetition >= 0.8 in the chunk_summary field. The downsampling keeps all positive examples and adds a random subset of negative examples in a 1:3 ratio. Examples whose chunk_summary contained no scored tokens (every repetition value null) have been dropped, since… See the full description on the dataset page: https://huggingface.co/datasets/lorenzo0312/degeneration-probe-instruct-token-level-balanced.