gist-sparse-attention/

GSA-PT-Qwen2-7B-Instruct-chunk16-data

This is the continue pretraining dataset used for training GSA (Gist Sparse Attention) models based on Qwen2-7B-Instruct with chunk size chunk16. Each sample is tokenized and formatted with GSA gist tokens for continue pretraining.

GSA: Gist Sparse Attention via Learnable Compression and Selective Unfolding

yuzhenm/GSA-PT-Qwen2-7B-Instruct-chunk16 — model trained on this dataset