Korean QA/Search dataset created from KorQuAD2 and AIHub official documents datasets.
See explore_dataset.ipynb notebook for details.
The data is created by whitespace tokenization, windowing the tokens into 256 windows, and creating passages with 128 token overlap.
Labels are based on the QA dataset answer spans. Any question overlap with passage is tagged as hard label. Results from BM25 and KoBigBird Cross-Encoder are provided for soft labels.… See the full description on the dataset page:
https://huggingface.co/datasets/neoALI/kq2-aihub-256w-128s.
