You need to manually combine each dataset if you want to use a multilingual dataset.
from datasets import load_dataset
xcsn_pt_python_en = load_dataset("ynklab/XCodeSearchNet", data_dir='pretraining/python/en')
"""
DatasetDict({
train: Dataset({
features: ['function_tokens', 'docstring'],
num_rows: 453623
})
validation: Dataset({
features: ['function_tokens', 'docstring'],
num_rows: 4596
})
test:… See the full description on the dataset page:
https://huggingface.co/datasets/ynklab/XCodeSearchNet.
