Finepdfs filtered using this code:
def my_filter(example):
lscore = example["page_average_lid_score"]example["full_doc_lid_score"] > 0.850.85
lenglish = example["language"] == "eng_Latn" and example["page_average_lid"] == "eng_Latn" and example["full_doc_lid"] == "eng_Latn"
fwscore = (sum(example["fw_edu_scores"])/len(example["fw_edu_scores"])) > 3
return lscore and lenglish and fwscore
This remove 87% of the dataset in average. This filtering have been applied on 1/4 of the… See the full description on the dataset page:
https://huggingface.co/datasets/styal/filtered-finepdfs-edu.eng.
