January 1, 2024 version of PanLex Language Vocabulary with 24,650,274 rows covering 6,152 languages.
vocab: contains the text entry.
639-3: contains the ISO 639-3 languages tags to allow users to filter on the language(s) of their choice.
639-3_english_name: the English language name associated to the code ISO 639-3.
var_code: contains a code to differentiate language variants. In practice, this is the code 639-3 + a number. If 000, it corresponds to… See the full description on the dataset page:
https://huggingface.co/datasets/lbourdois/panlex.
