OAB Exams is a dataset of more than 2,000 questions from the Brazilian Bar Association's exams, from 2010 to 2018. This task evaluates models on the multiple-choice questions from these exams. The model is given a question in Portuguese together with its answer choices and must respond with the correct alternative letter (A, B, C, or D), evaluated in a 3-shot setting.
This is a copy/port of the OAB exams task, straight out of the lm-evaluation-harness-pt (maintained by… See the full description on the dataset page:
https://huggingface.co/datasets/nicholasKluge/oab_exams.
