Large pre-trained language models have shown promise for few-shot learning, completing text-based tasks given only a few task-specific examples. Will models soon solve classification tasks that have so far been reserved for human research assistants?

RAFT is a few-shot classification benchmark that tests language models:

across multiple domains (lit review, tweets, customer interaction, etc.)
on economically valuable classification tasks (someone inherently cares about the task)
in a setting that mirrors deployment (50 examples per task, info retrieval allowed, hidden test set)

ought/raft

ought/
raft