Splinter-base is the pretrained model discussed in the paper Few-Shot Question Answering by Pretraining Span Selection (at ACL 2021). Its original repository can be found here. The model is case-sensitive.

Note: This model doesn't contain the pretrained weights for the QASS layer (see paper for details), and therefore the QASS layer is randomly initialized upon loading it. For the model with those weights, see tau/splinter-base-qass.

Splinter is a model that is pretrained in a self-supervised fashion for few-shot question answering. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts.

More precisely, it was pretrained with the Recurring Span Selection (RSS) objective, which emulates the span selection process involved in extractive question answering. Given a text, clusters of recurring spans (n-grams that appear more than once in the text) are first identified. For each such cluster, all of its instances but one are replaced with a special [QUESTION] token, and the model should select the correct (i.e., unmasked) span for each masked one. The model also defines the Question-Aware Span selection (QASS) layer, which selects spans conditioned on a specific question (in order to perform multiple predictions).

The prime use for this model is few-shot extractive QA.

The model was pretrained on a v3-8 TPU for 2.4M steps. The training data is based on Wikipedia and BookCorpus. See the paper for more details.