This model is a distilled version of projecte-aina/roberta-base-ca-v2.

It follows the same training procedure as DistilBERT, using the implementation of Knowledge Distillation from the paper's official repository.

The resulting architecture consists of 6 layers, 768 dimensional embeddings and 12 attention heads. This adds up to a total of 82M parameters, which is considerably less than the 125M of standard RoBERTa-base models. This makes the model lighter and faster than the original, at the cost of a slightly lower performance.

This model has been trained using a technique known as Knowledge Distillation, which is used to shrink networks to a reasonable size while minimizing the loss in performance.

It basically consists in distilling a large language model (the teacher) into a more lightweight, energy-efficient, and production-friendly model (the student).

So, in a “teacher-student learning” setup, a relatively small student model is trained to mimic the behavior of a larger teacher model. As a result, the student has lower inference time and the ability to run in commodity hardware.

The training corpus consists of several corpora gathered from web crawling and public corpora, as shown in the table below:

This model has been fine-tuned on the downstream tasks of the Catalan Language Understanding Evaluation benchmark (CLUB), which includes the following datasets:

This is how it compares to its teacher when fine-tuned on the aforementioned downstream tasks:

¹ : Trained on CatalanQA, tested on XQuAD-ca (no train set).