Saba is a BERT model for Italian poetry. It was obtained via continued pretraining of dbmdz/bert-base-italian-xxl-cased on ~40k Italian song lyrics from Wikisource and Biblioteca Italiana. The objective was Masked Language Modeling (MLM).

All the processing, training and evaluation code is available on GitHub.

We evaluate Saba, the base model and Alberti (a prominent multilingual model for poetry) on various tasks.

Evaluates the model's statistical modeling of Italian poetry by computing the cross-entropy loss and pseudo-perplexity (PPPL) on test poems with 15% masked tokens.

Evaluates the model's ability to capture phonetic information by predicting the rhyme (the last three characters) of words using a multinomial logistic regression classifier trained on the embeddings of multiple words. We consider the top 50 most frequent rhymes for evaluation.

Evaluates stylistic understanding by predicting the author of a poem using a multinomial logistic regressor fitted on the [CLS] token embeddings.

Assesses the model's generative abilities through masked language modeling by predicting the masked final word of a poem, evaluated on both token-level and word-level accuracy.

Analyzes the model's internal computations by calculating the average attention score for the last tokens of previous lines (both rhyming and non-rhyming) to determine if it has developed specific mechanisms to detect and process rhymes.

Following the tradition of giving Italian names to BERT models for the Italian language (see AlBERTo, GilBERTo, UmBERTo), we dedicate this model to the Italian poet and novelist Umberto Saba (9 March 1883 – 25 August 1957).