The Journaux-LM is a language model pretrained on historical French newspapers. Technically the model itself is an ELECTRA model, which was pretrained with the TEAMS approach.

Version 1 of the Journaux-LM was pretrained on the following publicly available datasets:

In total, the pretraining corpus has a size of 408GB.

We compare our Zeitungs-LM directly to the French Europeana BERT model (as Zeitungs-LM is supposed to be the successor of it) on various downstream tasks from the hmBench repository, which is focussed on Named Entity Recognition.

We report averaged micro F1-Score over 5 runs with different seeds and use the best hyper-parameter configuration on the development set of each dataset to report the final test score.

The results on the development set can be seen in the following table:

Our Journaux-LM leads to a performance boost of 1.12% compared to the German Europeana BERT model.

The final results on the test set can be seen here:

Our Journaux-LM beats the French Europeana BERT model by 0.98%.

Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Many Thanks for providing access to the TPUs ❤️

Made from Bavarian Oberland with ❤️ and 🥨.