FlauBERT-Oral models: Using ASR-Generated Text for Spoken Language Modeling

Available FlauBERT-Oral models

• flaubert-oral-asr : trained from scratch on ASR data, keeping the BPE tokenizer and vocabulary of flaubert-base-uncased
• flaubert-oral-asr_nb : trained from scratch on ASR data, BPE tokenizer is also trained on the same corpus
• flaubert-oral-mixed : trained from scratch on a mixed corpus of ASR and text data, BPE tokenizer is also trained on the same corpus
• flaubert-oral-ft : fine-tuning of flaubert-base-uncased for a few epochs on ASR data

Usage for sequence classification

References

@InProceedings{herve2022flaubertoral,
  author    = {Herv\'{e}, Nicolas and Pelloin, Valentin and Favre, Benoit and Dary, Franck and Laurent, Antoine and Meignier, Sylvain and Besacier, Laurent},
  title     = {Using ASR-Generated Text for Spoken Language Modeling},
  booktitle = {Proceedings of "Challenges & Perspectives in Creating Large Language Models" ACL 2022 Workshop},
  month     = {May},
  year      = {2022}
}