This model is a fine-tuned version for named-entity recognition of the Spanish bertin-project/bertin-gpt-j-6B checkpoint.
This model has been developed during the 2023 Hackaton organized by SomosNLP with the GPUs RTX 3090 provided by Q Blocks.
Motivation of the project
Podcasts are an incredible source of information and inspiration. We can listen to them while commuting, practising sport or cooking our favourite recipe. However, it can be difficult to retain specific facts, dates or people mentioned in them. The aim of this project has been to explore how to capture all those facts using named-entity recognition.
Instead of using a language model fine-tuned with a specific NER head, we have reframed the problem as text generation from a prompt of the kind:
text: Yo hoy voy a hablar de mujeres en el mundo del arte, porque me ha leído un libro fantástico que se llama Historia del arte sin hombres, de Katie Hesel. entities: (people, Katie Hesel), (books, Historia del arte sin hombres)
By fine-tuning a large generative model with this prompt, we are able to capture the entities mentioned in the podcast. We fine-tuned the bertin-gpt-j-6B following this strategy.
Training was performed in a RTX 3090 kindly provided by Q Blocks. It took 2h20m.
We employed the Low-Rank Adaptation (LoRA) strategy to substantially reduce the number of trainable parameters for downstream tasks while maintaining model quality.
The pre-trained checkpoint employed was bertin-project/bertin-gpt-j-6B. We didn't perform an extensive hyperparameter sweep so there's room to improve it.
