Micka Gen3 is a specialized language model based on the Microsoft RetNet architecture, fine-tuned for Retrieval-Augmented Generation (RAG) usage in Slovenian Cultural Heritage Domain.
It leverages an efficient retention mechanism, and should be used as baseline and in combination with the GAMS series of models.
A standalone series of models, based on the GaMS model will also be released.
Training Data
The model was trained from scratch on:
GigaFida corpus (Slovenian)
Slovenian Wikipedia
Random subset of 10,000 English Wikipedia articles
The model underwent 20 epochs of training on the above datasets.
Finetuning
The final stage involved finetuning on 10,000 culturally relevant samples prepared specifically for the Povejmo Project, focusing on cultural heritage content.
The architecture is optimized for long-context document retrieval and generation tasks in combination with large Generative AI models.
Usage
Designed specifically for Retrieval-Augmented Generation (RAG), Micka-Gen3 performs well in:
Generating contextually accurate responses from Cultural Heritage Texts.
Funding
The development of the Micka Tokenizer was partially funded by the PoVeJMo project, which aims to develop large language models for the Slovenian language.
The project PoVeJMo is cofinanced by:
@misc{micka-gen3,
author = {Semantika Research},
title = {Micka-Gen3: A RetNet-based Slovenian Language Model for RAG tasks},
year = {2024},
publisher = {Hugging Face},
url = {https://huggingface.co/klokedm/micka-gen3}
}
