Kanarya is a pre-trained Turkish GPT-J 2B model. Released as part of Turkish Data Depository efforts, the Kanarya family has two versions (Kanarya-2B, Kanarya-0.7B). Kanarya-2B is the larger version and Kanarya-0.7B is the smaller version. Both models are trained on a large-scale Turkish text corpus, filtered from OSCAR and mC4 datasets. The training data is collected from various sources, including news, articles, and websites, to create a diverse and high-quality dataset. The models are trained using a JAX/Flax implementation of the GPT-J architecture.
Model Details
Model Name: Kanarya-2B
Model Size: 2,050M parameters
Training Data: OSCAR, mC4
Language: Turkish
Layers: 24
Hidden Size: 2560
Number of Heads: 20
Context Size: 2048
Positional Embeddings: Rotary
Vocabulary Size: 32,768
Intended Use
This model is only pre-trained on Turkish text data and is intended to be fine-tuned on a wide range of Turkish NLP tasks. The model can be used for various Turkish NLP tasks, including text generation, translation, summarization, and other Turkish NLP tasks. This model is not intended to be used for any downstream tasks without fine-tuning.
Limitations and Ethical Considerations
The model is trained on a diverse and high-quality Turkish text corpus, but it may still generate toxic, biased, or unethical content. It is highly recommended to use the model responsibly and make sure that the generated content is appropriate for the use case. Please use the model responsibly and report any issues.
License: Apache 2.0
The model is licensed under the Apache 2.0 License. It is free to use for any purpose, including commercial use. We encourage users to contribute to the model and report any issues. However, the model is provided "as is" without warranty of any kind.
Citation
If you use the model, please cite the following paper:
bibtex
1@inproceedings{safaya-etal-2022-mukayese,
2 title = "Mukayese: {T}urkish {NLP} Strikes Back",
3 author = "Safaya, Ali and
4 Kurtulu{\c{s}}, Emirhan and
5 Goktogan, Arda and
6 Yuret, Deniz",
7 editor = "Muresan, Smaranda and
8 Nakov, Preslav and
9 Villavicencio, Aline",
10 booktitle = "Findings of the Association for Computational Linguistics: ACL 2022",
11 month = may,
12 year = "2022",
13 address = "Dublin, Ireland",
14 publisher = "Association for Computational Linguistics",
15 url = "https://aclanthology.org/2022.findings-acl.69",
16 doi = "10.18653/v1/2022.findings-acl.69",
17 pages = "846--863",
18}
Acknowledgments
During this work, Ali Safaya was supported by KUIS AI Center fellowship. Moreover, the pre-training of these models were performed at TUBITAK ULAKBIM, High Performance and Grid Computing Center (TRUBA resources).
