BahasaGPT-1 Fine-Tuning Documentation Summary (INT (8-BIT))
Introduction
This document provides an overview of the BahasaGPT-1 model, which is a fine-tuned model for a specific task in the Indonesian language. The model is based on the Bloomz-7B-mt architecture and is fine-tuned using a dataset of over 70,000 Indonesian instructions.
Model Details
Model Name: BahasaGPT-1
Model Source: Bloomz-7B-mt
Dataset for Fine-Tuning: Over 70k Indonesia Instruct Dataset generated using the Alpaca method from the following sources:
Fine-Tuning Process
The BahasaGPT-1 model was fine-tuned using a dataset of over 70,000 Indonesian instructions, which were generated using the Alpaca method from Stanford and translated instructions from OA. This combination of datasets allowed the model to be better adapted to the specific needs of Indonesian language tasks.
The fine-tuning process involved adjusting the model's weights and biases based on the input dataset. This was done iteratively to optimize the model's performance for the specific task in the Indonesian language.
Known Limitations
Despite the successful fine-tuning, the BahasaGPT-1 model still has some limitations:
-
Hallucination: The model sometimes generates outputs that may seem plausible but are not based on the input data. This may lead to incorrect or nonsensical responses in some cases.
-
Repeated Tokens: The model occasionally produces repeated tokens in the output, which may affect the overall coherence and readability of the generated text.
Conclusion
The BahasaGPT-1 model is a fine-tuned language model for Indonesian language tasks, based on the Bloomz-7B-mt architecture. The model was trained on a dataset of over 70,000 Indonesian instructions generated using the Alpaca method and translated instructions from OA. Despite some limitations, such as occasional hallucination and repeated tokens, the model provides a valuable tool for working with Indonesian language tasks.
