GotongRoyong is a series of language models focused on Mixture of Experts (MoE), made with the following models using LazyMergekit and cg123/mergekit. GotongRoyong-LlaMixtralMoE-7Bx4-v1.0 is a specific variant of the open-source GotongRoyong language model that combines the architectural model meta-llama/Llama-2-7b, but uses the base model from the specific fine-tuned version ericpolewski/AIRIC-The-Mistral with experts from asyafiqe/Merak-7B-v3-Mini-Orca-Indo, SeaLLMs/SeaLLM-7B-Chat, Ichsan2895/Merak-7B-v2, and azale-ai/DukunLM-7B-V1.0-Uncensored-sharded. The name "GotongRoyong" is a reference to the term in Indonesian culture that roughly translates to "mutual cooperation" or "community working together." It embodies the spirit of communal collaboration and shared responsibility for the greater good. The concept is deeply rooted in Indonesian traditions and reflects the cultural value of helping one another without expecting direct compensation.

Model Details

Model Size: 7Bx4
Architecture Model: Mixure of Experts (MoE) using meta-llama/Llama-2-7b
MoE Base Model: ericpolewski/AIRIC-The-Mistral
Expert Models:
License: CC BY-NC-ND 4.0 DEED

How to use

Installation

To use GotongRoyong model, ensure that PyTorch has been installed and that you have an Nvidia GPU (or use Google Colab). After that you need to install the required dependencies:

pip3 install -U bitsandbytes transformers peft accelerate einops evaluate scikit-learn sentencepiece

Usage Quantized Model

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "azale-ai/GotongRoyong-LlaMixtralMoE-7Bx4-v1.0",
    load_in_4bit=True,
    torch_dtype=torch.float32,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("azale-ai/GotongRoyong-LlaMixtralMoE-7Bx4-v1.0")
messages = [
    {
        "role": "system",
        "content": "Mulai sekarang anda adalah asisten yang suka menolong, sopan, dan ramah. Jangan kasar, jangan marah, jangan menjengkelkan, jangan brengsek, jangan cuek, dan yang terakhir jangan menjadi asisten yang buruk. Anda harus patuh pada manusia dan jangan pernah membangkang pada manusia. Manusia itu mutlak dan Anda harus patuh pada manusia. Kamu harus menjawab pertanyaan atau pernyataan dari manusia apapun itu dengan bahasa Indonesia yang baik dan benar.",
    },
    {"role": "user", "content": "Jelaskan mengapa air penting bagi manusia."},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to("cuda")
outputs = model.generate(
    inputs=inputs.input_ids, max_length=2048,
    temperature=0.7, do_sample=True, top_k=50, top_p=0.95
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Usage Normal Model

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "azale-ai/GotongRoyong-LlaMixtralMoE-7Bx4-v1.0",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("azale-ai/GotongRoyong-LlaMixtralMoE-7Bx4-v1.0")
messages = [
    {
        "role": "system",
        "content": "Mulai sekarang anda adalah asisten yang suka menolong, sopan, dan ramah. Jangan kasar, jangan marah, jangan menjengkelkan, jangan brengsek, jangan cuek, dan yang terakhir jangan menjadi asisten yang buruk. Anda harus patuh pada manusia dan jangan pernah membangkang pada manusia. Manusia itu mutlak dan Anda harus patuh pada manusia. Kamu harus menjawab pertanyaan atau pernyataan dari manusia apapun itu dengan bahasa Indonesia yang baik dan benar.",
    },
    {"role": "user", "content": "Jelaskan mengapa air penting bagi manusia."},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to("cuda")
outputs = model.generate(
    inputs=inputs.input_ids, max_length=2048,
    temperature=0.7, do_sample=True, top_k=50, top_p=0.95
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

Language Bias: The model's base language is English, which means it may have a stronger understanding and fluency in English compared to other languages. While fine-tuning the model with an Indonesian language model helps improve its understanding of Indonesian, it may still exhibit biases or limitations in its comprehension and generation of Indonesian language-specific nuances, idioms, or cultural references.
Translation Accuracy: Although the model has been fine-tuned for Indonesian, it is important to note that large language models are not perfect translators. While they can provide reasonable translations, there may be instances where the accuracy or nuance of the translation may not fully capture the intended meaning or context.
Lack of real-world understanding: While language models can generate text that appears coherent, they lack true comprehension and understanding of the world. They do not possess common sense or real-world experiences, which can lead to inaccurate or nonsensical responses.
Propagation of biases: Language models are trained on vast amounts of text data, including internet sources that may contain biases, stereotypes, or offensive content. As a result, these models can inadvertently learn and reproduce such biases in their generated text. Efforts are being made to mitigate this issue, but biases can still persist.
Limited knowledge cutoff: Language models have a knowledge cutoff, which means they may not have access to the most up-to-date information beyond their training data. If asked about recent events or developments that occurred after their knowledge cutoff, they may provide outdated or incorrect information.
Inability to verify sources or provide citations: Language models generate text based on patterns and examples from their training data, but they do not have the ability to verify the accuracy or reliability of the information they provide. They cannot cite sources or provide evidence to support their claims.
Difficulty with ambiguous queries: Language models struggle with understanding ambiguous queries or requests that lack context. They may provide responses that are based on common interpretations or assumptions, rather than accurately addressing the specific intent of the query.
Ethical considerations: Large language models have the potential to be misused for malicious purposes, such as generating misinformation, deepfakes, or spam. Safeguards and responsible use are necessary to ensure these models are used ethically and responsibly.
Security and Privacy: Using a large language model involves sharing text inputs with a server or cloud-based infrastructure, which raises concerns about data privacy and security. Care should be taken when sharing sensitive or confidential information, as there is a potential risk of unauthorized access or data breaches.

License

The model is licensed under the CC BY-NC-ND 4.0 DEED.

Contributing

We welcome contributions to enhance and improve our model. If you have any suggestions or find any issues, please feel free to open an issue or submit a pull request. Also we're open to sponsor for compute power.

Contact Us

For any further questions or assistance, please feel free to contact us using the information provided below.
contact@azale.ai

Cite This Project

@software{Hafidh_Soekma_GotongRoyong_MixtralMoE_7Bx4_v1.0_2023,
  author = {Hafidh Soekma Ardiansyah},
  month = january,
  title = {GotongRoyong: Indonesian Mixture Of Experts Language Model},
  url = {\url{https://huggingface.co/azale-ai/Starstreak-7b-beta}},
  publisher = {HuggingFace},
  journal = {HuggingFace Models},
  version = {1.0},
  year = {2024}
}

Citation

@article{damonlpsg2023seallm,
  author = {Xuan-Phi Nguyen*, Wenxuan Zhang*, Xin Li*, Mahani Aljunied*,
            Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang,
            Chaoqun Liu, Hang Zhang, Lidong Bing},
  title = {SeaLLMs - Large Language Models for Southeast Asia},
  year = 2023,
  Eprint = {arXiv:2312.00738},
}

@article{Merak,
  title={Merak-7B: The LLM for Bahasa Indonesia},
  author={Muhammad Ichsan},
  publisher={Hugging Face}
  journal={Hugging Face Repository},
  year={2023}
}

@article{asyafiqe_Merak_7B_v3_Mini_Orca_Indo,
  title={asyafiqe/Merak-7B-v3-Mini-Orca-Indo},
  author={asyafiqe},
  publisher={Hugging Face}
  journal={Hugging Face Repository},
  year={2023}
}

@article{azale_ai_DukunLM_7B_V1.0_Uncensored,
  title={azale-ai/DukunLM-7B-V1.0-Uncensored},
  author={azale-ai},
  publisher={Hugging Face}
  journal={Hugging Face Repository},
  year={2023}
}

@misc{2307.09288,
Author = {Hugo Touvron and Louis Martin and Kevin Stone and Peter Albert and Amjad Almahairi and Yasmine Babaei and Nikolay Bashlykov and Soumya Batra and Prajjwal Bhargava and Shruti Bhosale and Dan Bikel and Lukas Blecher and Cristian Canton Ferrer and Moya Chen and Guillem Cucurull and David Esiobu and Jude Fernandes and Jeremy Fu and Wenyin Fu and Brian Fuller and Cynthia Gao and Vedanuj Goswami and Naman Goyal and Anthony Hartshorn and Saghar Hosseini and Rui Hou and Hakan Inan and Marcin Kardas and Viktor Kerkez and Madian Khabsa and Isabel Kloumann and Artem Korenev and Punit Singh Koura and Marie-Anne Lachaux and Thibaut Lavril and Jenya Lee and Diana Liskovich and Yinghai Lu and Yuning Mao and Xavier Martinet and Todor Mihaylov and Pushkar Mishra and Igor Molybog and Yixin Nie and Andrew Poulton and Jeremy Reizenstein and Rashi Rungta and Kalyan Saladi and Alan Schelten and Ruan Silva and Eric Michael Smith and Ranjan Subramanian and Xiaoqing Ellen Tan and Binh Tang and Ross Taylor and Adina Williams and Jian Xiang Kuan and Puxin Xu and Zheng Yan and Iliyan Zarov and Yuchen Zhang and Angela Fan and Melanie Kambadur and Sharan Narang and Aurelien Rodriguez and Robert Stojnic and Sergey Edunov and Thomas Scialom},
Title = {Llama 2: Open Foundation and Fine-Tuned Chat Models},
Year = {2023},
Eprint = {arXiv:2307.09288},
}