You can deploy this model and start earning money today!

RichardErkhov/
MaralGPT_-_Maral-7B-alpha-1-gguf

Name: MaralGPT_-_Maral-7B-alpha-1-gguf
Availability: InStock

endpoints_compatible

gguf

template

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Maral-7B-alpha-1 - GGUF

Model creator: https://huggingface.co/MaralGPT/
Original model: https://huggingface.co/MaralGPT/Maral-7B-alpha-1/

Name	Quant method	Size
Maral-7B-alpha-1.Q2_K.gguf	Q2_K	2.53GB
Maral-7B-alpha-1.IQ3_XS.gguf	IQ3_XS	2.81GB
Maral-7B-alpha-1.IQ3_S.gguf	IQ3_S	2.96GB
Maral-7B-alpha-1.Q3_K_S.gguf	Q3_K_S	2.95GB
Maral-7B-alpha-1.IQ3_M.gguf	IQ3_M	3.06GB
Maral-7B-alpha-1.Q3_K.gguf	Q3_K	3.28GB
Maral-7B-alpha-1.Q3_K_M.gguf	Q3_K_M	3.28GB
Maral-7B-alpha-1.Q3_K_L.gguf	Q3_K_L	3.56GB
Maral-7B-alpha-1.IQ4_XS.gguf	IQ4_XS	3.67GB
Maral-7B-alpha-1.Q4_0.gguf	Q4_0	3.83GB
Maral-7B-alpha-1.IQ4_NL.gguf	IQ4_NL	3.87GB
Maral-7B-alpha-1.Q4_K_S.gguf	Q4_K_S	3.86GB
Maral-7B-alpha-1.Q4_K.gguf	Q4_K	4.07GB
Maral-7B-alpha-1.Q4_K_M.gguf	Q4_K_M	4.07GB
Maral-7B-alpha-1.Q4_1.gguf	Q4_1	4.24GB
Maral-7B-alpha-1.Q5_0.gguf	Q5_0	4.65GB
Maral-7B-alpha-1.Q5_K_S.gguf	Q5_K_S	4.65GB
Maral-7B-alpha-1.Q5_K.gguf	Q5_K	4.78GB
Maral-7B-alpha-1.Q5_K_M.gguf	Q5_K_M	4.78GB
Maral-7B-alpha-1.Q5_1.gguf	Q5_1	5.07GB
Maral-7B-alpha-1.Q6_K.gguf	Q6_K	5.53GB
Maral-7B-alpha-1.Q8_0.gguf	Q8_0	7.17GB

Original model description:

license: mit datasets:

sinarashidi/alpaca-persian language:
en
fa library_name: transformers

Maral 7B Alpha 1

What is Maral?

Maral is just a new large lanugage model, specializing on the Persian language. This model is based on Mistral and trained an Alpaca Persian dataset. This model is one of the few efforts in Persian speaking scene in order to bring our language to a new life in the era of AI.

Also, since Maral is based on Mistral, it's capable of producing English answers as well.

What does "Maral" mean?

Maral is the Persian name of Red Deer, which is a native species of deers in Iran. The name has chosen for quite a few reasons, one of them is that the environmental concerns we have and second, since it's a Persian LLM, made by Iranian people, it deserves an Iranian name.

Inference

Prompt Format

This model requires Guanaco format, which is like this:

### Human: <prompt>
### Assistant: <answer>

So in your code, you may write prompts like this:

python

prompt = "در سال ۱۹۹۶ چه کسی رییس جمهور آمریکا بود؟"
prompt = f"### Human:{prompt}\n### Assistant:"

More information about this on the inference sections.

4 bit Quantization

If you want to use 4 bit quantization, we have a PEFT for you here. Also, you can find Google Colab notebooks here.

Installing Libraries

pip install transformers accelerate bitsandbytes

NOTE: bitsandbytes library is only needed for 8 bit version. Otherwise, it's not necessary.

Inference on a big GPU

If you have a big enough GPU like an A100 in your posession, this code is for you.

python

from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
import torch

model_name_or_id = "MaralGPT/Maral-7B-alpha-1"

model = AutoModelForCausalLM.from_pretrained(model_name_or_id, torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name_or_id)

prompt = "در سال ۱۹۹۶ چه کسی رییس جمهور آمریکا بود؟"
prompt = f"### Human:{prompt}\n### Assistant:"

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

generation_config = GenerationConfig(
    do_sample=True,
    top_k=1,
    temperature=0.5,
    max_new_tokens=300,
    pad_token_id=tokenizer.eos_token_id
)

outputs = model.generate(**inputs, generation_config=generation_config)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Inference on a small GPU (Consumer Hardware/Free Colab)

The code is pretty much the same as above, but with a slight diferrence.

Make sure bitsandbytes is installed correctly.
Your model loading must be model = AutoModelForCausalLM.from_pretrained(model_name_or_id, load_in_8bit=True, torch_dtype=torch.bfloat16, device_map="auto")

On free version of Google Colab, you may face RAM problems. I guess using low_cpu_mem_usage=True in model loading would help.

Known Issues

The model produces GPT-3.5 level answers in terms of grammar (specially Persian) but is capable of extremely insane hallucinations. This problem can be solved by a better dataset and better training procedures (such as DPO).
According to the previous issue, the model can also generate misinforming answers specially when dealing with reasoning problems in Persian.
The model is huge, so it requires a lot of resources in order to work correctly. However, we may provide GPTQ or GGUF versions as well.
The prompt format works and it proves our concept of a instruct following LLM, but since we haven't changed eos_token and bos_token to our own, you may see unncessary information being generated by the model.
According to the previous issue, the model is capable of repeating itself. To solve this problem temporarily you have to keep temperature below 1. According to our tests somewhere between 0.5 to 0.7 is a sweet spot.

Our Team

Muhammadreza Haghiri (Website - Github - LinkedIn)
Mahi Mohrechi (Website - Github - LinkedIn)

Special Thanks

Mistral Team for providing the best open source base model ever.
Sina Rashidi, who translated Alpaca dataset to Persian.
Jupyto team for providing our infrastructure.

RichardErkhov/MaralGPT_-_Maral-7B-alpha-1-gguf