You can deploy this model and start earning money today!

RichardErkhov/
Haary_-_TinyLlama-1.1B-indo-v1-gguf

Name: Haary_-_TinyLlama-1.1B-indo-v1-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

TinyLlama-1.1B-indo-v1 - GGUF

Model creator: https://huggingface.co/Haary/
Original model: https://huggingface.co/Haary/TinyLlama-1.1B-indo-v1/

Name	Quant method	Size
TinyLlama-1.1B-indo-v1.Q2_K.gguf	Q2_K	0.4GB
TinyLlama-1.1B-indo-v1.IQ3_XS.gguf	IQ3_XS	0.44GB
TinyLlama-1.1B-indo-v1.IQ3_S.gguf	IQ3_S	0.47GB
TinyLlama-1.1B-indo-v1.Q3_K_S.gguf	Q3_K_S	0.47GB
TinyLlama-1.1B-indo-v1.IQ3_M.gguf	IQ3_M	0.48GB
TinyLlama-1.1B-indo-v1.Q3_K.gguf	Q3_K	0.51GB
TinyLlama-1.1B-indo-v1.Q3_K_M.gguf	Q3_K_M	0.51GB
TinyLlama-1.1B-indo-v1.Q3_K_L.gguf	Q3_K_L	0.55GB
TinyLlama-1.1B-indo-v1.IQ4_XS.gguf	IQ4_XS	0.57GB
TinyLlama-1.1B-indo-v1.Q4_0.gguf	Q4_0	0.59GB
TinyLlama-1.1B-indo-v1.IQ4_NL.gguf	IQ4_NL	0.6GB
TinyLlama-1.1B-indo-v1.Q4_K_S.gguf	Q4_K_S	0.6GB
TinyLlama-1.1B-indo-v1.Q4_K.gguf	Q4_K	0.62GB
TinyLlama-1.1B-indo-v1.Q4_K_M.gguf	Q4_K_M	0.62GB
TinyLlama-1.1B-indo-v1.Q4_1.gguf	Q4_1	0.65GB
TinyLlama-1.1B-indo-v1.Q5_0.gguf	Q5_0	0.71GB
TinyLlama-1.1B-indo-v1.Q5_K_S.gguf	Q5_K_S	0.71GB
TinyLlama-1.1B-indo-v1.Q5_K.gguf	Q5_K	0.73GB
TinyLlama-1.1B-indo-v1.Q5_K_M.gguf	Q5_K_M	0.73GB
TinyLlama-1.1B-indo-v1.Q5_1.gguf	Q5_1	0.77GB
TinyLlama-1.1B-indo-v1.Q6_K.gguf	Q6_K	0.84GB
TinyLlama-1.1B-indo-v1.Q8_0.gguf	Q8_0	1.09GB

Original model description:

license: llama2

TinyLlama-1.1B-indo-v1 (Contoh Card Model)

Base Model TinyLlama/TinyLlama-1.1B-Chat-v1.0

Python code example: inference from this GPTQ model

Install the necessary packages

Requires: Transformers 4.33.0 or later, Optimum 1.12.0 or later, and AutoGPTQ 0.4.2 or later.

shell

pip3 install --upgrade transformers optimum
# If using PyTorch 2.1 + CUDA 12.x:
pip3 install --upgrade auto-gptq
# or, if using PyTorch 2.1 + CUDA 11.x:
pip3 install --upgrade auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/

If you are using PyTorch 2.0, you will need to install AutoGPTQ from source. Likewise if you have problems with the pre-built wheels, you should try building from source:

shell

pip3 uninstall -y auto-gptq
git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
git checkout v0.5.1
pip3 install .

Example Python code

python

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model_name_or_path = "daptheHuman/Merak-7B-v4-GPTQ"
# To use a different branch, change revision
# For example: revision="gptq-4bit-32g-actorder_True"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                             device_map="auto",
                                             trust_remote_code=False,
                                             revision="main")
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
prompt = "Tell me about AI"
prompt_template=f'''### Instruction:
{prompt}
### Response:
'''
print("\n\n*** Generate:")
input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, temperature=0.7, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512)
print(tokenizer.decode(output[0]))
# Inference can also be done using transformers' pipeline
print("*** Pipeline:")
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.95,
    top_k=40,
    repetition_penalty=1.1
)
print(pipe(prompt_template)[0]['generated_text'])

Credits

TinyLlama/TinyLlama-1.1B-Chat-v1.0 for README template.

image source

pixabay.com

RichardErkhov/Haary_-_TinyLlama-1.1B-indo-v1-gguf