import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = "togethercomputer/Mistral-7B-Instruct-v0.2"
adapter = "Reubencf/mistral-7b-instruct-konkani-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    torch_dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)
model.eval()

messages = [
    {"role": "user", "content": "तुजें नांव किदें?"},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

with torch.no_grad():
    output = model.generate(
        inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )

print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))

Merge the adapter (optional)

To fold the LoRA weights into the base model for faster inference:

python

merged = model.merge_and_unload()
merged.save_pretrained("mistral-7b-konkani-merged")
tokenizer.save_pretrained("mistral-7b-konkani-merged")

Prompt format

Uses the Mistral instruct chat template:

<s>[INST] your message [/INST]

tokenizer.apply_chat_template(...) (shown above) applies this for you.

Adapter details


Base model	togethercomputer/Mistral-7B-Instruct-v0.2
Type	LoRA (PEFT)
Rank (r)	64
Alpha	128
Dropout	0.0
Target modules	q_proj, k_proj, v_proj, o_proj
Task	CAUSAL_LM

konkani/mistral-7b-instruct-konkani-lora

Mistral-7B-Instruct Konkani LoRA

How to run

Install

Load and generate

Merge the adapter (optional)

Prompt format

Adapter details

konkani/
mistral-7b-instruct-konkani-lora