You can use the model with a pipeline for a high-level helper or load the model directly. Here's how:
1# Use a pipeline as a high-level helper
2from transformers import pipeline
3pipe = pipeline("question-answering", model="asif00/mistral-bangla-4bit")
1# Load model directly
2from transformers import AutoTokenizer, AutoModelForCausalLM
3tokenizer = AutoTokenizer.from_pretrained("asif00/mistral-bangla-4bit")
4model = AutoModelForCausalLM.from_pretrained("asif00/mistral-bangla-4bit")
1prompt = """Below is an instruction in Bengali language that describes a task, paired with an input also in Bengali language that provides further context. Write a response in Bengali language that appropriately completes the request.
2
3### Instruction:
4{}
5
6### Input:
7{}
8
9### Response:
10{}
11"""
1def generate_response(question, context):
2 inputs = tokenizer([prompt.format(question, context, "")], return_tensors="pt").to("cuda")
3 outputs = model.generate(**inputs, max_new_tokens=1024, use_cache=True)
4 responses = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]
5 response_start = responses.find("### Response:") + len("### Response:")
6 response = responses[response_start:].strip()
7 return response
1question = "ভারতীয় বাঙালি কথাসাহিত্যিক মহাশ্বেতা দেবীর মৃত্যু কবে হয় ?"
2context = "২০১৬ সালের ২৩ জুলাই হৃদরোগে আক্রান্ত হয়ে মহাশ্বেতা দেবী কলকাতার বেল ভিউ ক্লিনিকে ভর্তি হন। সেই বছরই ২৮ জুলাই একাধিক অঙ্গ বিকল হয়ে তাঁর মৃত্যু ঘটে। তিনি মধুমেহ, সেপ্টিসেমিয়া ও মূত্র সংক্রমণ রোগেও ভুগছিলেন।"
3answer = generate_response(question, context)
4print(answer)
The asif00/mistral-bangla-4bit model has been trained on a limited dataset, and its responses may not always be perfect or accurate. The model's performance is dependent on the quality and quantity of the data it has been trained on. Given more resources, such as high-quality data and longer training time, the model's performance can be significantly improved.
Work in progress...
