Mistral-Indic-Chat-LoRA-v1

Overview

Base Model

Training Details

• LoRA Rank (r): 8
• Target Modules: q_proj, v_proj
• Trainable Parameters: 0.2%
• Epochs: 2
• Batch Size: 2
• Max Sequence Length: 256

Dataset

• Bhasha Wiki Indic Context (English → Hindi, Hindi → English)
• Synthetic Hinglish conversational data

Data Format

Usage

import torch  
from transformers import AutoModelForCausalLM, AutoTokenizer  
from peft import PeftModel  

model_name = "mistralai/Mistral-7B-v0.1"  

tokenizer = AutoTokenizer.from_pretrained(model_name)  
tokenizer.pad_token = tokenizer.eos_token  

base_model = AutoModelForCausalLM.from_pretrained(model_name)  
model = PeftModel.from_pretrained(base_model, "YOUR_MODEL_PATH")  

model.eval()  

prompt = "User: gravity kya hoti hai?\nAssistant:"  
inputs = tokenizer(prompt, return_tensors="pt")  

with torch.no_grad():  
    outputs = model.generate(  
        **inputs,  
        max_new_tokens=50,  
        do_sample=True,  
        temperature=0.7,  
        top_p=0.9,  
        repetition_penalty=1.2,  
        eos_token_id=tokenizer.eos_token_id  
    )  

response = tokenizer.decode(outputs[0], skip_special_tokens=True)  
print(response.split("Assistant:")[-1].strip())  

Capabilities

• English → Hindi translation
• Hindi → English translation
• Hinglish conversational understanding
• Chat-style response generation
• Basic multilingual alignment

Limitations

• Weak reasoning ability
• Can hallucinate incorrect outputs
• Inconsistent for long responses
• Limited generalization
• Not production-ready

Evaluation

• Qualitative evaluation only
• No standard benchmarks used
• Improvements mainly in multilingual behavior

Architecture

• Base model is frozen
• Only LoRA adapters trained
• Efficient parameter usage (~0.2%)

Intended Use

• Learning and experimentation
• Multilingual fine-tuning research
• Indic chatbot prototyping

License