from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base  = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    torch_dtype=torch.float16, device_map="auto"
)
tok   = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
model = PeftModel.from_pretrained(base, "gauravgarwal/NyayaGPT-Mistral7B-adapter")
model.eval()

messages = [
    {"role": "system", "content": "You are NyayaGPT, an expert Indian legal assistant."},
    {"role": "user",   "content": "What is Section 302 IPC?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids    = tok(prompt, return_tensors="pt").to(model.device)
out    = model.generate(**ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))

Training Details

Attribute	Value
Base model	mistralai/Mistral-7B-Instruct-v0.3
Method	QLoRA (LoRA r=16, α=32)
Training samples	1,521
Eval samples	169
Epochs	2
Learning rate	0.0002
Batch size	2 × 4 grad accum = 8 effective
Hardware	NVIDIA RTX 5090 (32 GB)
Dataset	IndianKanoon case law + GPT-4o synthetic

Evaluation

Metric	Score
ROUGE-1	0.5683
ROUGE-L	0.4028
RAGAS Faithfulness	0.6573

Quantization Benchmark

Variant	Memory (GB)	Latency (ms/tok)	ROUGE-L
FP16	14.5	10.8	0.3668
INT8	7.7	6.8	0.3711
INT4 (GGUF)	4.4	4.9	0.3706

Domain

Indian law: IPC, CrPC, Constitution, Contract Act, NI Act, IT Act, Civil Procedure Code, property law, PIL, Supreme Court / High Court judgments.

Disclaimer

This model is for educational and research purposes only. It does not constitute legal advice. Always consult a qualified advocate for legal matters.