Item	Value
Developer	Moksh Bhardwaj
Base Model	unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit
Fine-tuning Method	LoRA (PEFT)
Framework	Unsloth + Transformers + TRL
Task	Conversational Fine-tuning
Language	Hindi, Hinglish, English
Domain	Vedic Astrology Assistant

Dataset

The model was fine-tuned on the conversation dataset provided as part of the Vedaz AI internship assessment.

Dataset characteristics:

55 multi-turn conversations
System/User/Assistant format
Hindi + Hinglish
Safety-focused responses
Kundli analysis workflow
Empathetic conversational style

The dataset includes conversations related to:

Career
Marriage
Relationships
Finance
Mental health safety
General astrology guidance

Training

Fine-tuning

LoRA Fine-tuning
Unsloth
PEFT
TRL SFTTrainer

Hyperparameters

Parameter	Value
Epochs	10
Batch Size	2
Gradient Accumulation	4
Effective Batch Size	8
Learning Method	Supervised Fine-tuning
Precision	bfloat16 / FP16 (hardware dependent)

Approximately 1.9% of the model parameters were trained using LoRA adapters.

Intended Use

Suitable for:

Astrology chatbot demos
Conversational AI experiments
Fine-tuning demonstrations
Internship assignments
Educational purposes

Limitations

Because the training dataset contains only 55 conversations, the model has several limitations:

May repeat phrases.
May overfit training patterns.
Does not possess real astrological reasoning.
Responses should not be considered professional astrological advice.
Should not be used for medical, legal, or financial decision making.

Safety

The model is designed to:

Avoid fatalistic predictions.
Avoid guarantees about the future.
Respond empathetically.
Request birth details before horoscope analysis.
Encourage practical action alongside astrology.

Example

python

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "moksh0987654/Vedaz-Qwen-Finetuned"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

messages = [
    {"role":"system","content":"You are Vedaz AI astrologer."},
    {"role":"user","content":"Meri job kab lagegi?"}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(text, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.7
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Repository

GitHub:

https://github.com/0001Moksh/Vedaz-Qwen-Finetuning

Author

Moksh Bhardwaj

AI & Machine Learning Engineer

GitHub: https://github.com/0001Moksh

LinkedIn: https://linkedin.com/in/moksh-bhardwaj-0001moksh

Portfolio: https://mokshbhardwaj.netlify.app

Framework Versions

Transformers
PEFT 0.19.1
TRL
Unsloth
Accelerate
BitsAndBytes