This is a fine-tuned version of the
facebook/opt-1.3b model using the
LoRA (Low-Rank Adaptation) technique. The model has been trained on a dataset focused on Ayurveda and the concept of doshas (Vata, Pitta, Kapha). Compared to the previous model (
ai-ml-t-tes2-dftopcat-data-dsr1-1.5b), this version incorporates improved training parameters and adjustments to enhance coherence, reduce repetition, and minimize inaccuracies. However, challenges remain in achieving depth and specificity about Vata, Pitta, and Kapha doshas
.
Model Details
Model Description
This model is a fine-tuned adaptation of the facebook/opt-1.3b base model, optimized for generating explanations related to Ayurveda and doshas. It uses the LoRA technique to reduce computational costs while maintaining performance. The training data consists of instructional prompts and corresponding outputs that explain Ayurvedic concepts like doshic constitution, balance, and their influence on health.
Compared to the previous model (ai-ml-t-tes2-dftopcat-data-dsr1-1.5b), this version demonstrates:
Improved Coherence : Responses are more structured and logical.Reduced Repetition : Adjustments to generation parameters (e.g., repetition_penalty) have further minimized redundant phrases.Fewer Inaccuracies : Misinterpretations (e.g., "doshas as disease-causing elements") are less frequent but still present.
However, the model still struggles with depth and specificity, particularly in explaining Vata, Pitta, and Kapha doshas in detail. Additionally, it occasionally introduces unrelated content or formatting artifacts.
Developed by: tradexyModel type: Causal Language Model (Fine-Tuned)Language(s): EnglishLicense: MIT License Finetuned from model: facebook/opt-1.3b
Model Sources
Uses
Direct Use
The model can be used to generate responses to questions about Ayurveda, particularly focusing on doshas and their role in health. It is suitable for educational purposes, answering FAQs, or providing introductory insights into Ayurvedic principles.
Downstream Use
The model can be integrated into applications like chatbots, virtual assistants, or educational platforms that focus on alternative medicine and wellness.
Out-of-Scope Use
The model is not designed for medical diagnosis, treatment recommendations, or generating content outside the scope of Ayurveda. Misuse or reliance on the model for critical health decisions is strongly discouraged.
Bias, Risks, and Limitations
Known Limitations
While the model shows improvements over the previous version, it still occasionally generates repetitive or nonsensical phrases.
Responses lack depth and specificity about Vata, Pitta, and Kapha doshas compared to expert-level explanations.
The model sometimes introduces inaccuracies (e.g., misinterpreting doshas as "disease-causing elements") due to limitations in training data or fine-tuning.
Unrelated content or formatting artifacts occasionally appear in responses.
Improvements Over Previous Model
Increased Training Steps : Extended training steps (max_steps=1500) have allowed the model to better internalize nuances from the dataset.Adjusted LoRA Parameters : Higher values for r (32) and lora_alpha (64) provided greater capacity for adaptation.Smaller Learning Rate : A reduced learning rate (5e-5) ensured finer adjustments during training, reducing inaccuracies.Larger Validation Set : Increased validation set size (100 examples) improved evaluation metrics.Optimized Generation Parameters : Adjusted temperature, top_k, and repetition_penalty for better coherence and relevance.
Recommendations
Use post-processing techniques to filter out irrelevant or inaccurate statements.
Fine-tune the model further with more diverse and high-quality training data.
Experiment with even larger base models (e.g., facebook/opt-6.7b) for improved performance .
How to Get Started with the Model
To use this model, follow these steps:
1 from transformers import AutoTokenizer , AutoModelForCausalLM
2 from peft import PeftModel , PeftConfig
3 import torch
4
5 # Load the base model
6 base_model = AutoModelForCausalLM . from_pretrained (
7 "facebook/opt-1.3b" , # Original base model
8 torch_dtype = torch . float16 ,
9 device_map = "auto"
10 )
11
12 # Load the LoRA configuration and adapter
13 peft_config = PeftConfig . from_pretrained ( "tradexy/ai-ml-t-tes3-tradexy-data-dsr1-1.5b" )
14 model = PeftModel . from_pretrained ( base_model , "tradexy/ai-ml-t-tes3-tradexy-data-dsr1-1.5b" )
15
16 # Load the tokenizer
17 tokenizer = AutoTokenizer . from_pretrained ( "tradexy/ai-ml-t-tes3-tradexy-data-dsr1-1.5b" )
18 tokenizer . pad_token = tokenizer . eos_token
19
20 # Generate text
21 def generate_text ( prompt , max_new_tokens = 500 ) :
22 inputs = tokenizer ( prompt , return_tensors = "pt" ) . to ( 'cuda' )
23 with torch . no_grad ( ) :
24 output = model . generate (
25 ** inputs ,
26 max_new_tokens = max_new_tokens ,
27 do_sample = True ,
28 temperature = 0.3 , # Balanced creativity and coherence
29 top_k = 25 , # Focus on higher-probability tokens
30 top_p = 0.87 , # Consistent quality
31 repetition_penalty = 1.3 , # Prevents repetition without over-penalizing
32 use_cache = True
33 )
34 return tokenizer . decode ( output [ 0 ] , skip_special_tokens = True )
35
36 # Test the model
37 prompt = "Ayurveda emphasizes the balance between doshas. How can understanding our doshic constitution promote better health?"
38 output = generate_text ( prompt )
39 print ( output )
