Qwen-Desi: Multilingual Indian Language Model

Model Description

Qwen-Desi is a multilingual language model fine-tuned from Sarvam-AI's Sarvam M model, specifically designed to support Indian languages and their transliterated variants. This model excels at understanding and generating text in English, Hindi, Kannada, and their English-transliterated forms (Hinglish and Kannadish).

Developed by: Anshuman Suresh and Bharatdeep Hazarika
Model type: Causal Language Model
Base model: unsloth/Qwen2.5-14B-Instruct-unsloth-bnb-4bit
Parent model: sarvamai/sarvam-m (24B Mistral)
Language(s): English, Hindi (Devanagari), Hinglish, Kannada, Kannadish
License: Apache-2.0

Supported Languages

English: Standard English language
Hindi: Written in Devanagari script (हिंदी)
Hinglish: Hindi words written in English script (transliterated)
Kannada: Written in Kannada script (ಕನ್ನಡ)
Kannadish: Kannada words written in English script (transliterated)

Training Details

This Qwen2 model was efficiently fine-tuned using:

Training Framework: Unsloth (2x faster training)
Library: Hugging Face TRL (Transformer Reinforcement Learning)
Base model: unsloth/Qwen2.5-14B-Instruct-unsloth-bnb-4bit
Parent model: sarvamai/sarvam-m
Language(s): English, Hindi (Devanagari), Hinglish, Kannada, Kannadish
Architecture: Qwen2-based transformer

Usage

Direct Usage with OpenAI Compatible Package

python

import json
from openai import OpenAI

client = OpenAI(
    base_url="http://<URL>/v1/",
    api_key="your-api-key"
)

messages = [
    {
        "role": "system", 
        "content": """
You are a helpful assistant. You support five languages: English, Hindi, Hinglish, Kannada and Kannadish. 
English is the standard English language. Hindi is written in Devanagari script. 
Hinglish refers to Hindi words written in English script (Hindi transliterated to English). 
Kannada is written in Kannada script. Kannadish refers to Kannada words written in English script 
(Kannada transliterated to English). Infer user's query and answer in Kannadish (English script).
"""
    },
    {
        "role": "user",
        "content": "Mujhe Lebron James ke baare mai info do"
    },
]

response = client.chat.completions.create(
    model="1shoomun/qwen2.5-14b-desi",
    messages=messages,
    temperature=0.3,
    max_tokens=3096,
    frequency_penalty=0,
    presence_penalty=1.05,
    top_p=0.2,
    seed=42,
    stream=True,
    stream_options={"include_usage": True},
)

for token in response:
    if hasattr(token, 'choices') and token.choices[0].delta.content:
        print(token.choices[0].delta.content, end='', flush=True)

Model Capabilities

Code-mixed conversations: Seamlessly handles conversations mixing English with Hindi/Kannada
Script flexibility: Understands both native scripts and transliterated text
Multi-turn dialogue: Maintains context across conversation turns
Language detection: Automatically infers the preferred response language

Recommended Parameters

python

# Recommended inference parameters
temperature = 0.3        # For more focused responses
max_tokens = 4096       # Adjust based on your needs
top_p = 0.2            # For controlled generation
frequency_penalty = 0   # Prevent repetition
presence_penalty = 1.05 # Encourage diverse responses

Limitations

Primary focus on Indian languages may limit performance on other languages
Performance may vary between formal and informal language styles
Transliterated text quality depends on consistency of transliteration schemes

Ethical Considerations

Trained to be helpful and respectful across all supported languages
Aims to preserve cultural nuances and appropriate language use
Users should be aware of potential biases in multilingual outputs

Acknowledgments

Built with Unsloth for efficient training
Based on Sarvam-AI's excellent work on Indian large language models
Powered by Hugging Face's TRL library