Project Indus
Project Indus LLM is a groundbreaking open-source language model tailored for Hindi and its dialects, designed to enhance natural language processing and generation across diverse Indian linguistic applications.
Table of Contents
Model Details
Model Description
Project Indus LLM aims to provide a robust language model for Indian languages, starting with Hindi and its dialects. This open-source foundational model, hosted on Hugging Face, is tailored for easy integration and further development by researchers and developers focusing on Indian linguistic diversity.
The model is a pretrained model in Hindi and dialects which is instruct tuned.
- Developed by: Nikhil Malhotra, Nilesh Brahme, Satish Mishra, Vinay Sharma (Makers Lab, TechMahindra)
- Model type: Foundational Language model
- Language(s) (NLP): hin, bho, mai, doi
- License: other
- Parent Model: It is a grounds up model built on GPT-2 architecture starting from tokenizer to decoder
- Resources for more information: https://www.techmahindra.com/en-in/innovation/the-indus-project/
Uses
Uses include question and answeting and conversation in Hindi and Dialects. The model would be reward tuned to be used across various industries
- Call center
- Healthcare
- Automotive
- Telecom
Direct Use
Project Indus can be directly used for generating text, simulating conversation, and other text generation tasks without additional training.
Downstream Use
Uses include question and answeting and conversation in Hindi and Dialects. The model would be reward tuned to be used across various industries
- Call center
- Healthcare
- Automotive
- Telecom
Out-of-Scope Use
Project Indus is not designed for high-stakes decision-making tasks such as medical diagnosis or legal advice, nor can it be used for fill-in-the-blank exercises, multiple Q&A, and similar applications at the moment.
Bias, Risks, and Limitations
Significant research has explored bias and fairness issues with language models
(see, e.g.,
Sheng et al. (2021) and
Bender et al. (2021)).
Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
We have taken care across various biases by trying to remove them from training data. However since the model is a generative model, it would tend to produce hallucinations.
Any disturbing or harmful sterotype produced by the model is purely un-intentional and coincidental.
Recommendations
It is recommended to avoid biases and negative connotations in the model, and regular updates along with community feedback are crucial for addressing any emergent bias or misuse scenarios.
Training Details
The model was trained on a curated dataset comprising various sources of Hindi text, including literature, news articles, and web content.
Infrastructure
- Training Infrastructure: Utilized high-performance computing resources provided by CDAC, featuring NVIDIA A100 GPUs.
- Running Infrastructure: Tested for both GPU (NVIDIA GeForce RTX 3070 or higher) and CPU (Intel Xeon Platinum 8580) environments.
Training Data
The Project Indus LLM was trained on a diverse and extensive dataset comprising various sources of Hindi text and its dialects. The data collection and curation process was meticulously designed to cater to the linguistic diversity and complexity of Indian languages, particularly focusing on Hindi and its 37 dialects.
Data Sources and Collection
Data was collected in three main buckets:
-
Open-Source Hindi Data: This included publicly available sources from the internet across different categories such as news, and non-news. Automated scripts were used to scrape and extract text from web pages. Here are some of the sources:
- News: Articles from major Hindi news portals like Amar Ujala, Bhaskar, India TV, Jagran, Live Hindustan, and Patrika.
- Non-News: Diverse sources including Wikipedia, commoncrawl.org, and other culturally significant content like 'Man ki Baat' from AIR.
-
Translated Data: A portion of the Pile dataset, which is a large English dataset used for training AI models, was translated into Hindi using three different translation models. IndicTrans2 (AI4Bharat) was selected as the best model for this purpose based on its accuracy and efficiency.
-
Dialects: Data collection for dialects presented a unique challenge due to the limited material available on the internet. Data for major dialects like Maithili, Bhojpuri, Magahi, and Braj Bhasha was collected from multiple sources, including fieldwork where representatives collected old books and other texts, which were then digitized and converted into text data.
Training Procedure
Training involved extensive preprocessing to clean and standardize the text, followed by supervised learning on a high-performance computing setup.
- Pre-training: Conducted on a dataset of 22 billion tokens using advanced tokenization techniques.
- Fine-Tuning: Supervised fine-tuning performed with a focus on Indian languages, utilizing datasets specifically tailored for cultural, political, and social contexts.
Below is a table summarizing the datasets used for pre-training and fine-tuning the model:
| Phase | Data Source | Tokens | Notes |
|---|
| Pre-training | Cleaned dataset of Hindi and dialects | 22 billion | Utilized advanced tokenization |
| Fine-tuning | Custom datasets tailored for Indian languages | Varied | Focus on cultural, political, and social contexts |
- Training Infrastructure: Utilized high-performance computing resources provided by CDAC, featuring NVIDIA A100 GPUs.
- Running Infrastructure: Tested for both GPU (NVIDIA GeForce RTX 3070 or higher) and CPU (Intel Xeon Platinum 8580) environments.
Preprocessing
The collected data underwent several stages of cleaning and preprocessing to ensure high quality and usability for training:
- Cleaning: The data was cleaned of unwanted text, characters, and personal information like mobile numbers. Transliteration was performed where necessary, and unwanted tags from scraped web pages were removed.
- Bias Removal: A Bias Removal Toolkit was developed to detect and remove biased language from the training data. This toolkit helped in ensuring that the text used for training the model was ethical, correct, and socially responsible.
- Tokenization: The data was tokenized using a custom tokenizer developed specifically for Hindi and its dialects. This tokenizer was based on Byte Pair Encoding (BPE) with additional mechanisms like byte fallback to handle the peculiarities of Hindi script efficiently.
Summary
The final dataset used for training consisted of:
- Raw Data Size: Over 500 GB of raw data collected.
- Cleaned and Curated Data: Approximately 200 GB of clean Hindi and dialect text data.
- Tokenization: Utilized 22 billion tokens created from the cleaned data for pre-training.
This diverse and extensive training data foundation allowed Project Indus LLM to develop robust capabilities for understanding and generating Hindi text, making it a powerful tool for applications requiring Indian language processing.
Evaluation
Indic LLM Leaderboard Results
Project Indus LLM has been evaluated using the Indic LLM Leaderboard, which employs the indic_eval evaluation framework specifically designed for assessing models on Indian language tasks. This framework provides a comprehensive view of model performance across a variety of benchmarks tailored to Indian languages.
Detailed results from the Indic LLM Leaderboard (α), accessible at
Hugging Face Indic LLM Leaderboard, are shown below:
| Task | Version | Metric | Value | | Stderr |
|---|
| All | | acc | 0.2891 | ± | 0.0109 |
| | acc_norm | 0.3013 | ± | 0.0112 |
| indiceval:ARC-Challenge:hindi:10 | 0 | acc | 0.2167 | ± | 0.0120 |
| | acc_norm | 0.2474 | ± | 0.0126 |
| indiceval:ARC-Easy:hindi:5 | 0 | acc | 0.3615 | ± | 0.0099 |
| | acc_norm | 0.3552 | ± | 0.0098 |
These results highlight the model's capabilities in understanding and generating Hindi language text under controlled testing conditions. The standard error values indicate the variance observed during the evaluation, providing insights into the consistency of the model's performance across different evaluation runs.
Open LLM Leaderboard Evaluation Results
Additionally, Project Indus LLM has been evaluated on the Open LLM Leaderboard, which provides another layer of benchmarking by comparing the model's performance against other state-of-the-art language models. Below are the summarized results from the Open LLM Leaderboard:
| Metric | Value |
|---|
| Avg. | 20.07 |
| AI2 Reasoning Challenge (25-Shot) | 22.70 |
| HellaSwag (10-Shot) | 25.04 |
| MMLU (5-Shot) | 23.12 |
| Winogrande (5-shot) | 49.57 |
These benchmark results can be explored further on
Hugging Face Open LLM Leaderboard.
Evaluation Context
The evaluation metrics acc (accuracy) and acc_norm (normalized accuracy) are used to quantify the model's performance. The tasks are differentiated by their difficulty and the specific dataset used, such as the ARC Challenge and ARC Easy sets, both adapted to Hindi language conditions to ensure relevant assessment. This structured evaluation ensures that the Indus LLM not only performs well in generalized text generation tasks but also in more specialized, context-specific scenarios pertinent to the Indian linguistic framework.
Results
Project Indus demonstrates competitive performance, particularly in text generation tasks, as evidenced by its scores on standardized benchmarks.
Technical Specifications
Model Architecture and Objective
Project Indus LLM is based on a GPT-2.0-like architecture, tailored to handle the complexities of the Hindi language and its dialects. This model was designed to serve as a foundational model that can be fine-tuned for various applications, making it highly versatile and adaptable to different domains within the Indian context.
- Architecture Details:
- Layers: 22 transformer layers, which provide a deep neural network capable of understanding complex language patterns.
- Heads: 32 attention heads per layer, facilitating a broad attention mechanism across different parts of the input data.
- Embedding Size: 2048, which allows the model to represent a wide variety of information and nuances in the data.
- Vocabulary Size: 32,300, tailored to include a comprehensive set of Hindi words and common phrases found in the training data.
The objective of this model is to provide a robust tool for text generation and understanding in Hindi and its dialects, supporting the development of applications that require natural language processing in these languages. It also aims to bridge the gap in technology where Indian languages are underrepresented, providing a platform for further linguistic research and technological inclusion.
Compute Infrastructure
Hardware
The pre-training and fine-tuning of Project Indus LLM were conducted on high-performance computing infrastructure provided by the Centre for Development of Advanced Computing (CDAC). This setup included:
- Nodes and GPUs: Utilization of six nodes, each equipped with eight NVIDIA A100 GPUs. These GPUs are state-of-the-art for machine learning tasks and provide the necessary computational power to handle the large volumes of data and complex model architectures.
- Memory and Storage: Each node was equipped with ample memory and storage to handle the datasets and model parameters efficiently. Specific configurations included 40 GB of GPU memory per card, essential for training large models.
Inference performance was tested on GPU as well as CPU.
- GPU: On GPU NVIDIA GeForce RTX 3070 we have seen for 250-350 tokens inference time around ~5-10s.
- CPU: On Intel CPU Xeon(R) Platinum 8580 we have seen performance comparable to GPU with throughput of > 30 token/second.
Software
The software environment was crucial for efficiently training and running the model. Key components included:
- Operating System: Linux, chosen for its stability and support for high-performance computing tasks.
- Machine Learning Frameworks: PyTorch, used for its flexibility and efficiency in training deep learning models. It supports extensive parallel processing and GPU acceleration, which are critical for training large models like Project Indus LLM.
- Job Scheduler: SLURM (Simple Linux Utility for Resource Management) was used to manage and allocate resources effectively across the distributed system. This ensured optimal scheduling of training jobs without resource contention.
Citation
The detailed citation information will help in acknowledging the work and efforts of the team behind Project Indus LLM when it is used or referenced in academic or professional settings.
1@article{malhotra2024projectindus,
2 title={Project Indus: A Foundational Model for Indian Languages},
3 author={Malhotra, Nikhil and Brahme, Nilesh and Mishra, Satish and Sharma, Vinay},
4 journal={Tech Mahindra Makers Lab},
5 year={2024},
6 url={https://www.techmahindra.com/en-in/innovation/the-indus-project/}
7}
APA:
Malhotra, N., Brahme, N., Mishra, S., & Sharma, V. (2024). Project Indus: A Foundational Model for Indian Languages.
Tech Mahindra Makers Lab. Available at
https://www.techmahindra.com/en-in/innovation/the-indus-project/
Glossary
This glossary section explains key terms used throughout the model documentation and technical details, helping users unfamiliar with certain concepts to better understand the content.
- Transformer Layers: Part of a neural network architecture that uses self-attention mechanisms to process sequential data such as text. Essential for NLP tasks.
- Attention Heads: Sub-units of a model layer that allow the model to focus on different parts of the input sequence when making predictions.
- Embedding Size: The size of the vector used to represent each token or word in a dense numerical form. Larger embeddings can capture more detailed information.
- Block Size: The maximum length of the input tokens the model can process in one operation.
- Vocabulary Size: The total number of unique words or tokens that the model can understand and generate.
More Information
For further details on Project Indus LLM, including additional documentation, tutorials, and community discussions, visit the following resources:
- Project Repository: Hugging Face Repository
- Tech Mahindra Makers Lab: Insights into the research and development behind Project Indus can be found on the Tech Mahindra Innovation page.
- Community Forums: Engage with the community on Hugging Face Forums for support, brainstorming, and sharing of new ideas related to Project Indus.
Authors
The model card and documentation for Project Indus LLM were collaboratively authored by:
- Nikhil Malhotra: Chief Innovation Officer at Tech Mahindra.
- Nilesh Brahme: Senior AI Research Scientist and one of the primary contributors to the Project Indus development.
- Satish Mishra: AI Architect, whose insights have significantly shaped the model's capabilities.
- Vinay Sharma: LLM Engineer focused on the linguistic data processing and model training aspects of Project Indus.
Model Card Contact
For inquiries, support, or further information regarding Project Indus LLM, please reach out through the following channels:
- Email: projectindus@techmahindra.com - For direct queries and professional engagements.
- GitHub Issues: For technical issues, feature requests, or contributions, please use the Issues section of the Project Indus GitHub repository.
- Hugging Face Spaces: Questions and discussions related to model implementation and community projects can be posted in our dedicated space on Hugging Face.
How to Get Started with the Model
To begin using Project Indus LLM for your projects, follow these steps to set up and run the model:
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model = AutoModelForCausalLM.from_pretrained("nickmalhotra/ProjectIndus")
4tokenizer = AutoTokenizer.from_pretrained("nickmalhotra/ProjectIndus")
5
6# Example inference
7def format_template(user_prompt):
8 messages = [
9 {"role": "user", "content": user_prompt},
10 ]
11 response = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
12 return response
13
14user_prompt = """भारत के वर्तमान प्रधानमंत्री कौन हैं?"""
15
16input_ids = format_template(user_prompt)
17
18# Generate text using the model
19output = model.generate(input_ids,
20 eos_token_id=tokenizer.eos_token_id,
21 pad_token_id=tokenizer.eos_token_id,
22 max_length=1024,
23 num_beams=5,
24 do_sample=True,
25 early_stopping=True,
26 temperature=0.7,
27 top_k=50,
28 top_p=0.95,
29 repetition_penalty=1.2,
30 no_repeat_ngram_size=3,
31 num_return_sequences=1,
32 )
33print(tokenizer.decode(output[0], skip_special_tokens=False))
Disclaimer
Model Limitations
Project Indus LLM is trained with single instruction tuning, which may result in hallucinations—instances where the model generates plausible but inaccurate information. Users should exercise caution, especially in scenarios requiring high factual accuracy.
Adaptation for Specific Use Cases
Project Indus LLM is designed as a foundational model suitable for further development and fine-tuning. Users are encouraged to adapt and refine the model to meet specific requirements of their applications.
Recommendations for Fine-Tuning
- Identify Specific Needs: Clearly define the requirements of your use case to guide the fine-tuning process.
- Curate Targeted Data: Ensure the training data is relevant and of high quality to improve model performance.
- Continuous Evaluation: Regularly assess the model's performance during and after fine-tuning to maintain accuracy and reduce biases.
This disclaimer aims to provide users with a clear understanding of the model's capabilities and limitations, facilitating its effective application and development.
