This model is a continually pre-trained version of the meta-llama/Llama-3.2-1B architecture, fine-tuned on extensive Bangla datasets. The primary goal of the continual pretraining was to enhance the model's ability to generate high-quality Bangla text. By extending the pretraining process specifically on Bangla data, the model has demonstrated superior performance in Bangla language understanding evaluation benchmarks and text generation tasks.
Model Architecture: Llama 3.2 is an auto-regressive language model with optimized transformer architecture.
Training Data
Params
Input modalities
Output modalities
Context Length
GQA
Shared Embeddings
Token count
Knowledge cutoff
Llama 3.2 (text only)
Hishab curated Bangla text corpus
1B (1.23B)
Monolingual Text(Bangla)
Monolingual Text(Bangla)
4096
Yes
Yes
8.5B tokens
Supported Languages: Bengali (primary) and English (secondary)
Llama 3.2 Model Family: Token counts refer to pretraining data only. All model versions use Grouped-Query Attention (GQA) for improved inference scalability.
Model Release Date: October 24, 2024
Status: This is a static model trained on an offline dataset. Future versions may be released to improve model capabilities.
License: We are using a similar license to Llama 3.2. Use of Llama 3.2 is governed by the Llama 3.2 Community License (a custom, commercial license agreement).
Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.
Make sure to update your transformers installation via pip install --upgrade transformers.
Training Factors: We used llama-factory training library, Cloud GPU cluster, and production infrastructure for pretraining. Fine-tuning, annotation, and evaluation were also performed on cloud infrastructure.
Training Data
Overview: We have collected a large Bangla raw dataset of text data from various sources. Our collected data so far includes a mix of web documents, books, translated text, transliterated text, transcribe text, code-mixed text, conversations, and open sources raw data. The dataset is cleaned and filtered by different filtering criteria to ensure the quality of the data. Our collected data size is roughly around 268 GB. We separated 33GB data from that using a ratio of the data actual data size. Total trained tokens are 8.5B tokens.
Data sources summary:
Web documents: Extracted, clean, and filtered common crawl data
Books: Extracted, clean, filtered book data
Transcribed text: Used in-house Bangla ASR model to transcribe Bangla audio data
Translation data: We trained an English-Bangla translation LLM model and used it to translate English data to Bangla
Code-mixed data: We trained an English-Bangla code-mixed LLM model and used it to generate code-mixed data
Transliteration data: We trained a Bangla-English transliteration LLM model and used it to generate transliterated data
Synthetic data: We generated synthetic data using a Bangla LLM model
Others: We scrapped some selected website data, used open-source data, and used some other data sources
Benchmarks
In this section, we report the results for titulm-llama-3.2-1b-v1.1 models on standard automatic benchmarks. For all these evaluations, we used lm-evaluation-harness evaluations library.
Evaluation Datasets
We evaluated our pre-trained models on both Bangla and English benchmark datasets. Although the model is trained on Bangla data, its English capability is also evaluated on English benchmark datasets. The evaluation datasets are as follows:
Bangla Benchmark datasets
We evaluated the models on the following datasets:
Bangla MMLU: A private multiple choice question dataset developed by Hishab curated from various sources.
CommonsenseQa Bangla: A Bangla translation of the CommonsenseQA dataset. The dataset was translated using a new method called Expressive Semantic Translation (EST), which combines Google Machine Translation with LLM-based rewriting modifications.
OpenbookQA Bangla: A Bangla translation of the OpenbookQA dataset. The dataset was translated using a new method called Expressive Semantic Translation (EST), which combines Google Machine Translation with LLM-based rewriting modifications.
Piqa Bangla: A Bangla translation of the Piqa dataset. The dataset was translated using a new method called Expressive Semantic Translation (EST), which combines Google Machine Translation with LLM-based rewriting modifications.
BoolQ Bangla: The dataset contains 15,942 examples, with each entry consisting of a triplet: (question, passage, answer). The questions are naturally occurring, generated from unprompted and unconstrained settings. Input passages were sourced from Bangla Wikipedia, Banglapedia, and News Articles, and GPT-4 was used to generate corresponding yes/no questions with answers.
English Benchmark datasets
MMLU: This is a massive multitask test consisting of multiple-choice questions from various branches of knowledge.
CommonseQa: CommonsenseQA is a new multiple-choice question-answering dataset that requires different types of commonsense knowledge to predict the correct answers.
OpenbookQA: OpenBookQA aims to promote research in advanced question-answering, probing a deeper understanding of both the topic (with salient facts summarized as an open book, also provided with the dataset) and the language it is expressed in.
Piqa: The PIQA dataset focuses on physical commonsense reasoning, challenging AI to handle everyday situations requiring practical knowledge and unconventional solutions. Inspired by instructables.com, it aims to enhance AI's ability to understand and reason about physical interactions.
BoolQ: BoolQ is a question-answer dataset for yes/no questions containing 15942 examples. These questions are naturally occurring. They are generated in unprompted and unconstrained settings. Each example is a triplet of (question, passage, answer), with the title of the page as optional additional context. The text-pair classification setup is similar to existing natural language inference tasks.
Evaluation Results
Evaluation of Bangla Benchmark datasets
llama-3.2-1b performs better in Bangla MMLU, BoolQ BN, and OpenBook QA BN in the 0-shot setting, achieving top scores of 0.29, 0.55, and 0.33 respectively.
hishab/titulm-llama-3.2-1b-v1.1 outperforms in Commonsense QA BN and PIQA BN in both 0-shot and 5-shot settings, with the highest 5-shot scores of 0.31 and 0.57.
Model
Shots
Bangla MMLU
BoolQ BN
Commonsense QA BN
OpenBook QA BN
PIQA BN
llama-3.2-1b
0-shot
0.29
0.55
0.22
0.33
0.53
5-shot
0.28
-
0.23
0.31
0.54
hishab/titulm-llama-3.2-1b-v1.1
0-shot
0.28
0.54
0.28
0.31
0.56
5-shot
0.28
-
0.31
0.34
0.57
Evaluation of English Benchmark datasets
llama-3.2-1b dominates across all tasks, achieving the highest scores in MMLU, BoolQ, Commonsense QA, OpenBook QA, and PIQA in both 0-shot and 5-shot settings, with a 5-shot PIQA score of 0.759.
hishab/titulm-llama-3.2-1b-v1.1 shows competitive performance, particularly in Commonsense QA in the 0-shot setting but generally falls behind llama-3.2-1b in most tasks.
Model
Shots
MMLU
BoolQ
Commonsense QA
OpenBook QA
PIQA
llama-3.2-1b
0-shot
0.38
0.64
0.47
0.37
0.75
5-shot
0.309
0.662
0.317
0.396
0.759
titulm-llama-3.2-1b-v1.1
0-shot
0.26
0.62
0.34
0.35
0.73
5-shot
0.26
0.62
0.25
0.39
0.74
Instruction Tuned Models
Intended Use
Bangla text generation
Bangla language understanding tasks
Bangla instruction fine-tuning tasks
Citation
@misc{nahin2025titullmsfamilybanglallms,
title={TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking},
author={Shahriar Kabir Nahin and Rabindra Nath Nandi and Sagor Sarker and Quazi Sarwar Muhtaseem and Md Kowsher and Apu Chandraw Shill and Md Ibrahim and Mehadi Hasan Menon and Tareq Al Muntasir and Firoj Alam},
year={2025},
eprint={2502.11187},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.11187},
}
