TituLM-1B-ENBN-V1 is a large language model specifically trained for generating and understanding English and Bangla text. Utilizing a decoder-style transformer architecture, this model has been extensively trained on a dataset comprising 43.19 billion Bangla, English and codes tokens. This model is the part of iterative train and release Bilingual LLM from Hishab.
The training process was managed using the robust framework provided by MosaicML's
llm-foundry repository. Throughout the training phase, titulm-1b-bn-v1 underwent a total of 59 iterations, allowing for iterative refinements and optimization.
Notable training configs:
Datasets comprise Bangla, English, and Codes data. We mixed Bangla data with English Redpajama (C4, Github, StackExchange, Book, Arxiv, Wikipedia) data.
Token-wise distribution will be added soon below.
The basic use cases to generate text using this model are simple. Follow the below code to generate text using this model.
1pip install transformers
2pip install einops
3pip install accelerate
1import transformers
2from transformers import pipeline
3
4model_name = 'hishab/titulm-1b-enbn-v1'
5
6config = transformers.AutoConfig.from_pretrained(model_name, trust_remote_code=True)
7config.max_seq_len = 2048
8
9model = transformers.AutoModelForCausalLM.from_pretrained(
10 model_name,
11 config=config,
12 trust_remote_code=True
13)
14
15tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
16
17pipe = pipeline('text-generation', model=model, tokenizer=tokenizer, device='cuda:0')
18# for Bangla
19bn_output = pipe('আমি বাংলায় গান',
20 max_new_tokens=100,
21 do_sample=True,
22 use_cache=True)
23
24print(bn_output)
25# for English
26en_output = pipe('Bangla language plays',
27 max_new_tokens=100,
28 do_sample=True,
29 use_cache=True)
30
31print(en_output)
1@misc{hishab_2024_titulm_1b_enbn_v1,
2 author = {Hishab Technologies Ltd.},
3 title = {TituLM-1B-ENBN-V1},
4 year = {2024},
5 publisher = {HuggingFace Models},
6 howpublished = {https://huggingface.co/hishab/titulm-1b-enbn-v1},
7}
