You can deploy this model and start earning money today!

Naman0807/
gujarati-lm

Name: gujarati-lm
Availability: InStock

Views

No views yet

Gujarati Language Model (GPT-2 Small)

This is a compact Gujarati language model based on the GPT-2 architecture. It has been trained from scratch on a dataset of Gujarati headlines to demonstrate language modeling capabilities in low-resource settings.

Model Description

Architecture: GPT-2 (Decoder-only Transformer)
Model Size: ~52 Million Parameters
Context Window: 256 tokens
Vocabulary Size: 64,000
Tokenizer: SentencePiece (Unigram)
Training Framework: Hugging Face Transformers & Tokenizers

Intended Use

This model is intended for:

Text generation in Gujarati.
Educational purposes to understand LLM training from scratch.
Fine-tuning on specific Gujarati downstream tasks.

Note: Due to the small model size and limited training data (headlines), the generated text may not always be coherent for long paragraphs.

How to Use

You can use this model directly with the Hugging Face pipeline or AutoModel classes.

Using Pipeline

python

from transformers import pipeline

generator = pipeline("text-generation", model="Naman0807/gujarati-lm")
prompt = "સરકાર"
output = generator(prompt, max_length=50, num_return_sequences=1)

print(output[0]['generated_text'])

Using Model and Tokenizer

python

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Naman0807/gujarati-lm")
model = AutoModelForCausalLM.from_pretrained("Naman0807/gujarati-lm")

input_text = "ગુજરાત રાજ્ય"
input_ids = tokenizer.encode(input_text, return_tensors="pt")

output = model.generate(input_ids, max_length=50, num_return_sequences=1)
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)

print(generated_text)

Training Data

The model was trained on a dataset of Gujarati news headlines.

Source: Custom CSV dataset (train.csv, valid.csv)
Content: News headlines and short text snippets.
Preprocessing: Text was normalized using NFKC and tokenized using a custom SentencePiece tokenizer trained on the corpus.

Training Procedure

Hyperparameters

Epochs: 5
Batch Size: 8
Learning Rate: 5e-4
Weight Decay: 0.01
Optimizer: AdamW
Scheduler: Linear decay
Precision: Mixed Precision (FP16)

Model Configuration

python

GPT2Config(
    vocab_size=64000,
    n_positions=256,
    n_ctx=256,
    n_embd=512,
    n_layer=6,
    n_head=8
)

Limitations

Context Length: The model has a short context window of 256 tokens, limiting its ability to generate long, coherent articles.
Dataset Bias: Trained primarily on headlines, so the style is terse and news-oriented.
Size: As a "compact" model (6 layers, 512 hidden dim), it lacks the reasoning capabilities of larger LLMs.