You can deploy this model and start earning money today!

harsha-desaraju/
telugu-llama-small

Name: telugu-llama-small
Availability: InStock

harsha-desaraju/telugu-text

mit

text-generation-inference

endpoints_compatible

Views

No views yet

📌 Model Description

Telugu-Llama (64M) is a decoder-only transformer model based on the LLaMA architecture, trained from scratch on ~579M Telugu tokens. The model is designed to capture linguistic structure across a diverse mix of Telugu sources including classical literature, poetry, and modern content.

Architecture: LLaMA-style causal language model
Parameters: ~64M
Language: Telugu
Training Tokens: ~579 million
Training Epochs: 2
Final Eval Loss: 3.033

🧠 Model Architecture

python

LlamaConfig(
    vocab_size=16384,
    hidden_size=512,
    intermediate_size=2048,
    num_hidden_layers=12,
    num_attention_heads=8,
    num_key_value_heads=4,
    max_position_embeddings=2048,
    rms_norm_eps=1e-5,
    rope_theta=10000.0,
    hidden_act='silu'
)

🔤 Tokenizer

Type: Byte Pair Encoding (BPE)
Vocab Size: 16,384
Training Data: Same domain as model training corpus
Token Fertility: 1.9

This tokenizer is optimized for Telugu morphology and script efficiency.

📚 Training Data

The model was trained on a curated and diverse Telugu corpus:

Sources

ai4bharat-sangraha/verified/tel
Custom scraped datasets including:
- Telugu poetry (classical & modern)
- Ancient music literature
- Film songs and lyrics
- General web text

⚙️ Training Details

Training Configuration

python

TrainingArguments(
    num_train_epochs=2,
    per_device_train_batch_size=16,
    gradient_accumulation_steps=4,
    learning_rate=1e-4,
    weight_decay=0.1,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    max_grad_norm=1.0,
    fp16=True
)

📊 Evaluation

Metric: Cross-entropy loss
Final Eval Loss: 3.033
Perplexity: 20.76

Example

python

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("harsha-desaraju/telugu-llama-small")
model = AutoModelForCausalLM.from_pretrained("harsha-desaraju/telugu-llama-small")

inputs = tokenizer("ఎందరో మహానుభావులు అందరికి ", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))

⚠️ Limitations

Small model size (64M) limits:
- Long-range reasoning
- Factual accuracy
May produce:
- Hallucinations
- Repetitive text
Domain bias toward:
- Literary and cultural text
No instruction-following ability (just a text completion model)

⚖️ Bias & Risks

Training data includes scraped content → potential:
- Cultural bias
- Stylistic skew toward poetry/lyrics
May reflect biases present in Telugu web and literary corpora

Use cautiously in sensitive applications.