
Model	pothana-base-300M
Architecture	LLaMA (RoPE + SwiGLU + RMSNorm + GQA)
Parameters	387M (unique)
Hidden size	1024
Layers	30 unique (60 effective via weight sharing)
Attention heads	16 Q / 4 KV (Grouped Query Attention)
Intermediate size	2816
Context length	2048
Vocab size	48,000
Tokenizer	SentencePiece Unigram (48K)
Training	Single GPU, bf16 mixed precision
Developed by	Dvitva AI

Quick Start

Using pipeline

python

from transformers import pipeline

pipe = pipeline("text-generation", model="dvitvaai/pothana-base-300M", trust_remote_code=True)
result = pipe("తెలుగు భాష", max_new_tokens=50, do_sample=True, temperature=0.8)
print(result[0]["generated_text"])

Note: trust_remote_code=True is required for the custom tokenizer that cleans up SentencePiece word boundary markers for readable output.

Manual loading

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("dvitvaai/pothana-base-300M")
tokenizer = AutoTokenizer.from_pretrained("dvitvaai/pothana-base-300M", trust_remote_code=True)

text = "తెలుగు భాష చాలా అందమైనది"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.8,
        top_k=50,
        do_sample=True,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Tokenizer

This model uses a SentencePiece Unigram tokenizer with a 48K vocabulary, trained directly on Telugu text.

Handles raw Telugu text directly (no preprocessing needed)
Byte-fallback for out-of-vocabulary characters
Split digits for better number handling
NFKC normalization

Architecture

Key features:

Grouped Query Attention (GQA): 16 query heads, 4 KV heads — 4x KV cache reduction
Block-wise Weight Sharing: 30 unique blocks, each used twice = 60 effective layers (MobileLLM-LS)
SwiGLU MLP with 2816 intermediate size
RoPE positional encoding (theta=10000.0)
RMSNorm (no bias in any linear layer)

Training

Data: Telugu text corpus (Sangraha dataset)
Preprocessing: SentencePiece tokenization (raw text)
Optimizer: AdamW (lr=3e-4, weight_decay=0.1, beta1=0.9, beta2=0.95)
Schedule: WSD (Warmup-Stable-Decay)
Precision: bf16 mixed precision
Hardware: Single NVIDIA B200 GPU

Limitations

This is a base model (not instruction-tuned) — it performs text completion, not instruction following
Trained primarily on Telugu text; limited multilingual capability
Small model size (387M) limits reasoning and knowledge capacity

License

Apache 2.0

Citation

If you use this model, please cite:

@misc{pothana-base-300M,
  title={Pothana Base 300M: A Telugu Language Model},
  author={Dvitva AI},
  year={2025},
  url={https://huggingface.co/dvitvaai/pothana-base-300M}
}

dvitvaai/pothana-sp-base-300M

Pothana Base 300M

Model Details

Quick Start

Using pipeline

Manual loading

Tokenizer

Architecture

Training

Limitations

License

Citation

dvitvaai/
pothana-sp-base-300M