Bharat MiniGPT 350M (3.5B tokens Experiment)
Bharat MiniGPT 350M is a custom GPT-style causal language model trained from scratch by Harshvardhan Mishra using modern LLM architecture components such as RoPE, RMSNorm, SwiGLU, and SDPA Attention.
This is not a fine-tuned GPT-2 or LLaMA variant. The architecture and training pipeline were implemented manually in PyTorch and later integrated into the HuggingFace ecosystem.
A better version with more tokens and fine tune version available SOON.
Best suited for:
Knowledge completion
Educational prompts
Article continuation
English prose generationThis is a pretrained foundation model and is not instruction tuned yet.
Model Details
Model Name: Bharat MiniGPT 350MParameters: ~350 MillionArchitecture: Decoder-only TransformerTraining Tokens: 3.5 BillionFramework: PyTorch + Custom Hugging Face Transformers integrationDeveloper: Harshvardhan MishraOrganization: HVM Smart Solutions
Architecture
Component Details Layers 24 Transformer Blocks Heads 16 Attention Heads Embedding Size 1024 Context Length 768 Tokens Vocabulary Size 50,257 Position Encoding RoPE (Rotary Position Embedding) Normalization RMSNorm Feed Forward SwiGLU Attention SDPA / Flash Attention Compatible Weight Tying Yes Precision FP16 Training
Training Data
The model was trained using a weighted mixture of:
Dataset Weight HuggingFaceFW/fineweb (sample-10BT) 40% HuggingFaceFW/fineweb-edu (sample-10BT) 30% Wikimedia Wikipedia (20231101.en) 30%
Training Setup
Setting Value Optimizer AdamW Learning Rate 3e-4 Min LR 3e-5 Warmup Steps 51,200 LR Scheduler Cosine Decay Gradient Accumulation 128 Mixed Precision FP16 Gradient Clipping 1.0
Features
Custom GPT architecture
RoPE positional embeddings
RMSNorm normalization
SwiGLU feed-forward layers
Flash Attention compatible SDPA
HuggingFace generate() support
KV-cache compatible
Weight tying support
Gradient checkpointing during training
Benchmark Results
Evaluated using:
EleutherAI LM Evaluation Harness
Tasks Version Filter n-shot Metric Value Stderr arc_easy 1 none 0 acc ↑ 0.3312 ± 0.0097 none 0 acc_norm ↑ 0.3413 ± 0.0097 hellaswag 1 none 0 acc ↑ 0.2650 ± 0.0044 none 0 acc_norm ↑ 0.2636 ± 0.0044 piqa 1 none 0 acc ↑ 0.5631 ± 0.0116 none 0 acc_norm ↑ 0.5533 ± 0.0116
Notes:
Results are from the current 3B tokens pretrained base checkpoint.
This model is not instruction-tuned yet.
Further tokenizer and training improvements are planned.
Installation
pip install transformers torchUsage
Load Model
from transformers import AutoTokenizer, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"Harshhvm/bharat-minigpt-350m-pretrain-3b-tokens",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
"Harshhvm/bharat-minigpt-350m-pretrain-3b-tokens",
trust_remote_code=True
)
Generate Text
import torch
prompt = "India is a land of"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=80,
temperature=0.45,
top_p=0.82,
top_k=40,
repetition_penalty=1.35,
no_repeat_ngram_size=4,
do_sample=True,
use_cache=True,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.eos_token_id,
)print(tokenizer.decode(output[0], skip_special_tokens=True))
