CodeBharat-100M is a 100.68M parameter decoder-only Transformer pretrained from scratch on code (Python, JavaScript, TypeScript, C++, SQL, HTML/CSS, and synthetic textbooks).

Model Details

Architecture: Decoder-Only Transformer (Llama/Qwen-style: RMSNorm, RoPE, SwiGLU, Grouped-Query Attention)
Parameters: 100,679,424 (100.68M)
Vocabulary: 49,152 tokens (Byte-level BPE)
Context Window: 1,024 tokens
Training Device: NVIDIA GeForce RTX 5050 GPU
Final Validation Loss: 1.3891

Quickstart Usage

Native PyTorch / Tokenizers Usage

python

import torch
from tokenizers import Tokenizer
from pathlib import Path

# Load tokenizer and model weights
tokenizer = Tokenizer.from_file("tokenizer.json")
weights = torch.load("pytorch_model.bin", map_location="cuda" if torch.cuda.is_available() else "cpu")

Run Inference via CLI

python 100m-codebharat/scripts/11_generate.py --prompt "def binary_search(arr, target):"

Model Card Info

Developed by: CodeBharat Team
Model Type: Causal Language Model for Code
License: Apache 2.0

Ravi5528/codebharat-100m

CodeBharat-100M

Model Details

Quickstart Usage

Native PyTorch / Tokenizers Usage

Run Inference via CLI

Model Card Info

Ravi5528/
codebharat-100m