You can deploy this model and start earning money today!

akarki15/
bdh-nepali

Name: bdh-nepali
Availability: InStock

baby-dragon-hatchling

biologically-inspired

devanagari

2509.26507

mit

Views

No views yet

BDH Nepali — Baby Dragon Hatchling

A 25M parameter biologically-inspired language model trained on the NepBERTa Nepali news corpus. Generates Nepali text in Devanagari script using byte-level tokenization (no tokenizer needed — native UTF-8 support).

Try the live demo | GitHub

Architecture

Component	Value
Parameters	25M
Layers	6 (shared weights)
Embedding dim	256
Attention heads	4
Context length	128
Vocab size	256 (byte-level)

Key features:

Byte-level tokenization — works natively with UTF-8/Devanagari, no tokenizer needed
Shared weights across layers (iterative refinement)
Hebbian gating (multiplicative sparse activations)
ReLU sparsity for interpretable activations
RoPE position embeddings

Training

Dataset: NepBERTa corpus — clean Nepali news articles (~52MB)
Optimizer: AdamW (lr=1e-3, weight_decay=0.1)
Steps: 10,000
Hardware: Google Colab T4 GPU (~1 hour)
Val loss: 0.60 | Perplexity: 1.81

Usage

python

import torch

model_data = torch.hub.load_state_dict_from_url(
    "https://huggingface.co/akarki15/bdh-nepali/resolve/main/bdh_nepali.pt",
    map_location="cpu"
)
# Load with BDH model class — see GitHub repo for full inference code

Reference

Based on The Dragon Hatchling architecture by Pathway.