You can deploy this model and start earning money today!

akarki15/
bdh-shakespeare

Name: bdh-shakespeare
Availability: InStock

baby-dragon-hatchling

biologically-inspired

2509.26507

mit

Views

No views yet

BDH Shakespeare — Baby Dragon Hatchling

A 25M parameter biologically-inspired language model trained on the Tiny Shakespeare dataset. Generates Shakespearean-style English text using byte-level tokenization.

Try the live demo | GitHub

Architecture

BDH (Baby Dragon Hatchling) is a compact, biologically-inspired transformer:

Component	Value
Parameters	25M
Layers	6 (shared weights)
Embedding dim	256
Attention heads	4
Context length	128
Vocab size	256 (byte-level)

Key features:

Shared weights across layers (iterative refinement)
Byte-level tokenization (no external tokenizer needed)
Hebbian gating (multiplicative sparse activations)
ReLU sparsity for interpretable activations
RoPE position embeddings

Training

Dataset: Tiny Shakespeare (~1MB)
Optimizer: AdamW (lr=1e-3, weight_decay=0.1)
Steps: 5,000
Hardware: Google Colab T4 GPU (~30 min)
Val loss: 1.57 | Perplexity: 4.81

Usage

python

import torch

model_data = torch.hub.load_state_dict_from_url(
    "https://huggingface.co/akarki15/bdh-shakespeare/resolve/main/bdh_shakespeare.pt",
    map_location="cpu"
)
# Load with BDH model class — see GitHub repo for full inference code

Reference

Based on The Dragon Hatchling architecture by Pathway.