You can deploy this model and start earning money today!

Elriggs/
gpt2-bilinear-swiglu-12l-6h-768embd

Name: gpt2-bilinear-swiglu-12l-6h-768embd
Availability: InStock

pytorch

Views

No views yet

Modded NanoGPT Model

This is a GPT-2 style model trained with modifications from modded-nanogpt.

Model Config

Layers: 12
Heads: 6
Embedding dimension: 768
Vocab size: 50304
Squared MLP: False
Bilinear: True
Gated: True
Expansion factor: 4

Training

Training step: 6484

Usage

python

from huggingface_hub import hf_hub_download
import torch
from train_gpt2 import GPT, GPTConfig
import json

# Download config
config_path = hf_hub_download(repo_id="Elriggs/gpt2-bilinear-swiglu-12l-6h-768embd", filename="config.json")
with open(config_path) as f:
    config_dict = json.load(f)

# Remove non-GPTConfig fields
config_dict.pop('step', None)

# Create model
config = GPTConfig(**config_dict)
model = GPT(config)

# Download and load weights
weights_path = hf_hub_download(repo_id="Elriggs/gpt2-bilinear-swiglu-12l-6h-768embd", filename="pytorch_model.bin")
state_dict = torch.load(weights_path, map_location='cpu')
model.load_state_dict(state_dict)

model.eval()