All data was obtained ethically and in compliance with the site's terms and conditions. No copyright texts are used in the training of this model without the permission.

GPT-J 6B was trained on the Pile, a large-scale curated dataset created by EleutherAI.
Frankenstein; or, The Modern Prometheus, Mary Shelley, 1818 (Public domain)

Training procedure

This model was trained for 402 billion tokens over 383,500 steps on TPU v3-256 pod. It was trained as an autoregressive language model, using cross-entropy loss to maximize the likelihood of predicting the next token correctly.

How to use

This model can be easily loaded using the AutoModelForCausalLM functionality:

python

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("obake2ai/MaryGPT")
model = AutoModelForCausalLM.from_pretrained("obake2ai/MaryGPT")

Developed by

MaryGPT

Yuma Kishi

GPT-J

Ben Wang, developer of GPT-J
James Bradbury for valuable assistance with debugging JAX issues.
Stella Biderman, Eric Hallahan, Kurumuz, and Finetune for converting the model to be compatible with the transformers package.
Leo Gao for running zero shot evaluations for the baseline models for the table.
Laurence Golding for adding some features to the web demo.
Aran Komatsuzaki for advice with experiment design and writing the blog posts.
Janko Prester for creating the web demo frontend.