You can deploy this model and start earning money today!

ctxnn1/
gpt2-124m-fineweb-edu-10b

Name: gpt2-124m-fineweb-edu-10b
Availability: InStock

text-generation-inference

endpoints_compatible

Views

No views yet

GPT-2 124M FineWeb-Edu 10B

This is a GPT-2 124M base text-completion model trained from scratch on the FineWeb-Edu sample-10BT dataset. It is not instruction-tuned and is not a chatbot.

Try it live

Run this model in a free public Gradio Space:

Open the live demo →

The Space runs on CPU-basic (free) hardware and accepts a prompt with temperature, top-k, top-p, repetition penalty, and seed controls.

Model description

Architecture: GPT-2 decoder-only Transformer
Parameters: 124,439,808 after export-vocabulary trimming
Layers / heads / hidden size: 12 / 12 / 768
Context length: 1,024 tokens
Export vocabulary: 50,257 GPT-2 tokens
Training tokens: 9,999,745,024
Hardware: one NVIDIA H100

The native trainer padded its embedding/output matrix from 50,257 to 50,304 rows for efficient kernels. The final 47 rows were never tokenizer-addressable. This export keeps rows 0–50,256, sets config.vocab_size=50257, and preserves tied input/output embeddings, so generation cannot emit a padded ID.

Final evaluation

Metric	Value
Training step	19,073
Train loss	3.103327
Validation loss	3.030832
Validation perplexity	20.714451
HellaSwag accuracy	30.0339% (3,016/10,042)

Intended use

The model is intended for research, education, reproducibility studies, and experiments with small pretrained language models. It performs ordinary next-token completion.

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ctxnn1/gpt2-124m-fineweb-edu-10b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
inputs = tokenizer("The future of artificial intelligence is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64, do_sample=True, top_k=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations, risks, and biases

This is a 124M-parameter base model and is not competitive with modern large language models.
It is not instruction-tuned, preference-aligned, safety-tuned, or suitable as a chatbot.
Outputs can be inaccurate, incoherent, biased, offensive, unsafe, or memorized from pretraining data.
HellaSwag accuracy is only modestly above the 25% random-choice baseline.
The run did not include comprehensive safety, fairness, memorization, or downstream-task evaluation.
Users must evaluate outputs and suitability for their own domain before deployment.

Training and conversion provenance

Source code: https://github.com/ctxnn/gpt-2
W&B: https://wandb.ai/ctxnn-thapar-university/gpt2-from-scratch/runs/65e78f54c14046ef99e04e12e7b3e810
Native checkpoint SHA-256: e519d993d20c98c841ef061f76a1dec3e6ee24d5e55162bdea2a3e2da280fd40
Training Git SHA: 36bfc9edd044eb828e118d49c79532eef8440a2a
Cloud execution Git SHA: 9862792ab4024f9ebec758be73ebe7e75419d09b

The native Linear weights for attention and MLP projections were transposed into Hugging Face GPT-2 Conv1D orientation. Positional embeddings, LayerNorm parameters, attention/MLP projections, and tied token embeddings were preserved and validated with native-versus-Hugging-Face logit and loss comparisons.