This is a large GPT-2-style causal LM trained from scratch on the Open Reason SFT split. It is larger than theworker02/open-reason-medium (13,867,008 parameters) and is not a 1B model and is not theworker02/open-reason-1b.

Weights live on this Hub repo. They are not stored in the GitHub git tree.

Training facts

Parameters: 91,544,064
Architecture: GPT-2-style from scratch; n_layer=12, n_embd=768, n_head=12, vocab_size=8192, max_seq_len=256
Steps: 400 (batch size 2)
Final training loss: 5.7361 (next-token NLL on training batches; not a benchmark score)
SFT rows: 3,175 from data/release/all.jsonl
Dataset: theworker02/open-reason pipeline 1.4.0
License: Apache-2.0
Hardware: AMD Ryzen 9 9950X host CPU; torch 2.12.0+cpu; torch.cuda.is_available()=False; Docker not installed and not used; nvidia-smi not present. NVIDIA CUDA was not used. AMD GPU / ROCm / DirectML were not used.

Dataset: https://huggingface.co/datasets/theworker02/open-reason
Small (~1.3M): https://huggingface.co/theworker02/open-reason-small
Medium (13,867,008): https://huggingface.co/theworker02/open-reason-medium
Large (this repo, 91,544,064): https://huggingface.co/theworker02/open-reason-large
GitHub: https://github.com/theworker02/open-reason

No Reddit sources.

python

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("theworker02/open-reason-large")
model = AutoModelForCausalLM.from_pretrained("theworker02/open-reason-large")

theworker02/open-reason-large

Open Reason large (CPU)

Training facts

Related

theworker02/
open-reason-large