SoloLLM-1.0-123M-Base is a from-scratch decoder-only base language model from the SoloLLM project. It is trained for text completion, not instruction following or chat.

Smaller-than-GPT-2 SoloLLM 1.0 base checkpoint; wins the fixed multiple-choice suite and most external checks, but does not beat GPT-2 small across every metric.

Results

Metric	SoloLLM-1.0-123M-Base	GPT-2 small
Parameters	123,551,232	124,439,808
Train tokens	9,800,572,928	public
Project held-out PPL	25.56	25.32
WikiText-2 PPL	43.75	45.32
LAMBADA PPL	35.45	40.62
LAMBADA last-token acc	46.30%	46.67%
LAMBADA last-word acc	31.69%	32.60%
MC avg normalized acc	43.13%	41.05%

Model Details

Item	Value
Architecture	Decoder-only GPT-style transformer
Parameters	123,551,232
Context length	1024
Tokenizer	GPT-2 tokenizer
Embedding width	768
Layers	12
Attention heads	12
Positional method	RoPE
Normalization	RMSNorm
MLP	SwiGLU
Dropout	0.0
Weight tying	Input/output embeddings tied
Training hardware	Single RTX 3090

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "bmax16634/solollm-1.0-123m-base"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)

Original Format

The original/ folder contains the raw PyTorch state dict and sanitized training metadata from the SoloLLM training run. The root-level model.safetensors file is the preferred Hugging Face/Transformers format.

Limitations

This is a base model. It is not instruction-tuned, RLHF-tuned, safety-tuned, or optimized for chat behavior.