
Parameters	353,502,208
d_model	1024
Layers	24
Attention heads	16
KV heads	16 (standard MHA, no GQA)
Position encoding	RoPE
Normalization	RMSNorm, Pre-LN
Feed-forward	SwiGLU (8/3 hidden multiplier)
Context length	1024
Vocab size	50,304 (tiktoken GPT-2 BPE, padded)
Optimizer	AdamW
Tied embeddings	Yes

Training

Same base architecture as modern-llm-v1-base, further trained via supervised fine-tuning. [describe your SFT dataset/setup here — e.g. dataset name, size, format]

How to load

python

from huggingface_hub import snapshot_download
from safetensors.torch import load_file
import sys

local_dir = snapshot_download("JohnEnev/modern-llm-v1-sft")
sys.path.insert(0, local_dir)

from modeling.gpt import GPT, GPTConfig

# NOTE: GPTConfig() defaults are shaped for V2 — override explicitly for V1.
config = GPTConfig(
    vocab_size=50304, d_model=1024, n_layers=24, n_heads=16, n_kv_heads=16,
    max_seq_len=1024, use_flash=True, tie_weights=True,
    use_qk_norm=False, use_diff_attn=False, use_mhc=False, use_xsa=False,
)
model = GPT(config)

state_dict = load_file(f"{local_dir}/model.safetensors")
model.load_state_dict(state_dict, strict=False)  # lm_head re-tied below
model.lm_head.weight = model.token_embeddings.weight
model.eval()

How to sample from it

python

import torch, tiktoken

enc = tiktoken.get_encoding("gpt2")
input_ids = torch.tensor([enc.encode("The meaning of life is")])
output = model.generate(input_ids, max_new_tokens=50, temperature=0.8, top_k=50)
print(enc.decode(output[0].tolist()))

Related checkpoints

modern-llm-v1-base — the pretrained base model, for comparison
modern-llm-v1-grpo — this model, further trained with GRPO

Limitations

English only, 1024-token context. [any known SFT-specific issues]