A 361.8M-parameter decoder-only transformer, trained from scratch on ~30B tokens on 2× RTX 4090. Part of TIM (Transformer Inference Model) — an end-to-end, from-scratch LLM built to be legible and honestly evaluated.

Exported as Qwen3ForCausalLM — it loads with transformers and no trust_remote_code.

Architecture

SmolLM2-360M skeleton with modern choices: pre-LN RMSNorm, RoPE (θ=100000), SwiGLU, grouped-query attention (15 query / 5 KV heads) with per-head QK-norm, tied embeddings, no biases. 32 layers, hidden 960, head_dim 64, FFN 2560, vocab 49,152 (byte-level BPE). Context: max_position_embeddings 4096, trained at seq_len 2048.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("timothywong731/tim-360m-base")
m = AutoModelForCausalLM.from_pretrained("timothywong731/tim-360m-base", torch_dtype=torch.bfloat16)
ids = tok("The capital of France is", return_tensors="pt")
print(tok.decode(m.generate(**ids, max_new_tokens=20)[0]))

Training

~30B tokens, mix: 70% web (FineWeb-Edu sample-100BT + DCLM-Baseline), 15% Stack-Edu (code), 10% FineMath-3+, 5% misc. Muon + AdamW, WSD (warmup–stable–decay) schedule, bf16, 2× RTX 4090 (DDP, no NVLink).

Evaluation

lm-evaluation-harness (0.4.12), 0-shot, acc_norm where applicable:

HellaSwag	ARC-Easy	ARC-Challenge	PIQA	WinoGrande	OpenBookQA	SciQ	SocialIQA	CommonsenseQA	WikiText (bpb ↓)
0.388	0.515	0.259	0.658	0.525	0.294	0.722	0.378	0.196	1.058

Healthy for the scale: at ~30B tokens this is ~130× less training data than SmolLM2-360M (4T). Near-parity on early-saturating tasks (PIQA, SciQ), a real gap on token-hungry ones (HellaSwag, ARC) — as the token budget predicts.

Limitations

Small (360M) and lightly trained (~30B tokens). Not instruction-tuned — this is the base model. Weak at arithmetic and multi-step reasoning; will hallucinate facts; English-only; no safety tuning — do not deploy to end users without your own filtering.

Provenance & license

Apache-2.0. Trained only on named, public datasets: FineWeb-Edu (ODC-BY), DCLM-Baseline (CC-BY-4.0), Stack-Edu (derived from The Stack v2), FineMath (ODC-BY). No gated or scraped-private data. Attribution and removal requests: open an issue on the source repository.