A 303M-parameter, LLaMA-architecture language model trained from scratch on a home server (2× NVIDIA Tesla P100). This is the base (pretrained) model — it has not been instruction-tuned, so it completes text rather than following chat instructions.

👉 For the chat/assistant version, see TinyBrainBot 303M Instruct.

Model details


Parameters	~303M
Architecture	LLaMA-style decoder (RoPE, RMSNorm, SwiGLU, pre-norm)
Layers	24
Hidden size	1024
Attention heads	16 (GQA, 4 KV heads)
Head dim	64
FFN size	2816
Vocab	32,000 (custom SentencePiece BPE)
Context length	1024
Tied embeddings	Yes
Precision	fp16

It re-expresses cleanly as a LlamaForCausalLM, so it loads with standard 🤗 Transformers.

Training

Pretrained on a mix of open English text plus synthetic distillation data:

Web / educational text: FineWeb-Edu, English Wikipedia, OpenWebText2
Narrative: TinyStories
Math/reasoning: Orca-Math
Synthetic fact distillation: Q&A and short-fact data generated by a stronger teacher model (Ling 2.6 Flash) to densify factual coverage.

Total training tokens are on the order of a few billion — far fewer than models like SmolLM/Pythia (hundreds of billions). This is a hobbyist/research model; capability scales with tokens, and this one is intentionally token-limited. (4.7B)

Intended use

A starting point for continued pretraining or fine-tuning at small scale.
Text completion and research on small language models.
Not for chat/instruction following — use the Instruct version for that.

Limitations

Shallow, fragile factual knowledge. It knows common facts but is sensitive to phrasing and capitalization (e.g. "France" vs "france" can flip the answer).
Weak at reasoning and math; roughly random on knowledge benchmarks like MMLU (typical for this size).
Will produce fluent plausible-but-wrong text confidently. Pair with retrieval (RAG) for anything factual.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("your-username/tinybrainbot-303m-base")
model = AutoModelForCausalLM.from_pretrained("your-username/tinybrainbot-303m-base", torch_dtype=torch.float16)

ids = tok("The water cycle is the process by which", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, temperature=0.6, top_p=0.9, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))

License

Apache-2.0 (change if you prefer a different license).

nkthebass/TinyBrainBot-303m-base

TinyBrainBot 303M — Base