A 89.8M parameter causal language model built entirely from scratch using a custom transformer architecture, trained on WikiText-103 + synthetic instruction data.

Try it Online

Test the model directly in your browser (free, no login required):

Try Chat

GitHub

Architecture

Parameter	Value
Parameters	89.8M
Layers	10
Hidden dim	768
Attention heads	12
KV heads (GQA)	4
Max sequence length	512
Vocab size	50,304
FFN activation	SwiGLU
Position encoding	RoPE (θ=500000)
Norm	RMSNorm
Training	Float32, full precision

Key design choices:

Grouped Query Attention (GQA): 12 query heads, 4 KV heads for efficient inference
SwiGLU FFN: Gated activation for better training dynamics
RoPE: Rotary Position Embeddings for length extrapolation
Tied embeddings: Input/output embeddings share weights

Installation

Requirements: Python 3.8+ and pip.

bash

# 1. Clone the repository
git clone https://huggingface.co/JustScriptzz/nexus-smAll-v1
cd nexus-smAll-v1

# 2. Install dependencies
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install tokenizers

GPU users: Replace --index-url https://download.pytorch.org/whl/cpu with the appropriate CUDA version, e.g. --index-url https://download.pytorch.org/whl/cu124 for CUDA 12.4.

Usage

Command line (quick start)

python chat.py --weights weights/nexus_instruct.pt

Python API

python

from src.model import Nexus
from src.config import NexusConfig
from tokenizers import Tokenizer
import torch

config = NexusConfig()
model = Nexus(config)

checkpoint = torch.load("weights/nexus_instruct.pt", map_location="cpu", weights_only=False)
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

tokenizer = Tokenizer.from_file("data/tokenizer.json")

prompt = "User: What is Python?\nAssistant:"
encoded = tokenizer.encode(prompt)
tokens = torch.tensor([encoded.ids])

output, _ = model.generate(tokens, max_new_tokens=64, temperature=0.2, top_k=40, top_p=0.9)
reply = tokenizer.decode(output)
print(reply)

Training

Phase 1: 100k steps on WikiText-103 (next-token prediction, ~212k sequences)
Phase 2: 5k steps on instruction data (Dolly-15k + synthetic QA, ~1500 examples)

Limitations

90M parameters is very small by modern standards — outputs may be incoherent
Not instruction-tuned on diverse enough data
Best used as a learning experiment, not production

Nexus Plus v2

Looking for a larger, more capable model? Check out Nexus Plus v2 — a Qwen3-4B fine-tune with QLoRA, trained on the same instruction dataset.

Plus v2

License

Apache 2.0