At 33.5M parameters this model learned the shape of a good answer — markdown structure, numbered steps, worked examples, mitigation sections — and is frequently wrong about the content.

Measured over 8 greedy prompts:

category	mean 8-gram repetition	stopped on EOS
security	18.7%	1 / 4
general	18.9%	1 / 2
code	19.9%	0 / 2
overall	19.0%	2 / 8

Do not use this as a factual security reference. Do not run code it generates. It is a demonstration of a correct end-to-end training pipeline and a baseline for scaling.

Where it works

Well-covered in-domain questions come out correct and terminate cleanly:

What is SQL injection and how do I prevent it?

SQL injection (SQLi) is a security vulnerability that allows attackers to manipulate database queries by injecting malicious SQL code through input fields. It occurs when user-supplied data is improperly sanitized, allowing attackers to manipulate the database. For example, if an attacker submits a username like admin' -- as the username, the query becomes: SELECT * FROM users WHERE username = '[input]' AND password = '[input]'.

What is a firewall?

A firewall is a network security device or software that monitors and controls incoming and outgoing network traffic based on predefined security rules. It acts as a barrier between trusted internal networks and untrusted external networks, such as the internet.

Where it fails

All observed, reproducible, and the majority case:

Wrong content, confident tone. Asked to contrast symmetric and asymmetric encryption it answered about hashing and IKE, never addressing the question.
Circular definitions. "A buffer overflow is a type of buffer overflow that could lead to arbitrary code execution."
Topically-adjacent but wrong vocabulary. Asked how to investigate a phishing email it produced SameSite and Strict — real security terms, wrong topic (they are cookie attributes).
Degenerate loops in code. port: The port to use repeated to the token limit.
Unreliable termination. Only 2 of 8 prompts stopped on EOS; the rest ran to the token limit.

These are consequences of scale, not of the training run — the loss curve is healthy and the pipeline is machine-verified (35 architecture checks, 173 tests).

Model details


parameters	33,531,264
architecture	12 layers, d_model 384, 6 heads, SwiGLU 1024, RoPE, RMSNorm, tied head
context	2048
vocab	32,000 (SentencePiece BPE)
base model	sabari2005/cyberslm-base
SFT data	23,540 conversations, 15.1M supervised tokens
epochs	3 (2,208 optimizer steps)
optimiser	AdamW, lr 2e-5, 3% warmup, cosine, bf16
best val loss	2.2627 (response tokens only)

Loss is computed on assistant responses only; prompts are masked. 88% of tokens in the SFT set are supervised.

Usage

bash

pip install torch sentencepiece
git clone https://huggingface.co/sabari2005/cyberslm-instruct
cd cyberslm-instruct
python infer_chat.py --prompt "What is SQL injection and how do I prevent it?"

Interactive:

python infer_chat.py --interactive

Options:

bash

python infer_chat.py \
    --prompt "What is a buffer overflow?" \
    --max-new-tokens 200 \
    --temperature 0.0      # 0 = greedy, recommended for this model

Prompt format

The model was trained on this exact layout, with a real BOS token id prepended and EOS terminating each response:

### User:
{question}

 ### Assistant:
{response}<eos>

Build prompts with the bundled formatter (infer_chat.py does this). Hand-assembling the string produces different token ids at every segment boundary, because SentencePiece prepends a word-boundary marker per encode() call — the model then sees something it was never trained on.

python

import torch
from configs.sft_config import default_config
from data.prompt_formatter import PromptFormatter, Tokenizer
from model.cyberslm import CyberSLM

cfg = default_config()
cfg.tokenizer.model_path = "tokenizer/tokenizer.model"
cfg.model.max_seq_len = cfg.data.max_seq_len = 2048

tok = Tokenizer(cfg.tokenizer.model_path)
fmt = PromptFormatter(cfg=cfg, tokenizer=tok)

model = CyberSLM(cfg.model)
model.load_state_dict(torch.load("models/instruct.pt", map_location="cpu",
                                 weights_only=False))
model.eval()

ids = fmt.format_for_inference({"messages": [{"role": "user",
                                              "content": "What is XSS?"}]})
out = model.generate(torch.tensor([ids]), max_new_tokens=200,
                     temperature=0.0, eos_id=tok.eos_id)
print(tok.decode(out[0, len(ids):].tolist()))

Decoding uses a KV cache — roughly 50–70 tok/s on CPU.

Intended use

Research into small language models; a scaling baseline; a demonstration of a verified training pipeline. Not for security advice, incident response, code generation, or anything where accuracy matters.

Training data

Not published. Curated cybersecurity instruction data; not redistributed.

License

Apache-2.0 for the code and weights. Verify licensing for downstream use against the sources the data was curated from.