You can deploy this model and start earning money today!

oodgnas/
nanochat-pos5-k3

Name: nanochat-pos5-k3
Availability: InStock

Views

No views yet

d24_mid_pos5_k3

d24 MTP pos=5 k_start=3

Model Details

Field	Value
Architecture	Nanochat (custom transformer)
Parameters	~780M
Layers	24
Hidden dim	1536
Heads (Q/KV)	12/12
Vocab size	32768
Context length	2048
Window pattern	L
Val BPB	0.727460
MTP probe layers	[5]
MTP k_start	3

Architecture Notes

RoPE positional embeddings (no absolute pos embeds)
QK norm in attention
ReLU² activation in MLP
RMSNorm (no learnable parameters)
Logit softcap (tanh, ±20.0)
GQA (grouped-query attention)
Per-layer scalars (resid_lambdas, x0_lambdas)
Sliding window attention pattern: L
Untied token embedding and LM head

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("d24_mid_pos5_k3", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("d24_mid_pos5_k3", trust_remote_code=True)

inputs = tokenizer("Hello, world!", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))

oodgnas/nanochat-pos5-k3

d24_mid_pos5_k3

Model Details

Architecture Notes

Usage

oodgnas/
nanochat-pos5-k3