Qana-mini-5M is a 4.94M-parameter base causal language model that
studies content-dependent mixing across the expanded channels of a SwiGLU
feed-forward block. It was trained as a compact generalist model and serves as a
controlled architecture-research checkpoint. The final checkpoint was recorded
at step 40,000 with a WikiText-103 normalized BPB of 1.4241.
Quick start
The model uses a custom Transformers architecture, so
trust_remote_code=True is required.
The checkpoint is stored in native bfloat16 Safetensors. The exported model does
not implement a KV cache, so autoregressive generation recomputes the visible
context at each step.
Architecture
Component
Configuration
Parameters
4,943,712
Hidden width / layers
216 / 10
Context length
1,024 tokens
Vocabulary
4,096 tokens
Token attention
6 query heads, 2 KV heads
SwiGLU expansion
2x, producing 432 channels
Channel layout
18 chunks of 24 channels
Channel-mixing heads
3
Position encoding
Contiguous-half RoPE
Weight format
BF16 Safetensors
The expanded SwiGLU representation is divided into channel chunks that act as
queries, keys, and values for a small dense mixing graph. Each 24-channel chunk
is split across three heads, and each head computes an 18x18 content-dependent
mixing matrix.
The centered operator is
V_mix = [I + s(A(x) - A0)]V. The mixed representation then passes through the
block's single SwiGLU down projection. Shared diagonal Q/K scales keep the mixer
lightweight, while the reference distribution provides an identity-preserving
initialization.
What is different?
Token-to-token attention remains a standard causal grouped-query attention
layer. The architectural change is inside SwiGLU: after expansion, each token's
features are divided into small channel chunks and content-dependent attention
mixes information between those chunks. The centered update
V_mix = [I + s(A(x) - A0)]V preserves the ordinary SwiGLU representation at
its reference state while allowing learned cross-channel interactions. The
mixed representation then passes through the block's single down projection,
contracting it back to model width before it rejoins the residual stream.
Tokenizer
The model uses the 4,096-token tokenizer from
AxiomicLabs/GPT-S-5M at revision
275b9c3ca78736bf6aeb154c7e2d5f5764fe9035. The exported checkpoint preserves
its vocabulary, tokenization pipeline, and native special tokens. Standalone
prompts receive the tokenizer's native BOS prefix.
Training
The model was trained for 40,000 updates on 20.97B tokens, with an effective
batch of 524,288 tokens per update and a sequence length of 1,024.
Source
Share
FineWeb-Edu 100BT
55%
Cosmopedia v2
25%
FineWeb-HQ
10%
FineMath 4+
10%
Training used native BF16 compute, AdamW for embeddings and scalar parameters,
Muon for hidden matrices, a 1,000-update warmup, a flat learning-rate phase
through update 30,000, and cosine decay to zero at update 40,000.
Evaluation
The following results characterize the released checkpoint across language
modeling, commonsense completion, reading comprehension, and science question
answering. Results are reported task by task without composite aggregation.
All tasks were evaluated zero-shot with lm-eval 0.4.12 at model revision
fa2cd38b2888846e350bf533dc7bd21fbcfc344f. Model weights were BF16,
likelihood softmax was FP32, the evaluation batch size was 8, a native BOS token
was added, and the maximum context was 1,024 tokens. Inputs exceeding the model
context, encountered in BoolQ, were left-truncated by the harness. Standard
errors are reported directly from lm-eval; n/a means the task does not define
that metric.
Task
acc
acc stderr
acc_norm
acc_norm stderr
Perplexity
PPL stderr
HellaSwag
26.85%
0.44%
27.60%
0.45%
n/a
n/a
ARC-Easy
36.03%
0.99%
34.97%
0.98%
n/a
n/a
ARC-Challenge
16.21%
1.08%
23.21%
1.23%
n/a
n/a
PIQA
58.11%
1.15%
57.18%
1.15%
n/a
n/a
LAMBADA OpenAI
17.87%
0.53%
n/a
n/a
229.3926
10.0397
BoolQ
54.86%
0.87%
n/a
n/a
n/a
n/a
WinoGrande
52.64%
1.40%
n/a
n/a
n/a
n/a
OpenBookQA
14.80%
1.59%
26.80%
1.98%
n/a
n/a
SciQ
68.90%
1.46%
60.50%
1.55%
n/a
n/a
SWAG
31.67%
0.33%
36.92%
0.34%
n/a
n/a
BLiMP
70.13%
0.15%
n/a
n/a
n/a
n/a
ArithMark-3
n/a
n/a
29.90%
n/a
n/a
n/a
BLiMP was evaluated over all 67 official subtasks with automatic batch-size
selection; the table reports its group accuracy. ArithMark-3 was evaluated
separately using its official acc_norm metric. A standard error was not
recorded for that ArithMark-3 run and is therefore left as n/a.
WikiText-103 validation produced loss 3.1497, perplexity 23.33, and normalized
BPB 1.4241 over 359,037 scored tokens using 1,024-token windows with a 512-token
stride.
Intended use and limitations
Qana-mini-5M is intended for architecture research, representation
analysis, educational experiments, and controlled comparisons. It is a compact
base model rather than an instruction-tuned assistant. Its scale and context
length limit factual reliability, generation quality, and long-context
capability. Outputs may contain inaccuracies or biases inherited from the
training data and should not be used for consequential decisions.
Credits
Qana-mini-5M was led, developed, trained, evaluated, and released by
User01110. Additional authorship and
project credit belong to Paul Courneya and Jonathon LY. The project is
part of the FromZiro research community.
The complete release is covered by the
Qana Open Attribution License 1.0,
including the weights, architecture, implementation, configuration, tokenizer
artifacts, metadata, and documentation.
No permission or fee is required for private testing, research, evaluation,
use, modification, fine-tuning, implementation, or commercial use.
Redistribution, derivative models, and public products or services are also
allowed, but must preserve the license and provide visible credit to
Qana-mini-5M and its authors: User01110, Paul Courneya, and
Jonathon LY with a repository link where practical.
Users are solely responsible for their use, outputs, compliance, and any
misuse. The authors and FromZiro provide no warranty, accept no liability,
and do not endorse downstream applications.
Third-party components, datasets and tokenizer materials remain subject to
their respective licenses.
