⚠️ Pretraining is still in progress. This repository currently holds a mid-training checkpoint (step 174,000); it will be updated with the final weights as soon as training completes. The -base suffix denotes "no instruction tuning" — an -it variant will follow once pretraining is finished.
Eleven sequence-mixing mechanisms across 121 layers, each appearing exactly once per row and per column. Colour marks the family: attention (blue), linear (teal), state-space/Mamba2 (orange), convolutional/Hyena (purple) — the non-attention families are what make this heterogeneous rather than a variant sweep.
Latin square layer map
A 121-layer network that places 11 different sequence-mixing mechanisms on an 11x11 Latin square.
To our knowledge, this is the first model to integrate 11 heterogeneous sequence-mixing mechanisms — attention, state-space, convolutional and linear families — inside a single network. Existing hybrids combine two or three (e.g. Mamba + Transformer).
Release scope: open weights. This repository ships the weights plus the minimum code needed to load them. Unlike Aether-7B-5Attn — which is fully open (data recipe, training code, logs, checkpoints) — the training data recipe, training code, logs and intermediate checkpoints for this model are not released.
The 11 mechanisms
121 layers = 11 x 11. Each mechanism occupies exactly 11 layers, one per row and column of the Latin square, so no mechanism concentrates at any depth.
#
Mechanism
Family
Distinctive parameters (verifiable in the weights)
0
nsa
sparse attention
compress_mlp, gate_logit, select_score
1
differential
attention
lambda_q1/k1/q2/k2, subln
2
full
attention
q/k/v/o_proj
3
mla
latent attention
kv_a_proj, kv_b_proj, q_a_proj, q_b_proj
4
sliding
attention
windowed masking over the full path
5
compress
attention
full_attn.* submodule
6
hybrid
attention
nsa.* + diff.* + gate + merge_norm
7
linear
linear attention
gate, norm
8
mamba2
state-space
A_log, D, conv1d, dt_proj, B_proj, C_proj
9
gdn
gated delta net
conv1d_q, conv1d_k, gate_proj
10
hyena
convolutional
filter_dense (implicit filter)
Precision matters here: full and sliding share the same parameters (sliding is masking over the full path), so the weights show 10 distinct parameter signatures for 11 labels. And several of these are not "attention" in the strict sense — mamba2, hyena, gdn and linear are other sequence-mixing families, which is exactly the point of the experiment.
Specifications
Parameters
5.79B total (MoE)
Layers
121 (11 x 11 Latin square)
Hidden size
1024
Attention heads
8 query / 2 key-value (GQA), head dim 128
Experts
25 routed (top-7) + 1 shared
Vocabulary
151,936
Precision
bfloat16
Checkpoint
training step 174,000
License
Apache-2.0
Provenance and honest caveats
This model was reconstructed from a sharded FSDP training checkpoint. The exact training code was not archived, so the architecture code shipped here was rebuilt and then validated two ways:
Exact parameter match — all 11,047 checkpoint tensors load with 0 missing / 0 unexpected keys, and the parameter count matches the checkpoint exactly (5.7885B).
Perplexity — the loaded model predicts real text sensibly (PPL 11.8 English / 6.5 Korean on sample sentences). A mis-reconstructed architecture would produce perplexity orders of magnitude worse.
Treat this as a research artifact, not a polished product: it is a mid-training checkpoint of an architecture experiment, with no instruction tuning.
trust_remote_code=True is required — this is a custom architecture (aether_v2_11attn)
and the modeling code ships in this repository.
Run at batch_size = 1 (the NSA branches do not consume a padding mask), and run the forward under torch.autocast(dtype=torch.bfloat16) — several branches create fp32 intermediates.
Greedy decoding degenerates into repetition (expected for an un-tuned base checkpoint) — use sampling with a repetition penalty. Generation is ~10 s/token: 121 layers with no KV cache. This is a research artifact, not a serving build.
Related
Fully-open sibling:Aether-7B-5Attn (weights + data recipe + training code + logs + checkpoints)
