⚠️ Use batch_size=1. Some attention branches (NSA family) do not consume a padding mask, so batching padded sequences can silently corrupt results. Run inference one sequence at a time (batch_size=1).
🇰🇷 The first Korean foundation model to open its training data and training code
A 6.59B MoE language model that arranges its attention across 49 layers on a 7×7 Latin square.
Weights · training data recipe · training code · complete training logs · full architecture source — all released.
Trained from scratch on 16 × NVIDIA B200 (2-node FSDP) over ~46 days, 144.2B tokens total.
Apache-2.0
1. Why release everything
Most "open" LLMs today give you weights only. Weights alone let you neither understand a model, nor verify it, nor rebuild it. A model whose diet is secret is a black box, and you cannot build national or institutional AI sovereignty on a black box.
The Allen Institute's OLMo reset the bar for what "open" means — that only by releasing the training data, the training code, and the logs alongside the weights does a model become science. Aether follows that bar.
What we release
Item
Aether
1
Weights (annealed base)
✅
2
Full architecture source — 5 attention types, MoE, Latin-square placement
✅ aether_pkg/
3
Training data recipe — source repos, configs, token counts, mixing weights, tokenizer, EOS
✅ §5
4
Tokenization script — the exact file we ran
✅ tokenize_one.py
5
Training code — FSDP launcher, training loop, node scripts
✅ launcher_v2b_multi.py et al.
6
Every training hyperparameter
✅ §6
7
Complete training log — all 162,000 steps, 30 days
✅ logs/
8
Evaluation code
✅ eval/lmeval_run.py
9
Intermediate checkpoints (110k · 115k · 162k)
✅
10
License
✅ Apache-2.0
The recipe in §5 alone is enough to reconstruct our training data byte-for-byte. Every source is a public repository, so identical tokenizer + EOS + mixing weights produce identical binaries. Please verify it. That is what it is for.
A first for Korean foundation models
Korea has built foundation models from scratch before. None of them released their training data. To our knowledge, Aether is the first.
This is what sovereign AI actually means. Any country, any company, any lab should be able to rebuild its own foundation model from scratch using nothing but this repository. Downloading someone else's weights is not sovereignty.
Across six sovereign fully-open LLMs, disclosure is comparable; Aether is the only single AI startup, with the most attention types (5) in a Latin-square layout.
2. Design — why should every layer use the same attention?
2.1 Nothing says one attention must serve all layers
Since GPT, nearly every transformer uses the same attention at every layer. A 49-layer model runs the same operation 49 times.
But layers do different jobs. Early layers pick up local morphology and grammar; middle layers handle syntactic structure; late layers work across the whole document. If the job is different, why must the operation be identical?
This is a question about a design assumption, not about performance. "Every layer, same attention" is not a proven conclusion — it is a convention that was never seriously questioned. Aether relaxes it.
2.2 Each attention carries a different inductive bias
Type
What it is good at
What it costs
full
Sees every token pair exactly. No information lost
Quadratic in length
sliding
Sees locality very cheaply (linear in length)
Blind beyond the window
differential
Subtracts two attention maps to cancel common-mode noise
Splits the head dimension in half
nsa
Gates compressed, selected and sliding branches to reach far cheaply
Structurally complex
hybrid
Combines nsa's reach with differential's noise suppression
The most expensive
None of them dominates the others. Each is good at something different and pays a different price. Which means picking one and repeating it 49 times also repeats its weakness 49 times.
2.3 The Latin square is a control, not decoration
"Mixing should help" is an easy intuition, but mixing carelessly makes the result impossible to attribute. If full happens to cluster in the late layers, the model did well "because its late layers are full" — not "because it is heterogeneous."
That is why the 49 layers form a 7 × 7 Latin square. A Latin square is a combinatorial arrangement that guarantees each type appears exactly once in each position, structurally preventing any type from concentrating at any depth.
The Latin square is therefore a control mechanism: it lets the question "what does heterogeneous placement do?" be asked without placement bias.
To our knowledge, no released model places heterogeneous attention in a Latin square.
3. Attention composition — a 7×7 Latin square over 49 layers
Seven attention labels are placed across 49 layers along the Latin square, exactly 7 layers each (7 × 7 = 49).
Label
Layers
Description
full
7
Standard causal attention (SDPA)
differential
7
Splits Q·K in half, builds two attention maps, subtracts them λ-weighted to cancel common noise
sliding
7
Sliding window (window = 512) — masking over the full path
nsa
7
Compressed / selected / sliding branches combined by a learned gate
hybrid
7
nsa + differential combined
compress
7
NSA block-mean path — runs the full path in this checkpoint
linear
7
SDPA + a learned gate
Total
49
Where the name 5Attn comes from. The seven labels reduce to five distinct mechanisms: compress runs the full path, and linear is SDPA plus a gate. The model is named for the number of mechanisms (5), not the number of labels (7).
The placement rule and every implementation are in aether_pkg/. See LATIN_SQUARE_7x7 for the layer↔label mapping.
3.1 Measured cost profile per mechanism
Each of the five mechanisms was measured as a single standalone layer: prefill latency and peak memory. An attention's compute and memory cost is a property of the architecture, independent of trained weight values, so this table is reproducible by anyone.
Type
2K (ms / GB)
8K (ms / GB)
32K (ms / GB)
full
0.4 / 0.0
1.5 / 0.2
13.6 / 0.7
differential
0.5 / 0.1
3.7 / 0.3
46.5 / 1.1
sliding
0.6 / 0.1
1.9 / 0.2
7.6 / 0.8
nsa
1.0 / 0.1
3.6 / 0.3
26.8 / 3.5
hybrid
1.5 / 0.1
7.7 / 0.3
74.7 / 3.5
The "different prices" of §2.2 show up as numbers.
At 32K, sliding (7.6 ms) is 1.8× faster than full (13.6 ms) — exactly as a locality-only design should behave. The gap widens with context (at 2K, full is actually faster).
hybrid runs bothnsa and differential, so its cost converges to their sum (26.8 + 46.5 ≈ 74.7).
The full model handles a 32K context in 3.46 GB.
To our knowledge, no measured per-type cost profile for heterogeneous attention has been published. The table itself is material for follow-up work.
This table describes cost only. It makes no claim about output quality. Comparing quality requires a controlled experiment in which attention composition is the only variable, and that must be run separately.
3.2 Structure in depth — the 7×7 Latin square
7x7 Latin-Square Attention Placement
The 49 layers form a 7 × 7 Latin square. Each layer's attention type is:
attention_type(layer) = ATTN_TYPES[(row + col) mod 7]
where row = layer // 7, col = layer % 7
Latin-square property
Each type appears exactly once in every row and exactly once in every column (7 types × 7 layers = 49) — this is the definition of a Latin square.
As a result, no attention type concentrates at any depth (early/middle/late). The cyclic shift spreads every type evenly across depth.
Why 7: 7 is prime, so the cyclic shift (row+col) mod 7 yields a proper Latin square (each row and column is a permutation of the 7 types), and 7² = 49 layers fits exactly.
Why this placement — a control
"Mixing should help" is easy to assume, but careless mixing makes the result impossible to attribute: if full happens to cluster in late layers, the model did well "because its late layers are full", not "because it is heterogeneous". The Latin square removes that bias structurally, so the question "what does heterogeneous placement do?" can be asked without depth bias. It is a control mechanism for reproducible science, not decoration.
7 labels → 5 mechanisms
Seven labels are placed (nsa, differential, full, linear, sliding, compress, hybrid), but there are five distinct mechanisms: compress runs the full path and linear is SDPA + a gate (hatched in the figure). The name 5Attn follows the number of mechanisms (5), not labels (7).
To our knowledge, no released model places heterogeneous attention in a Latin square.
4. Specifications
Item
Value
Total parameters
6.59B
Active parameters
~2.98B (per token)
Layers
49 (7×7 Latin square)
Experts
25, top-7 routing, 1 shared expert
Expert intermediate
640
hidden / intermediate
2048 / 6144
heads / KV heads / head_dim
16 / 4 / 128
Vocabulary
151,936 (Qwen-compatible tokenizer)
Training context
4096
dtype
bfloat16
Size
13.2 GB
Class
AETHERV27wayForCausalLM — load with trust_remote_code=True
5. Training data — the complete recipe
The information below alone is enough to reconstruct our training data byte-for-byte.
5.1 Sources
File
Repository
Config
Tokens
License
eng_fineweb_edu.bin
HuggingFaceFW/fineweb-edu
sample-100BT
15.000B
ODC-By
syn_cosmopedia.bin
HuggingFaceTB/smollm-corpus
cosmopedia-v2
8.000B
ODC-By
math_finemath.bin
HuggingFaceTB/finemath
finemath-3plus
6.002B
ODC-By
code_opc.bin
OpenCoder-LLM/opc-fineweb-code-corpus
default
5.001B
MIT
math_owm.bin
open-web-math/open-web-math
default
4.004B
see source repo
kor_webtext.bin
HAERAE-HUB/KOREAN-WEBTEXT
default
2.492B
see source repo
kor_synth.bin
HAERAE-HUB/KOREAN-SyntheticText-1.5B
default
1.650B
see source repo
phase15_mix_90b.bin
Earlier blend of the sources above
—
90B
—
Preprocessing: AutoTokenizer.from_pretrained("Qwen/Qwen3-14B"), EOS 151645 inserted at document boundaries, written as a flat uint32 array. The reproduction script tokenize_one.py is included.
python tokenize_one.py HuggingFaceFW/fineweb-edu sample-100BT text 15 out/eng_fineweb_edu.bin
trust_remote_code=True is required: this is a custom architecture (aether_v2_7way),
and the modeling code ships in this repository. Loading needs roughly 14 GB of VRAM
in bfloat16.
The module files also remain under aether_pkg/ for anyone who was importing them
directly; the copies at the repository root are what trust_remote_code resolves.
2. use_cache=False is mandatory
This architecture ships no KV cache. Leaving use_cache enabled raises an
IndexError from the standard cache path. Set it on the config and keep it off
in generate().
3. Generation is slow — plan for it
With no KV cache, every new token re-runs the full forward pass, so decoding is
O(n²) in sequence length. Measured throughput:
Hardware
Throughput
NVIDIA T4 (16 GB)
~1.5 tokens/s — 64 tokens takes about 40 s
NVIDIA B200
~7 tokens/s
Keep max_new_tokens small. This is a property of the released architecture, not a
configuration problem.
python
1# REQUIRED: this checkpoint was trained with attention_mask=None. generate() builds a2# mask automatically, which puts the model off-distribution and degenerates the output3# (you get things like "국가의 수도는 국가의 수도입니다..." instead of an answer).4# Drop the mask before it reaches the model:5_forward = model.forward
6def_forward_without_mask(*a,**kw):7 kw.pop("attention_mask",None)8return _forward(*a,**kw)9model.forward = _forward_without_mask
1011out = model.generate(12 tok(prompt, return_tensors="pt").input_ids.to("cuda"),13 max_new_tokens=64, do_sample=False, use_cache=False,14 pad_token_id=tok.eos_token_id,15)16print(tok.decode(out[0], skip_special_tokens=True))
Why the mask has to go
Measured on 2026-07-20, same prompt, greedy decoding, only the mask varied:
attention_mask
Output for 너는 누구야?
absent
저는 비드래프트(VIDRAFT)의 AETHER 모델입니다. 7종의 서로 다른 어텐션 메커니즘을…
This is a property of how the checkpoint was tuned, not a bug in generate().
Greedy decoding is also recommended — sampling drifts off-distribution on this checkpoint.
4. This is a base model
It continues text rather than following instructions. Prompt it with a prefix to
complete, not with a question:
prompt = "대한민국의 수도는" # completion, not an instruction
For instruction-style behaviour see
Aether-7B-5Attn-it,
and read its quality caveats first.
