This checkpoint has been refreshed with an expanded Korean instruction-tuning run.
Training data
14,865 samples — Korean general instructions (12,000) + Korean exam-style reasoning (2,400) + model identity (465)
Method
LoRA (r16, alpha32) on q/k/v/o/gate/up/down projections, completion-only loss, entropy-gated weighting
Schedule
1 epoch, LR 2e-4, merged back into the base weights
Language policy
Korean prompts are answered in Korean, English prompts in English (language-matched training data)
What changed
Korean responsiveness. The previous checkpoint frequently returned empty completions for
Korean prompts; this revision answers them.
Model identity. The model now identifies itself as an AETHER model developed by VIDRAFT.
Language matching. Korean in, Korean out; English in, English out.
Known limitations
Factual accuracy in Korean history/knowledge remains weak. The instruction mix was not
large enough to reliably ground factual questions; verify factual claims before relying on them.
Repetition under greedy decoding. Use sampling (e.g. temperature=0.7, top_p=0.9)
rather than pure argmax decoding.
This is a 7B-class MoE base with light instruction tuning, not a fully aligned chat model.
🧩 Intermediate checkpoints (110k · 115k · 162k) are released as a dataset: Aether-7B-5Attn-checkpoints.
Instruction-tuned (SFT) version of the fully-open Aether-7B-5Attn base model. Post-trained for multiple-choice / benchmark-style answering. 6.59B MoE (~2.98B active), 49 layers on a 7×7 Latin square. Apache-2.0.
Learning rate chosen by held-out accuracy, not loss
Full-parameter SFT was run at three learning rates (2e-6 / 6e-6 / 2e-5). The final checkpoint was selected by held-out benchmark accuracy, not training loss — for small models a high LR lowers training loss while degrading real capability, so loss is the wrong selector.
vs base (before SFT): base 26.7% → 34.9% (+8.2pp) — same held-out set, same harness.
All evaluations use held-out sets not seen in training. SFT was performed on MMLU-auxiliary (multiple-choice format), which does not overlap with the evaluation subjects (GPQA / K-AI).
Pretraining data mix (inherited from base)
Domain
Share
Math (finemath + open-web-math)
37.8%
Korean (webtext + synth)
21.6%
English web & synthetic (fineweb-edu + cosmopedia)
21.6%
Code (opc)
13.5%
phase15 pre-blend
5.4%
This is the base pretraining mix. This model's post-training (SFT) data is MMLU-auxiliary.
Architecture (inherited from base)
Identical to Aether-7B-5Attn base: 49 layers placed on a 7×7 Latin square with heterogeneous attention (7 labels / 5 distinct mechanisms) and a 25-expert MoE (top-7 + 1 shared). Full structural detail and the diagram are in the base card, §3.2:
FINAL-Bench/Aether-7B-5Attn.
Relative to the base Aether. Among six sovereign fully-open models, VIDRAFT is the only single AI startup, and Aether has the most attention types (5) in a Latin-square layout.
trust_remote_code=True is required: this is a custom architecture (aether_v2_7way),
and the modeling code ships in this repository. Loading needs roughly 14 GB of VRAM
in bfloat16.
The module files also remain under aether_pkg/ for anyone who was importing them
directly; the copies at the repository root are what trust_remote_code resolves.
2. use_cache=False is mandatory
This architecture ships no KV cache. Leaving use_cache enabled raises an
IndexError from the standard cache path. Set it on the config and keep it off
in generate().
3. Generation is slow — plan for it
With no KV cache, every new token re-runs the full forward pass, so decoding is
O(n²) in sequence length. Measured throughput:
Hardware
Throughput
NVIDIA T4 (16 GB)
~1.5 tokens/s — 64 tokens takes about 40 s
NVIDIA B200
~7 tokens/s
Keep max_new_tokens small. This is a property of the released architecture, not a
configuration problem.
python
1# REQUIRED: this checkpoint was trained with attention_mask=None. generate() builds a2# mask automatically, which puts the model off-distribution and degenerates the output3# (you get things like "국가의 수도는 국가의 수도입니다..." instead of an answer).4# Drop the mask before it reaches the model:5_forward = model.forward
6def_forward_without_mask(*a,**kw):7 kw.pop("attention_mask",None)8return _forward(*a,**kw)9model.forward = _forward_without_mask
1011out = model.generate(12 tok(prompt, return_tensors="pt").input_ids.to("cuda"),13 max_new_tokens=64, do_sample=False, use_cache=False,14 pad_token_id=tok.eos_token_id,15)16print(tok.decode(out[0], skip_special_tokens=True))
Why the mask has to go
Measured on 2026-07-20, same prompt, greedy decoding, only the mask varied:
attention_mask
Output for 너는 누구야?
absent
저는 비드래프트(VIDRAFT)의 AETHER 모델입니다. 7종의 서로 다른 어텐션 메커니즘을…
This is a property of how the checkpoint was tuned, not a bug in generate().
Greedy decoding is also recommended — sampling drifts off-distribution on this checkpoint.
4. Prompt format
Instruction tuning used a plain format — the instruction, a blank line, then the
response. The bundled chat_template.jinja reproduces exactly this, so
apply_chat_template and the raw form below are equivalent:
prompt = "대한민국의 수도는 어디인가요?" + "\n\n"
5. Quality caveats — please read before use
Instruction tuning here is light: 14,865 samples for a single epoch. Measured
consequences, stated plainly:
Sensitive to prompt format and decoding settings. Outputs degrade sharply
once you move away from the trained format. Greedy decoding on the format above
is the only configuration we validated.
Factual accuracy in Korean history and general knowledge is weak. The model
produces fluent but confidently wrong statements. Verify anything factual.
Benchmark: on a held-out 4-subject Korean set (CLIcK / KMMLU / MuSR / Com2,
80 items, no train overlap) this revision scores 31.2% against 33.8% for
the previous one — within noise of each other and near the 25% random baseline.
Treat this as a lightly instruction-tuned research checkpoint, not a
production assistant.
