[!Note]
This repository contains model weights and configuration files for the post-trained (instruction-tuned) model in the Hugging Face Transformers format.
These artifacts are compatible with Hugging Face Transformers, llama.cpp, ExecuTorch, and vLLM. Executable binaries optimized for NPU inference (Qualcomm Hexagon NPU, Apple ANE) are also provided.
In light of its parameter scale and on-device design goals, the intended use cases are on-device text generation, Korean-English bilingual assistants, task-specific fine-tuning, and other research or development purposes.
Opt.Gear is the first generation of OptAI Foundation Models, designed for efficient on-device deployment, real-time inference, and strong task capability. Rather than simply shrinking a server-scale LLM, Gear jointly optimizes downstream quality, device-side latency, memory efficiency, long-context capability, and practical deployability under hardware constraints such as limited SRAM, memory bandwidth, and power.
Opt.Gear Highlights
Data-efficient training: Trained on a curated 0.5T-token subset selected from a 2T-token candidate corpus, without knowledge distillation from a teacher model — the most data-efficient recipe among comparable open foundation models.
Hybrid architecture with ConvKV-Gated Mixer: A small set of global GQA layers handles long-range routing, while ConvKV-Gated Mixer blocks replace many local attention layers with softmax-free, matrix-multiplication-free local mixing. The persistent local state scales with the convolution kernel (L_conv = 3) instead of the sliding window (W = 512), substantially reducing live decoding state and memory bandwidth.
NPU-friendly by design: Dynamic QKV matrix multiplications and softmax normalization are replaced with static linear, convolution, and element-wise operations, delivering up to ×4.9 faster prefill and decoding on NPUs compared to models of similar scale (7,042 tokens/s prefill on Snapdragon GEN5).
Korean-English bilingual: Built on the KORMo tokenizer (125,184 vocab) with Korean-English pretraining, achieving distinct superiority on Korean benchmarks (KMMLU, KoBEST, CLIcK, HAERAE) against same-scale baselines.
64K context on device: Hybrid attention layout with separate global/local RoPE frequencies supports long-context modeling up to 65,536 tokens while keeping the local cache footprint small.
For more details, please refer to our tech report and blog post.
Model Overview
Type: Causal Language Model (hybrid attention + convolutional mixer)
Word Embedding: untied (separate input embedding and LM head)
Context Length: 65,536 natively
[!Important]
Base and Instruction models share the same tokenizer, but use different end-of-generation tokens: the Base model emits <EOS> while the Instruction model emits <EOT>. When fine-tuning, make sure training examples terminate with the same convention used by the target runtime — malformed termination wastes decoding steps and increases latency on device.
Benchmark Results
All results are obtained using the LM Evaluation Harness and may differ from scores reported elsewhere. Perplexity-based evaluation is adopted for HellaSwag, PIQA, WinoGrande, MMLU, MMLU-Pro, GPQA, ARC, KMMLU, and KoBEST.
Opt.Gear-1B
Gemma3-1B
Llama3.2-1B
EXAONE4-1.2B
LFM2.5-1.2B
Qwen3-1.7B
# Trained Tokens
0.5T
2T
9T
12T
28T
36T
Distilled
✗
✓
✓
✓
✓
✓
English
MMLU
43.2
39.8
46.1
37.2
51.1
60.3
Korean
KMMLU
36.0
30.7
29.9
32.6
29.2
41.6
KoBEST
60.3
59.5
51.8
50.7
59.1
62.8
CLIcK
39.7
37.8
30.6
32.9
38.3
49.4
HAERAE
44.2
35.3
32.6
30.2
33.9
52.0
* All results are obtained using LM Evaluation Harness and may differ from other reported scores.
* While baselines are trained on 2T–36T tokens and frequently leverage knowledge distillation, Opt.Gear-1B is trained on only 0.5T tokens without any distillation.
On-Device Inference Performance
Measured across hardware processors (CPU/GPU/NPU) and runtimes (Qualcomm AI Engine Direct, llama.cpp, CoreML) with W4A16 quantization; embedding and LM head kept in 16-bit. Numbers are Prefill / Decode (tokens/sec).
Model
Snapdragon GEN5 (NPU)
Snapdragon GEN4 (NPU)
iPhone 17 Pro (NPU)
iPhone 16 Pro (NPU)
Opt.Gear-1B
7042 / 86
5882 / 80
3085 / 97
2589 / 80
Llama3.2-1B
4481 / 65
3013 / 50
1073 / 34
696 / 25
Gemma3-1B
3226 / 61
2724 / 51
2380 / 90
1862 / 70
EXAONE4-1.2B
4386 / 57
3436 / 49
590 / 20
319 / 15
* Snapdragon: Qualcomm AI Runtime (QAIRT). iPhone: CoreML Runtime. Full CPU/GPU results and llama.cpp results are available in the tech report.
* Note: the llama.cpp NPU backend does not allocate the sliding-window attention pattern to hardware accelerators, which degrades prefill performance in that specific combination. QAIRT is recommended for Snapdragon NPU deployment.
llama-cli -m opt-gear-1b-w4.gguf -p "프랑스의 수도가 뭐야?"" -n 128
On-Device Deployment
Executable binaries optimized for NPU inference are provided for:
Qualcomm Hexagon NPU — via Qualcomm AI Engine Direct (QAIRT); recommended for Snapdragon devices
Apple ANE — via CoreML Runtime
ExecuTorch — for mobile/edge PyTorch deployment
Quantization
A QAT (Quantization-Aware Training) checkpoint is available at OptAI/Opt.Gear-1B-QAT.
The QAT model uses group-size-32 symmetric INT4 weights (W4A16) with Hessian-based scale initialization and Learned Step Size Quantization. On six English/Korean benchmarks, the QAT model loses only ~1.0 point on average from the Base model, and remains stable even when re-quantized with other INT4 PTQ methods (RTN/AWQ/GPTQ) — see the tech report for details.
Best Practices
Termination tokens: The Instruction model terminates generation with <EOT>, while the Base model uses <EOS>. When fine-tuning either variant, ensure training examples terminate with the matching token — this is particularly important for on-device inference, where malformed termination wastes decoding steps and increases latency.
Runtime selection:
Snapdragon NPU → QAIRT (Qualcomm AI Engine Direct). Avoid the llama.cpp NPU backend, which cannot hardware-accelerate the sliding-window attention pattern.
Apple devices → CoreML (best-in-class GPU/ANE throughput at this scale).
CPU/GPU → llama.cpp works well and delivers stable performance.
Languages: The model is trained primarily on English (~92%) and Korean (~6%) with mathematical text (~2%). It is best suited for Korean-English bilingual understanding, summarization, rewriting, and instruction following.
Not intended for code generation: The pre-training corpus deliberately excludes code data — models at the 270M–1B scale are not typically used for code generation tasks. Use a code-specialized model for such workloads.
Long context: The model natively supports 65,536 tokens. The hybrid ConvKV architecture keeps the local cache footprint fixed, so long-context decoding remains memory-efficient on device.
Limitations
The constrained 0.5T-token training budget limits capabilities in complex reasoning and mathematical tasks that typically benefit from larger training corpora. The current results reflect both the capacity of the proposed architecture and the limitations of the available data and compute budget. See the tech report for a detailed discussion.
Citation
If you find our work helpful, feel free to give us a cite.
