This is the vision encoder — a Vision Transformer that encodes images into patch-level features.
This vision encoder is trained from scratch and generatively: a frozen in-house LLM decoder consumes the visual tokens and is optimized with a Next-Token-Prediction (NTP) objective, so the encoder is aligned from the very beginning to the exact form it will take in the final VLM. A.X VE is developed as part of the Korean government's Sovereign AI foundation model project.
The full A.X VLM built on top of this vision encoder will be released soon.
Key Features
Trained from scratch, generatively. A dedicated in-house LLM decoder is used directly as the decoder, and only the generative (NTP) signal is used as the training objective. The pretraining pipeline mirrors real VLM usage, so the encoder is aligned to its final deployment form from the start.
Two-stage curriculum (low-res alignment → high-res refinement). Stage 1 builds vision–language grounding on large-scale captioning at lower resolution; Stage 2 scales up the resolution and strengthens document/text (OCR) understanding.
Task-diverse spatial & OCR strength. Beyond captions, training mixes Parsing / Transcription / Grounding data to directly boost document/text recognition and object-location/spatial understanding from the pretraining stage.
Multi-level residual aggregation. Intermediate features from blocks are added back into the final hidden state, giving a multi-scale representation that combines low-level visual cues with deep semantics.
Model Details
The Vision Encoder is a Vision Model (ViT) that encodes images into patch-level features.
The vision encoder is trained from scratch using a generative pipeline built around an in-house LLM decoder.
Setup
Vision Model: ViT
Projector (training only): a 2-layer MLP that concatenates 2×2 spatially-merged patches and projects visual features into the decoder's embedding space.
Decoder (training only, frozen): a compact in-house language model, pretrained from scratch and used only as the decoder during vision-encoder training.
The Vision Model and Projector are trainable; the LLM is frozen so its language ability is preserved and the visual features are aligned into a space the decoder can already understand. The projector aligns visual features into the language embedding space, and training focuses on vision–language alignment via Next-Token-Prediction (NTP).
Two-Stage Curriculum
The Vision Encoder is trained in two stages that differ in resolution and data composition, with the language decoder kept frozen throughout.
Stage 1 — low-resolution alignment. Learns the basic correspondence between visual information and language from scratch, using large-scale captioning data at lower resolution. A relatively large learning rate with generous warmup and cosine decay aligns the representation space quickly and stably, emphasizing general visual and spatial understanding.
Stage 2 — high-resolution refinement. Scales up the resolution and reinforces text/document recognition. The learning rate and warmup are lowered to minimize disruption to the Stage-1 representation, while adding Parsing / Transcription / Grounding data on top of captioning to strengthen OCR and grounding.
Training characteristics
Generative training with an in-house LLM decoder — the decoder is used directly and only the generative loss serves as the training signal. Because the pretraining pipeline matches how the encoder is used in the final VLM, the encoder is aligned to its final deployment form from the start, keeping the training and inference objectives consistent.
NTP-based learning — visual tokens from the Vision Encoder are fed to the frozen decoder, which is trained to generate text about the image; only the Vision Encoder is updated, so the visual representation is shaped into a space the decoder can consume while the LLM's language ability is preserved.
Task-diverse spatial & OCR strengthening — Parsing/Transcription directly strengthen document/text recognition (OCR), and Grounding directly strengthens object-location/spatial understanding, raising document understanding and position-based task performance from the pretraining stage.
Performance
Benchmark comparison. Scores for A.X VE and the SigLIP2:
Model
KRETA
MMMU
ChartQA
DocVQA
OCRBench
InfoVQA
TextVQA
A.X VE
60.8
31.2
72.6
82.7
62.3
48.7
67.8
SigLIP2
54.1
30.7
67.4
78.6
56.7
45.9
66.6
Note: These results were obtained from experiments using a small in-house LLM as the decoder, so the absolute scores are for relative comparison between vision encoders rather than final VLM performance.
Usage
Requirements
pip install "transformers>=4.55.4" torch pillow
The model ships with custom modeling code, so load it with trust_remote_code=True.
Extracting image features
This checkpoint holds the vision encoder only (no language backbone), so the vision tower is instantiated directly from the config and its weights are loaded from model.safetensors.
This is a vision encoder — it produces visual features, not text. It is meant to be used as the visual front-end of a VLM together with a language backbone, not as a standalone model.
The output visual features are aligned to the training decoder's embedding space; pairing the encoder with a different language backbone may require additional adaptation.
Encoding quality may vary with image domain and characteristics (resolution, aspect ratio, visual style) depending on the coverage of the training data.
For questions about A.X VE — including model behavior, deployment, and licensing — contact the A.X team at a.x@sk.com. Please send reports of vulnerabilities, harmful outputs, suspected misuse, or copyright infringement claims to the same address.
