Kelix training pipeline: Kelix-Tok, Unified LLM, and Image DiT
Figure 1: The auto-regressive training workflow of Kelix, including the Kelix Tokenizer, the Unified LLM, and the Image DiT de-tokenizer.
Introduction
Kelix-SFT is the complete end-to-end release of Kelix at the supervised-fine-tuning (SFT) stage — a fully discrete autoregressive unified multimodal model. This release bundles all three components of the Kelix pipeline:
Kelix-LLM — the unified Qwen3-8B backbone trained with Next-Block Prediction (NBP), with the vocabulary expanded by 65,536 visual entries,
Kelix-DiT (SFT) — the SFT-stage diffusion-based image de-tokenizer, fine-tuned from OpenOneRec/Kelix-DiT.
Together they form a modular Tokenizer → LLM → Detokenizer system that unifies multimodal understanding and image generation under a single autoregressive objective. Use this repo if you want the full Kelix pipeline for both understanding and generation; use OpenOneRec/Kelix-DiT if you only need the pretraining-stage image de-tokenizer component.
Kelix achieves state-of-the-art results among comparable-scale unified models on both understanding and generation benchmarks; notably, it reaches 86.7 on OCRBench, matching continuous-feature VLMs and surpassing the previous best discrete model by +23%.
What's in this release
Component
Description
Init / Base
Kelix-Tok
NaViT-style vision encoder + multi-token VQ (N=8 independent sub-codebooks, total size S=65,536, sum-pooled on the encoder side). Encodes images into discrete visual codes aligned with the LLM text space.
NaViT initialized from Keye-VL 1.5; codebooks K-means initialized, trained with SimVQ
Kelix-LLM
Unified autoregressive backbone with Block Encoder / Block Decoder and a Next-Block Prediction (NBP) objective (text block size 2, visual block size N+1=9).
Qwen3-8B, vocabulary expanded by 65,536 visual entries
Kelix-DiT (SFT)
SFT-stage diffusion-based image de-tokenizer (flow-matching, SANA-DiT based), conditioned on the LLM's last hidden states.
⚠️ The DC-AE-F32C32 VAE used to map pixels ↔ latents for Kelix-DiT is frozen and not retrained, so it is not duplicated in this release. Please load it from its original repo when running the image-generation path.
Training (SFT Stage)
This release corresponds to the Stage 4: Supervised Fine-Tuning of the Kelix-LLM pipeline and the SFT Stage of the Kelix-DiT pipeline, building directly on the annealed Stage-3 checkpoints:
Kelix-LLM (Stage 4 — SFT):
Data: replayed high-quality Stage-3 data, task-specific chain-of-thought (CoT) data, targeted image-generation data (attribute / subject / scene / style / perspective control), and task-specific enhancement data (complex OCR, math reasoning, STEM).
Filtering: rejection sampling is retained to filter noisy samples and mitigate hallucinations.
Objective: preserves the unified multimodal capability while boosting task-specific performance, keeping understanding and generation in balance.
Kelix-DiT (SFT Stage):
Data: high-quality text-to-image data filtered via rejection sampling to remove low-quality pairs and mitigate hallucinations.
Fine-grained control: curated to cover key control dimensions — attribute control (color, size), subject control (quantity, type), scene generation (indoor/outdoor, time of day), artistic style (oil painting, sketch), and perspective (top-down, close-up).
Target image: strictly constrained to 1024×1024.
Optimization: all DiT parameters are updated; the Kelix LLM, DC-AE encoder, and DC-AE decoder are frozen.
Together, these significantly improve the model's adherence to diverse and precise textual instructions, enabling high-fidelity generation that strictly complies with complex prompts while maintaining strong multimodal understanding.
Usage
This release is the full Kelix pipeline and can be used for both understanding and generation end-to-end (the only external dependency is the frozen DC-AE-F32C32 VAE for the generation path).
Image understanding (VQA / OCR / reasoning):
Encode the input image(s) with Kelix-Tok into N-way discrete visual codes, sum-pooled into a single composite token per patch on the encoder side.
Feed the visual codes + text tokens into Kelix-LLM (Block Encoder → NBP backbone → Block Decoder).
Decode the predicted text tokens via the standard text tokenizer.
Image generation (text-to-image, 1024×1024):
Feed a text prompt (and optional images) into Kelix-LLM, which autoregressively produces last hidden states {h_*} for the image blocks.
Use {h_*} as the semantic condition (y-embedder input) for Kelix-DiT (SFT).
Run flow-matching denoising in the DC-AE latent space (32×32) and decode with the frozen DC-AE to obtain a 1024×1024 image.
💡 For broad-scenario generalization on the generation side, the pretraining-stage de-tokenizer OpenOneRec/Kelix-DiT can be swapped in for Kelix-DiT (SFT); Kelix-Tok and Kelix-LLM in this release are compatible with both.
Key Results
Kelix (8B) results with this release:
Image understanding
Benchmark
SEED-Bench
RealWorldQA
MMBench-EN
AI2D
MMMU
MathVista
ChartQA
TextVQA
OCRBench
Score
76.0
72.1
80.2
82.4
54.1
76.5
83.0
81.4
86.7
Image generation
Benchmark
GenEval (Overall)
WISE (Overall)
DPG-Bench (Overall)
Score
87.6
57.0
85.5
Highlights (see the technical report for full tables):
OCRBench 86.7 — matching Qwen2.5-VL-7B (86.4), +23% over the previous best discrete unified model X-Omni (70.4). First discrete-token unified model to close the gap with continuous-feature VLMs on text-rich tasks.
MathVista 76.5 — surpasses the 14B Bagel (73.1) and the 30B Manzano (73.3) despite a much smaller parameter count.
GenEval 87.6 — SOTA among discrete-tokenization unified models, +0.6 over the 27B Qwen-Image.
WISE 57.0 — 2nd only to Nextflow (7B, 59.0), beating all continuous-tokenization unified models and larger dedicated T2I models (e.g., FLUX.1-dev 12B, 50.0).
DPG-Bench 85.5 — competitive with the SOTA X-Omni (7B, 87.7) without using reinforcement learning.
Citation
If you find Kelix useful, please cite our technical report.
bibtex
1@article{kelix2026,
2 title = {Kelix Technique Report: Closing the Understanding Gap of Discrete Tokens in Unified Multimodal Models},
3 author = {Kuaishou Technology},
4 journal = {arXiv preprint arXiv:2602.09843},
5 year = {2026},
6 url = {https://arxiv.org/abs/2602.09843}
7}
License
Please contact the OneRec Team for the license of the Kelix series. The base components (Qwen3-8B, SANA-DiT, DC-AE, Keye-VL / NaViT) are subject to their respective original licenses.
