Highlights
From scratch 24.7M vision encoder — 12.4× smaller than LLaVA's CLIP encoder (307M) Adaptive Token Budget (ATB) Token count is a runtime knob — dense regions get more tokens, blank regions fewer Hierarchical Concept Tokenization (HCT) Every token carries a semantic level: global / region / detail 3-phase pipeline Vision pretraining → projector alignment → SFT + DPO DPO improved every benchmark e.g. VQAv2 +30.7pp , ScienceQA +14.0pp
What This Is
A complete Vision-Language Model (VLM) proof-of-concept built entirely from scratch — no CLIP, no pretrained vision backbone.
Component Details Vision Encoder 24.7M params, trained from scratch on CC3M + CC12M (~15M pairs) Projector LevelAwareProjector (384 → 2048 → 4096)LLM Mistral-7B-Instruct-v0.3 (4-bit NF4 QLoRA)SFT LLaVA-Instruct-150K, 30,000 steps DPO RLHF-V dataset, 5,733 pairs, 3,000 steps
Architecture
Image (256×256)
↓
CNN Stem → ATB Tokenizer → Spatial Transformer (12 layers, embed_dim=384)
↓
KeuralEncoderOutput {tokens, level_ids, spatial_metadata, saliency_scores, pooled}
↓
LevelAwareProjector (384 → 2048 → 4096)
↓
Visual Tokens (N_vis × 4096)
↓
Mistral-7B-Instruct-v0.3 + SFT LoRA + DPO LoRA
↓
Text Response
Keural VLM detailed architecture
Key Innovations
Adaptive Token Budget (ATB) Tokenization — token count is a runtime parameter; dense regions get more tokens, blank regions fewer.
1 out = encoder ( image , token_budget = 64 ) # fast / cheap
2 out = encoder ( image , token_budget = 256 ) # default
3 out = encoder ( image , token_budget = 1024 ) # full fidelity
Hierarchical Concept Tokenization (HCT) — every token carries a semantic level tag.
1 out = encoder ( image )
2 print ( out . level_ids ) # {0=global, 1=region, 2=detail}
Training Pipeline
Phase What trains Data Steps Result 1 · Vision Encoder Encoder from scratch CC3M + CC12M (~15.3M) ~75,000 24.7M params · 1× RTX 5090 2A · Projector Align Projector + LLM LoRA (r=64), encoder frozen LLaVA-Instruct-150K 10,000 — 2B · SFT LLM LoRA (r=64, α=128) LLaVA-Instruct-150K 30,000 final loss 1.022 2B · DPO DPO LoRA (r=16, α=32) RLHF-V (5,733 pairs) 3,000 loss 0.235 · reward acc 95% · margin 2.11
Benchmark Results
Evaluated on 1,000 samples each (where applicable). The vision encoder is 12.4× smaller than LLaVA's CLIP encoder (307M).
Benchmark Keural SFT-30K Keural SFT+DPO LLaVA 1.5 (307M enc) LLaVA 1.6 (307M enc) VQAv2 Accuracy 12.9% 43.6% 78.5% 81.8% POPE F1 66.9% 67.0% 85.9% 86.5% MME Total Score 704.3 838.8 1510.7 1519.3 TextVQA Accuracy 0.8% 6.6% 58.2% 64.9% ScienceQA Accuracy 39.7% 53.7% 66.8% 70.6%
POPE F1 (67.0%) is the standout — within 19pp of LLaVA 1.6 using a 12× smaller encoder.
DPO improved every benchmark , most dramatically VQAv2 (+30.7pp) and ScienceQA (+14.0pp).
TextVQA is low by design — no OCR training. EasyOCR integration in the GUI bridges this gap.
All benchmarks
Qualitative & Saliency
The ATB tokenizer concentrates tokens on salient regions. Left → right: original · saliency heatmap · token placement.
Saliency — cat Qualitative examples
DPO Training Curves
Metric Step 0 Step 3000 Loss 0.694 0.235 Reward Accuracy ~50% 95% Reward Margin 0.0 2.11
DPO training curves
Repository Structure
keural-vlm-poc/
├── vision_encoder/ # Keural encoder weights (config + safetensors)
├── sft_adapter/ # SFT LoRA (30K steps)
├── dpo_adapter/ # DPO LoRA (3K steps, RLHF-V) — stacks on SFT
├── assets/ # figures, diagrams, animation
└── tokenizer.json … # Mistral tokenizer + chat templateFor inference, load: Vision Encoder → Projector → Mistral-7B + SFT LoRA + DPO LoRA .
Usage
1 import torch
2 from transformers import AutoModel , AutoModelForCausalLM , AutoTokenizer , BitsAndBytesConfig
3 from peft import PeftModel
4 from alignment . projectors import LevelAwareProjector
5
6 device = "cuda"
7
8 # 1. Vision encoder (frozen)
9 encoder = AutoModel . from_pretrained (
10 "mkd-hika/keural-vision-encoder-poc" ,
11 trust_remote_code = True , torch_dtype = torch . bfloat16 ,
12 ) . to ( device ) . eval ( )
13
14 # 2. Projector
15 projector = LevelAwareProjector ( encoder_dim = 384 , hidden_dim = 2048 , llm_dim = 4096 )
16 projector . load_state_dict ( torch . load ( "projector.pt" , map_location = device ) )
17 projector = projector . to ( device , dtype = torch . bfloat16 ) . eval ( )
18
19 # 3. LLM + SFT LoRA + DPO LoRA
20 bnb_cfg = BitsAndBytesConfig ( load_in_4bit = True , bnb_4bit_compute_dtype = torch . bfloat16 ,
21 bnb_4bit_use_double_quant = True , bnb_4bit_quant_type = "nf4" )
22 base_llm = AutoModelForCausalLM . from_pretrained (
23 "mistralai/Mistral-7B-Instruct-v0.3" ,
24 quantization_config = bnb_cfg , device_map = "auto" , torch_dtype = torch . bfloat16 ,
25 )
26 llm = PeftModel . from_pretrained ( base_llm , "sft_adapter" ) # SFT LoRA
27 llm = PeftModel . from_pretrained ( llm , "dpo_adapter" ) # DPO LoRA
28 llm . eval ( )
Roadmap
Phase Encoder Params Hardware Status PoC (this model)24.7M 1× RTX 5090 Complete (SFT + DPO) Mid-level ~183.5 8× H100 80 GB Planned Commercial ~1.1B 64× H100 80 GB Future
Citation
1 @misc{keural_vlm_2026,
2 title = {Keural VLM: Vision-Language Model with Content-Adaptive Encoding via Saliency-Guided Token Budgets},
3 author = {Hika Barki and MKD Co., Ltd.},
4 year = {2026},
5 },
6 }
📄 License
Training data: CC3M, CC12M, LLaVA-Instruct-150K, RLHF-V — respective data licenses apply.
