A unified Mixture-of-Experts Vision-Language Model designed from scratch for Intel NPU hardware.
Why?
Intel NPUs (Arrow Lake 3720 and similar) have severe constraints:
4 MB SRAM (2 MB per NCE tile), software-managed, no cache
<10 GB/s DMA bandwidth to system RAM
FP16-only compute (6.5 TOPS), INT8 at 13 TOPS
Static shapes only — no dynamic dimensions
Standard dense models are memory-bound on decode: a 1B model at INT4 needs 500 MB DMA'd through a 10 GB/s pipe every token → 50ms/token just moving data. Compute is negligible.
MoE solves this. With top-2 routing over 32 experts, only ~125M active params (63 MB INT4) need loading per token → 6.3ms DMA theoretical. Experts are sized to tile cleanly in NPU SRAM.
No separate ViT, no projection layer. One transformer:
Image → Conv2d patch embed → [visual tokens]
Text → Embedding lookup → [text tokens]
↓ ↓
[visual_tokens, text_tokens] ← single sequence
↓
Shared MoE Transformer
↓
Text logits
The patch embedding is a single nn.Conv2d(3, 512, kernel_size=16, stride=16). Cross-modal attention is free — it's just self-attention over the concatenated sequence. Experts naturally specialize between visual and linguistic processing.
Training
Streams everything from HuggingFace — zero local disk required.
Auto-resume: Picks up from local or HF Hub checkpoint (whichever is newer)
Auto-push: Checkpoints to HuggingFace Hub every 1000 steps
NPU Deployment (WIP)
After training, the model exports to OpenVINO IR with INT4 weight compression and runs on Intel NPU via openvino_genai. The expert dimensions are designed to tile within the NPU's 2 MB SRAM per NCE tile for maximum DMA efficiency.
