from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("axiom-of-choice/OmniParser-v2-icon-caption-4-bit (group_size=64)")
prompt = apply_chat_template(processor, model.config, "<CAPTION>", num_images=1)
output = generate(model, processor, prompt, image=crop, max_tokens=20, temperature=0.0)
print(output.text)

Source

Architecture: Florence-2-base fine-tune (BART decoder + ViT vision encoder)
Parameters: ~270M (quantized language model, vision tower bf16)
Input: 64×64 RGB crop of a detected icon
Output: Short text caption (e.g., &quot;Cancel&quot;, &quot;a yellow circle&quot;)

Fine-tuned weights from microsoft/OmniParser-v2.0/icon_caption, processor from microsoft/Florence-2-base. Config patched for transformers 5.x compatibility (auto_map removed, text_config.model_type set to "bart").

axiom-of-choice/OmniParser-v2-icon-caption-4bit

OmniParser v2.0 Icon Captioner (Florence-2) — MLX 4-bit (group_size=64)

Model Details

Usage

Source

axiom-of-choice/
OmniParser-v2-icon-caption-4bit