from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("axiom-of-choice/OmniParser-v2-icon-caption-bf16")
prompt = apply_chat_template(processor, model.config, "<CAPTION>", num_images=1)
output = generate(model, processor, prompt, image=crop, max_tokens=20, temperature=0.0)
print(output.text)

Source

Architecture: Florence-2-base fine-tune (BART decoder + ViT vision encoder)
Parameters: 270.9M
Input: 64×64 RGB crop of a detected icon
Output: Short text caption (e.g., &quot;Cancel&quot;, &quot;a yellow circle&quot;)

Fine-tuned weights from microsoft/OmniParser-v2.0/icon_caption, processor from microsoft/Florence-2-base. Config patched for transformers 5.x compatibility (auto_map removed, text_config.model_type set to "bart").