Image
  -> SigLIP2 vision encoder
  -> pooled visual feature
  -> visual projector
  -> visual token

English phrase
  -> mBART encoder

[visual token + mBART encoder hidden states]
  -> mBART decoder with LoRA adapters
  -> Vietnamese translation

Intended Use

This model is intended for English-to-Vietnamese multimodal phrase translation where image context can help disambiguate the source phrase.

Example input:

text

Image: an image containing a bicycle
English phrase: "a man riding a bicycle"
Output: Vietnamese translation

Repository Files

text

model_state.pt
model_config.json
inference.py
tokenizer/
image_processor/
requirements.txt
README.md

Usage

Install dependencies:

pip install -r requirements.txt

Load and run the model:

python

from inference import load_model, translate_image_phrase

repo_id = "AnhDau/siglip2-mbart-envi-vlmt"

model, tokenizer, image_processor, config, device = load_model(repo_id)

prediction = translate_image_phrase(
    model=model,
    tokenizer=tokenizer,
    image_processor=image_processor,
    image_path="example.jpg",
    english_phrase="a man riding a bicycle",
    config=config,
    device=device,
)

print(prediction)

Notes

This repository contains a custom Vision-Language model architecture. It should be loaded with inference.py rather than directly with AutoModel.from_pretrained.