Vie-BLIP is a Vietnamese image captioning model fine-tuned from Salesforce/blip-image-captioning-base. The model takes an image as input and generates a Vietnamese natural-language caption describing the image content.

This repository contains the model weights, processor configuration, tokenizer files, and generation configuration required to load the model with 🤗 Transformers.

Model Details

Model name: Vie-BLIP
Repository: nviquang/Vie-BLIP
Task: Image captioning / Image-to-text
Language: Vietnamese (vi)
Base model: Salesforce/blip-image-captioning-base
Architecture: BlipForConditionalGeneration
Library: 🤗 Transformers
Image processor: BlipImageProcessorFast
Tokenizer: PhobertTokenizer
Image size: 384 x 384
Model weights format: safetensors

Intended Use

Vie-BLIP is intended for generating Vietnamese captions for images. Example use cases include:

Vietnamese image captioning research
Vision-language model experimentation
Educational projects
Assistive captioning prototypes
Webcam or local image captioning demos

Repository Files

The repository includes:

text

.
├── added_tokens.json
├── bpe.codes
├── config.json
├── generation_config.json
├── model.safetensors
├── processor_config.json
├── tokenizer_config.json
├── training_curves.png
└── vocab.txt

Installation

Install the required dependencies:

pip install -U transformers torch safetensors pillow

If tokenizer loading requires additional dependencies in your environment, install:

pip install sentencepiece

Usage

python

from PIL import Image
import torch
from transformers import BlipProcessor, BlipForConditionalGeneration

repo_id = "nviquang/Vie-BLIP"

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = BlipProcessor.from_pretrained(repo_id)
model = BlipForConditionalGeneration.from_pretrained(repo_id).to(device)
model.eval()

image = Image.open("example.jpg").convert("RGB")

inputs = processor(images=image, return_tensors="pt").to(device)

with torch.no_grad():
    generated_ids = model.generate(
        pixel_values=inputs["pixel_values"],
        max_length=64,
        num_beams=5,
        early_stopping=True,
        no_repeat_ngram_size=3,
        repetition_penalty=1.5,
    )

caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(caption.strip())

Example Output

một người đang đi xe máy trên đường phố

Actual outputs may vary depending on the image content and generation parameters.

Dataset

The model was fine-tuned for Vietnamese image captioning using the UIT-OpenViIC dataset.

Preprocessing and dataset analysis used in the project reported the following statistics:

Split	Images	Captions
Train	9,088	41,238
Dev	2,011	10,002
Test	2,001	10,001
Total	13,100	61,241

After basic cleaning, 61,239 valid captions were retained.

Training Summary

The model was fine-tuned from Salesforce/blip-image-captioning-base for Vietnamese caption generation. The training pipeline used image-caption pairs from UIT-OpenViIC and saved the final checkpoint in the standard 🤗 Transformers format.

The included training_curves.png file shows the training curves from the fine-tuning process.

Evaluation

The project evaluation pipeline used common image captioning and text generation metrics:

BLEU
ROUGE-L
METEOR

Exact metric values are not included in this model card yet. If available, they should be added here for reproducibility.

Limitations

The model is optimized for Vietnamese image captioning and may not perform well in other languages.
The model may produce generic, incomplete, or incorrect captions.
The model may struggle with images that are very different from the UIT-OpenViIC training distribution.
Generated captions may miss fine-grained details, counts, text in images, or rare objects.
The model may inherit biases or annotation patterns from the training dataset.

Ethical Considerations

This model should be used as an assistive caption generation system, not as a source of factual truth. Outputs should be reviewed before use in high-stakes settings such as accessibility, education, medical, legal, or safety-critical applications.

License

The license is currently marked as other. Please verify the license terms of the base model and the UIT-OpenViIC dataset before using this model in commercial or public production settings.

Citation

If you use this model, please cite the original BLIP work and the UIT-OpenViIC dataset where appropriate.

bibtex

@inproceedings{li2022blip,
  title={BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation},
  author={Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven},
  booktitle={International Conference on Machine Learning},
  year={2022}
}

Acknowledgements

This model is based on Salesforce/blip-image-captioning-base and was fine-tuned for Vietnamese image captioning.