LLaVA Vietnamese AIO is a Vietnamese vision-language checkpoint for image understanding, visual question answering, and instruction-style multimodal responses. It combines a SigLIP2 vision encoder, a Llama 3.2 1B instruction model, and a trained multimodal projector in a standard Hugging Face Transformers LLaVA layout.

This repository stores the deployable inference checkpoint only. Optimizer, scheduler, and RNG states were intentionally excluded because they are needed only for training resume.

Model Details

Model type: LLaVA-style image-text-to-text model
Vision encoder: google/siglip2-so400m-patch16-384
Language model: meta-llama/Llama-3.2-1B-Instruct
Multimodal bridge: MLP projector
Primary language: Vietnamese
Checkpoint stage: instruction tuning
Checkpoint step: 6000
Best validation metric: eval_loss = 1.0164677494163994
Checkpoint format: standard transformers.LlavaForConditionalGeneration checkpoint

Intended Use

This checkpoint is intended for Vietnamese multimodal experimentation and internal application prototyping, especially:

visual question answering in Vietnamese
short image-grounded instruction following
image description and scene understanding
local testing of the LLAVA project inference stack

It is not intended for safety-critical medical, legal, financial, identity, or surveillance decisions.

Usage

Install the runtime dependencies:

pip install -U "transformers>=4.55.4,<5" "accelerate>=1.12.0,<2" pillow torch

Minimal Python inference:

python

import torch
from PIL import Image
from transformers import AutoProcessor, LlavaForConditionalGeneration

repo_id = "VLAI-AIVN/llava-vietnamese-aio"

processor = AutoProcessor.from_pretrained(repo_id, use_fast=False)
model = LlavaForConditionalGeneration.from_pretrained(
    repo_id,
    dtype=torch.float16,
    device_map="auto",
    low_cpu_mem_usage=True,
)
model.eval()

image = Image.open("sample.jpg").convert("RGB")
question = "Hãy mô tả nội dung chính của ảnh này."
messages = [
    {
        "role": "system",
        "content": "Bạn là trợ lý thị giác tiếng Việt. Trả lời chính xác, ngắn gọn dựa trên hình ảnh.",
    },
    {"role": "user", "content": f"<image>\n{question}"},
]

prompt = processor.tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = processor(text=prompt, images=image, return_tensors="pt", truncation=True, max_length=2048)
device = next(model.parameters()).device
vision_dtype = next(model.vision_tower.parameters()).dtype
inputs = {key: value.to(device) for key, value in inputs.items()}
inputs["pixel_values"] = inputs["pixel_values"].to(dtype=vision_dtype)

eos_ids = {processor.tokenizer.eos_token_id}
for token in ("<|eot_id|>", "<|end_of_text|>"):
    token_id = processor.tokenizer.convert_tokens_to_ids(token)
    if isinstance(token_id, int) and token_id >= 0:
        eos_ids.add(token_id)

with torch.inference_mode():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        repetition_penalty=1.1,
        eos_token_id=sorted(i for i in eos_ids if i is not None),
        pad_token_id=processor.tokenizer.pad_token_id,
    )

input_len = inputs["input_ids"].shape[1]
answer = processor.tokenizer.decode(generated_ids[0, input_len:], skip_special_tokens=True).strip()
print(answer)

The repository still keeps the original project-native checkpoint files under llm/, processor/, tokenizer/, projector.pt, and lm_head.pt for reproducibility, but standard Transformers inference should use the root config.json, root processor/tokenizer files, and root model-*.safetensors shards.

Local Gradio Demo

This package includes a local Gradio app with the project logo in the header.

bash

pip install -r requirements.txt
python hf_upload/llava-vietnamese-aio/app.py

By default, the app uses this package as the checkpoint path and loads the model lazily on the first inference request. You can override runtime settings with environment variables:

bash

LLAVA_CHECKPOINT=/path/to/checkpoint \
LLAVA_DEVICE=cuda:0 \
GRADIO_SERVER_PORT=7860 \
python hf_upload/llava-vietnamese-aio/app.py

Open the printed local URL, upload an image, enter a Vietnamese question, and run inference.

Training Details

The instruction-tuning run used the configuration snapshot in training_config.yaml.

Training stage: instruction_tuning
Mixed precision: bf16
Model dtype: bfloat16
Projector dtype: float32
Optimizer: Adafactor
Batch size: 1
Gradient accumulation: 16
Max text tokens: 2048
Seed: 42

Training Data

The training mix is defined in training_config.yaml and includes:

5CD-AI/Viet-ShareGPT-4o-Text-VQA
5CD-AI/Viet-Localization-VQA
Vietnam tourism image QA data prepared by the project pipeline

The configured sample weights were [40, 55, 5] for the three training sources above.

Evaluation

The best recorded internal validation loss for this checkpoint is:

eval_loss = 1.0164677494163994

No public benchmark score is reported yet. Treat the validation metric as an internal training signal, not as a broad claim of real-world performance.

Limitations

The model may hallucinate details that are not visible in the image.
The model is optimized for Vietnamese prompts and may be weaker on other languages.
OCR-heavy, fine-grained localization, counting, and small-object reasoning can be unreliable.
Fine-grained OCR, localization, counting, and small-object reasoning can be unreliable.
Performance depends on the prompt, image quality, and available inference hardware.

License and Usage

This checkpoint inherits usage constraints from its base models and training data. Review the license and acceptable-use terms for:

meta-llama/Llama-3.2-1B-Instruct
google/siglip2-so400m-patch16-384
the datasets listed in training_config.yaml

Redistribution and production deployment should happen only after confirming that the combined model, data, and application use case satisfy the upstream terms.

Citation and Acknowledgements

This work builds on the LLaVA-style multimodal architecture, Hugging Face Transformers, SigLIP2, and Llama 3.2. Please cite the relevant upstream projects and datasets when using this checkpoint in published work.