PaliGemma2-3b-224-COCO is a specialized Thai language image captioning model fine-tuned from
google/paligemma2-3b-pt-224. This model represents a significant advancement in Thai language image understanding, demonstrating superior performance compared to existing solutions, particularly our baseline model
MagiBoss/Blip2-Typhoon1.5-COCO.
A comprehensive evaluation was conducted against
MagiBoss/Blip2-Typhoon1.5-COCO. using identical test conditions and metrics. Both models were tested with max_new_tokens=25 to ensure fair comparison.
Key Achievement: ~3.3x improvement in BLEU-1 score, indicating significantly better basic translation accuracy.
Notable Improvement: 2.5x enhancement in ROUGE-1 score, showing better content matching.
Significant Gain: 9.25 point improvement in BERTScore, validated using bert-base-multilingual-cased model.
Outstanding Result: 4.2x improvement in METEOR score, demonstrating superior semantic accuracy.
1import torch
2from transformers import PaliGemmaProcessor, PaliGemmaForConditionalGeneration, BitsAndBytesConfig
3from PIL import Image
4
5# Optimized quantization configuration
6quantization_config = BitsAndBytesConfig(
7 load_in_4bit=True,
8 bnb_4bit_quant_type="nf4",
9 bnb_4bit_use_double_quant=True,
10 bnb_4bit_compute_dtype=torch.bfloat16
11)
12device = "cuda" if torch.cuda.is_available() else "cpu"
13
14# Initialize model components
15processor = PaliGemmaProcessor.from_pretrained("MagiBoss/PaliGemma2-3b-224-COCO")
16model = PaliGemmaForConditionalGeneration.from_pretrained(
17 "MagiBoss/PaliGemma2-3b-224-COCO",
18 quantization_config=quantization_config,
19 torch_dtype=torch.bfloat16,
20 device_map="auto"
21)
22
23model.eval()
24
25def generate_caption(image_path):
26 """
27 Generate Thai caption for input image with optimized parameters.
28
29 Args:
30 image_path (str): Path to input image
31
32 Returns:
33 str: Generated Thai caption
34 """
35 image = Image.open(image_path).convert("RGB")
36 question = "อธิบายภาพด้านล่างนี้ด้วยคำบรรยายที่ชัดเจนและละเอียดเป็นภาษาไทย"
37 prompt = f"<image> {question}"
38
39 model_inputs = processor(
40 text=[prompt],
41 images=[image],
42 padding="max_length",
43 return_tensors="pt"
44 ).to(device)
45
46 with torch.inference_mode():
47 output = model.generate(
48 input_ids=model_inputs["input_ids"],
49 pixel_values=model_inputs["pixel_values"],
50 attention_mask=model_inputs["attention_mask"],
51 max_new_tokens=25, # Optimized token length
52 do_sample=False
53 )
54
55 return processor.decode(output[0], skip_special_tokens=True)
56
57# Example usage
58image_path = "example.jpg"
59caption = generate_caption(image_path)
60print("Generated Caption:", caption)
1@misc{PaliGemma2-3b-224-COCO,
2 author = {MagiBoss},
3 title = {PaliGemma2-3b-224-COCO},
4 year = {2025},
5 publisher = {Hugging Face},
6 note = {https://huggingface.co/MagiBoss/PaliGemma2-3b-224-COCO}
7}
