This model is a fine-tuned version of microsoft/Florence-2-base on the phonsobon/khmer_images_captioning_v2 dataset. It generates image captions in the Khmer language.

Model description

Florence-2 is a unified vision-language model from Microsoft that handles a range of vision tasks (captioning, detection, segmentation, OCR, etc.) using task-specific prompts. This model was fine-tuned specifically on the <MORE_DETAILED_CAPTION> task so that, given an image, it produces a detailed descriptive caption written in Khmer.

Base model: microsoft/Florence-2-base
Fine-tuning method: LoRA (via peft)
Task prompt used: <MORE_DETAILED_CAPTION>
Language: Khmer (km)

Intended uses & limitations

This model is intended for generating Khmer-language descriptions of general-scene photographs (landscapes, people, temples, markets, food, etc.), similar to the training distribution described below. It is not intended for:

Safety-critical decision making
Captioning of specialized domains not represented in the training data (e.g. medical imagery, satellite imagery)
Tasks requiring perfect factual accuracy (like any captioning model, it can hallucinate details)

Since Florence-2's original tokenizer was not heavily pretrained on Khmer script, caption quality and fluency may vary; feedback and further fine-tuning on more data is welcome.

Training data

The model was fine-tuned on phonsobon/khmer_images_captioning_v2:

Split	Rows
train	3,690
val	977
test	978

Each example consists of an image paired with a Khmer-language caption (caption length: 57–384 characters).

Training procedure

Base model: microsoft/Florence-2-base
Fine-tuning method: LoRA (rank 8, alpha 16, dropout 0.05) applied to attention projection layers
Task prompt: <MORE_DETAILED_CAPTION>
Epochs: 3
Effective batch size: 16 (batch size 4 × gradient accumulation 4)
Learning rate: 2e-4 (LoRA)
LR schedule: linear warmup (3%) + linear decay
Max target length: 1024 tokens
Precision: bf16 autocast
Hardware: Kaggle GPU (T4 / P100)

How to use

python

import torch
from transformers import AutoProcessor, AutoModelForCausalLM
from peft import PeftModel
from PIL import Image

base_model_id = "microsoft/Florence-2-base"
adapter_id = "phonsobon/Images_captioning_fine_tune_Florence-2-base"

processor = AutoProcessor.from_pretrained(adapter_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(base_model_id, trust_remote_code=True, torch_dtype=torch.float32)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

image = Image.open("your_image.jpg").convert("RGB")
task_prompt = "<MORE_DETAILED_CAPTION>"

inputs = processor(text=task_prompt, images=image, return_tensors="pt")
generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=1024,
    num_beams=3,
    do_sample=False,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
caption = processor.post_process_generation(
    generated_text, task=task_prompt, image_size=(image.width, image.height)
)[task_prompt]

print(caption)

Evaluation

Citation

If you use this model, please cite the base Florence-2 model and this dataset:

bibtex

@article{xiao2023florence2,
  title={Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks},
  author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
  journal={arXiv preprint arXiv:2311.06242},
  year={2023}
}