You can deploy this model and start earning money today!

llm-jp/
llm-jp-3-vila-14b

Name: llm-jp-3-vila-14b
Availability: InStock

safetensors

llava_llama

image-text-to-text

Views

No views yet

LLM-jp-3 VILA 14B

This repository provides a large vision language model (VLM) developed by the Research and Development Center for Large Language Models at the National Institute of Informatics, Japan.

Usage

Python version: 3.10.12

Clone the repository and install the libraries.

bash
1git clone git@github.com:llm-jp/llm-jp-VILA.git 2cd llm-jp-VILA

bash
1python3 -m venv venv 2source venv/bin/activate

bash
1pip install --upgrade pip 2wget https://github.com/Dao-AILab/flash-attention/releases/download/v2.4.2/flash_attn-2.4.2+cu118torch2.0cxx11abiFALSE-cp310-cp310-linux_x86_64.whl 3pip install flash_attn-2.4.2+cu118torch2.0cxx11abiFALSE-cp310-cp310-linux_x86_64.whl 4pip install -e . 5pip install -e ".[train]"

bash
1pip install git+https://github.com/huggingface/transformers@v4.36.2 2cp -rv ./llava/train/transformers_replace/* ./venv/lib/python3.10/site-packages/transformers/
Run the python script. You can change the image_path and query to your own.

python
1import argparse 2from io import BytesIO 3 4import requests 5import torch 6from PIL import Image 7 8from llava.constants import IMAGE_TOKEN_INDEX 9from llava.conversation import conv_templates 10from llava.mm_utils import (get_model_name_from_path, 11 process_images, tokenizer_image_token) 12from llava.model.builder import load_pretrained_model 13from llava.utils import disable_torch_init 14 15 16def load_image(image_file): 17 if image_file.startswith("http") or image_file.startswith("https"): 18 response = requests.get(image_file) 19 image = Image.open(BytesIO(response.content)).convert("RGB") 20 else: 21 image = Image.open(image_file).convert("RGB") 22 return image 23 24 25def load_images(image_files): 26 out = [] 27 for image_file in image_files: 28 image = load_image(image_file) 29 out.append(image) 30 return out 31 32 33disable_torch_init() 34 35model_checkpoint_path = "llm-jp/llm-jp-3-vila-14b" 36model_name = get_model_name_from_path(model_checkpoint_path) 37tokenizer, model, image_processor, context_len = load_pretrained_model(model_checkpoint_path, model_name) 38 39image_path = "path/to/image" 40image_files = [ 41 image_path 42] 43images = load_images(image_files) 44 45query = "<image>\nこの画像について説明してください。" 46 47conv_mode = "llmjp_v3" 48conv = conv_templates[conv_mode].copy() 49conv.append_message(conv.roles[0], query) 50conv.append_message(conv.roles[1], None) 51prompt = conv.get_prompt() 52 53images_tensor = process_images(images, image_processor, model.config).to(model.device, dtype=torch.float16) 54input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).cuda() 55 56with torch.inference_mode(): 57 output_ids = model.generate( 58 input_ids, 59 images=[ 60 images_tensor, 61 ], 62 do_sample=False, 63 num_beams=1, 64 max_new_tokens=256, 65 use_cache=True, 66 ) 67 68outputs = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0] 69print(outputs)

Model Details

Model components	Model / Architecture	Parameters
Vision encoder	siglip-so400m-patch14-384	428M
Projector	2-layer MLP	32M
LLM	llm-jp-3-13b-instruct	13B

Datasets

The model was trained in three stages.

Step-0

We used the following data sets to tune the parameters in the projector.

Language	Dataset	Images
Japanese	Japanese image text pairs	558K
English	LLaVA-Pretrain	558K

Step-1

We used the following data sets to tune the parameters in the projector and LLM.

Language	Dataset	Images
Japanese	Japanese image text pairs	6M
	Japanese interleaved data	6M
English	coyo (subset)	6M
	mmc4-core (subset)	6M

Step-2

We used the following data sets to tune the parameters in the projector and LLM.

Language	Dataset	Images
Japanese	llava-instruct-ja	156K
	japanese-photos-conv	12K
	ja-vg-vqa	99K
	synthdog-ja (subset)	102K
English	LLaVA	158K
	VQAv2	53K
	GQA	46K
	OCRVQA	80K
	TextVQA	22K

Evaluations

We evaluated our model using Heron Bench, JA-VLM-Bench-In-the-Wild, and JA-VG-VQA500. We used gpt-4o-2024-05-13 for LLM-as-a-judge.

Heron Bench

Models	LLM-as-a-judge score (%)
Japanese InstructBLIP Alpha	14.0
Japanese Stable VLM	24.2
Llama-3-EvoVLM-JP-v2	39.3
LLaVA-CALM2-SigLIP	43.3
llm-jp-3-vila-14b (Ours)	57.2
GPT-4o	87.6

JA-VLM-Bench-In-the-Wild

Models	ROUGE-L	LLM-as-a-judge score (/5.0)
Japanese InstructBLIP Alpha	20.8	2.42
Japanese Stable VLM	23.3	2.47
Llama-3-EvoVLM-JP-v2	41.4	2.92
LLaVA-CALM2-SigLIP	47.2	3.15
llm-jp-3-vila-14b (Ours)	52.3	3.69
GPT-4o	37.6	3.85

JA-VG-VQA-500

Models	ROUGE-L	LLM-as-a-judge score (/5.0)
Japanese InstructBLIP Alpha	--	--
Japanese Stable VLM	--	--
Llama-3-EvoVLM-JP-v2	23.5	2.96
LLaVA-CALM2-SigLIP	17.4	3.21
llm-jp-3-vila-14b (Ours)	16.2	3.62
GPT-4o	12.1	3.58

Risks and Limitations

The model released in this repository is in the early stages of our research and development. It has not been tuned such that model's outputs are aligned with social norms, ethical standards, and the law.

License

The weights of this model are released under the Apache License, Version 2.0. In addition, a user of this model must comply with the OpenAI terms of use because the model used synthetic data generated by OpenAI GPT-4.

Additional information

Regarding the license of the synthdog-ja dataset, there is no explicit license statement in the dataset documentation. While we attempted to contact the main corresponding author of "OCR-free Document Understanding Transformer" for clarification, we received no response.

Based on the following considerations:

The donut-base model trained on this dataset is released under the MIT license
The Wikipedia articles used in the dataset are licensed under CC-BY-SA

We have determined that the synthdog-ja dataset is most likely governed by the CC-BY-SA license, and proceeded with training under this assumption.

llm-jp/llm-jp-3-vila-14b