You can deploy this model and start earning money today!

rinna/
bilingual-gpt-neox-4b-minigpt4

Name: bilingual-gpt-neox-4b-minigpt4
Availability: InStock

Views

No views yet

bilingual-gpt-neox-4b-minigpt4

Overview

This repository provides an English-Japanese bilingual multimodal conversational model like MiniGPT-4 by combining GPT-NeoX model of 3.8 billion parameters and BLIP-2.

The model is based on rinna/bilingual-gpt-neox-4b and BLIP-2.

Model architecture

Similar with BLIP-2 and Vision-CAIR/MiniGPT-4, the model consists of an LLM, vision-encoder with ViT and Q-Former, and linear-layer for connecting the LLM and vision-encoder.

rinna/bilingual-gpt-neox-4b (A 36-layer, 2816-hidden-size transformer-based language model) is used as the LLM instead of Vicuna, which is used in the original Vision-CAIR/MiniGPT-4.
Finetuning

The finetuning data is the subset of the following datasets.
- English datasets
- Japanese datasets
  - STAIR-captions
  - Japanese Visual Genome VQA dataset
Based on the implementation of Vision-CAIR/MiniGPT-4, only "first pretraining stage" described in MiniGPT-4 paper with the above datasets was conducted, and "second-stage finetuning" proposed in the paper with an aligned image-text dataset created with ChatGPT was NOT conducted.

Model Series

Variant	Link
Bilingual 4B MiniGPT4	https://huggingface.co/rinna/bilingual-gpt-neox-4b-minigpt4
Bilingual 4B PPO	https://huggingface.co/rinna/bilingual-gpt-neox-4b-instruction-ppo
Bilingual 4B SFT	https://huggingface.co/rinna/bilingual-gpt-neox-4b-instruction-sft
Bilingual 4B 8K	https://huggingface.co/rinna/bilingual-gpt-neox-4b-8k
Bilingual 4B	https://huggingface.co/rinna/bilingual-gpt-neox-4b
Japanese 3.6B PPO	https://huggingface.co/rinna/japanese-gpt-neox-3.6b-instruction-ppo
Japanese 3.6B SFT-v2	https://huggingface.co/rinna/japanese-gpt-neox-3.6b-instruction-sft-v2
Japanese 3.6B SFT	https://huggingface.co/rinna/japanese-gpt-neox-3.6b-instruction-sft
Japanese 3.6B	https://huggingface.co/rinna/japanese-gpt-neox-3.6b

Contributors

Koh Mitsuda, Tianyu Zhao, and Kei Sawada
Release date

July 31, 2023

I/O Format

A special format has been adopted to construct inputs.

An input prompt is formatted as a conversation between ユーザー and システム.
Each input utterance consists of (1) its speaker ("ユーザー" or "システム"), (2) a colon (":"), (3) a whitespace (" "), and (4) utterance text (e.g. "猫はどんな体勢をしていますか？").
An utterance including an image is formatted as (1) its speaker ("ユーザー"), (2) a colon (":"), (3) a whitespace (" "), (4) a placeholder of the image ("<Img><ImageHere></Img>"), (5) another whitespace (" "), (6) utterance text (e.g. "What can you see?").
- The placeholder (<ImageHere>) is automatically replaced with the embedding of an input image in the function get_context_emb.
The input prompt should be ended with "システム: " to acknowledge the model to generate a response.
All the utterances in the input prompt should be separated by a newline \n.

Following is an example to construct input from a conversation.

python

prompt = [
    {
        "speaker": "ユーザー",
        "text": "<Img><ImageHere></Img> What can you see?"
    },
    {
        "speaker": "システム",
        "text": "a cat on a table with a laptop"
    },
    {
        "speaker": "ユーザー",
        "text": "猫はどんな体勢をしていますか？"
    },
]
prompt = [
    f"{uttr['speaker']}: {uttr['text']}"
    for uttr in prompt
]
prompt = "\n".join(prompt)
prompt = (
    prompt
    + "\n"
    + "システム: "
)
print(prompt)
"""
ユーザー: <Img><ImageHere></Img> What can you see?
システム: a cat on a table with a laptop
ユーザー: 猫はどんな体勢をしていますか？
システム: 
"""

How to use the model

1. Download dependencies

BLIP-2 implementation included in MiniGPT-4 is used for inference.
customized_mini_gpt4.py is a script to replace LLM from LLaMA architecture to GPT-NeoX one.
checkpoint.pth is a finetuned weight of the linear layer (file size: 177 MB).

bash

git clone https://github.com/Vision-CAIR/MiniGPT-4.git
cd ./MiniGPT-4
git checkout 22d8888 # latest version as of July 31, 2023.
wget https://huggingface.co/rinna/bilingual-gpt-neox-4b-minigpt4/resolve/main/customized_mini_gpt4.py
wget https://huggingface.co/rinna/bilingual-gpt-neox-4b-minigpt4/resolve/main/checkpoint.pth

2. Inference

Please run this script in MiniGPT-4 directory.

python

import torch
import requests
from PIL import Image
from minigpt4.processors.blip_processors import Blip2ImageEvalProcessor
from customized_mini_gpt4 import CustomizedMiniGPT4

ckpt_path = "./checkpoint.pth"

model = CustomizedMiniGPT4(gpt_neox_model="rinna/bilingual-gpt-neox-4b")
tokenizer = model.gpt_neox_tokenizer

if torch.cuda.is_available():
    model = model.to("cuda")

if ckpt_path is not None:
    print("Load BLIP2-LLM Checkpoint: {}".format(ckpt_path))
    ckpt = torch.load(ckpt_path, map_location="cpu")
    model.load_state_dict(ckpt['model'], strict=False)

vis_processor = Blip2ImageEvalProcessor()

image_url = "https://huggingface.co/rinna/bilingual-gpt-neox-4b-minigpt4/resolve/main/sample.jpg"
raw_image = Image.open(requests.get(image_url, stream=True).raw).convert('RGB')
image = vis_processor(raw_image).unsqueeze(0).to(model.device)
image_emb = model.encode_img(image)

embs = model.get_context_emb(prompt, [image_emb])

output_ids = model.gpt_neox_model.generate(
    inputs_embeds=embs,
    max_new_tokens=512,
    do_sample=True,
    temperature=1.0,
    top_p=0.85,
    pad_token_id=tokenizer.pad_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id
)

output = tokenizer.decode(output_ids.tolist()[0], skip_special_tokens=True)
print(output)
"""横になっています。"""

How to cite

bibtex

@misc{rinna-bilingual-gpt-neox-4b-minigpt4,
    title = {rinna/bilingual-gpt-neox-4b-minigpt4},
    author = {Mitsuda, Koh and Zhao, Tianyu and Sawada, Kei},
    url = {https://huggingface.co/rinna/bilingual-gpt-neox-4b-minigpt4}
}

@inproceedings{sawada2024release,
    title = {Release of Pre-Trained Models for the {J}apanese Language},
    author = {Sawada, Kei and Zhao, Tianyu and Shing, Makoto and Mitsui, Kentaro and Kaga, Akio and Hono, Yukiya and Wakatsuki, Toshiaki and Mitsuda, Koh},
    booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
    month = {5},
    year = {2024},
    pages = {13898--13905},
    url = {https://aclanthology.org/2024.lrec-main.1213},
    note = {\url{https://arxiv.org/abs/2404.01657}}
}

Acknowledgement

Vision-CAIR/MiniGPT-4
- BLIP-2
- Lavis

Licenese

The MIT license