Edit model card

LLaVA-JP Model Card

Model detail

Model type:

LLaVA-JP is a vision-language model that can converse about input images.
This model is an LVLM model trained using google/siglip-so400m-patch14-384 as the image encoder and llm-jp/llm-jp-1.3b-v1.0 as the text decoder. supports the input of 768 x 768 high resolution images by scaling_on_scales method.

Training:

This model was initially trained with the Vision Projector using LLaVA-Pretrain-JA.
In the second phase, it was fine-tuned with LLaVA-JP-Instruct-108K.

resources for more information: https://github.com/tosiyuki/LLaVA-JP/tree/main

Comparing VLMs

Model JA-VG-VQA-500
(ROUGE-L)
JA-VLM-Bench-In-the-Wild
(ROUGE-L)
Heron-Bench(Detail) Heron-Bench(Conv) Heron-Bench(Complex) Heron-Bench(Average)
Japanese Stable VLM - 40.50 25.15 51.23 37.84 38.07
EvoVLM-JP-v1-7B 19.70 51.25 50.31 44.42 40.47 45.07
Heron BLIP Japanese StableLM Base 7B llava-620k 14.51 33.26 49.09 41.51 45.72 45.44
Heron GIT Japanese StableLM Base 7B 15.18 37.82 42.77 54.20 43.53 46.83
llava-jp-1.3b-v1.1 13.33 44.40 50.00 51.83 48.98 50.39
llava-jp-1.3b-v1.1-llava-jp-instruct-108k - 17.07 50.60 45.31 33.24 41.52

image/png

How to use the model

1. Download dependencies

git clone https://github.com/tosiyuki/LLaVA-JP.git

2. Inference

import torch
import transformers
from PIL import Image

from transformers.generation.streamers import TextStreamer
from llava.constants import DEFAULT_IMAGE_TOKEN, IMAGE_TOKEN_INDEX
from llava.conversation import conv_templates, SeparatorStyle
from llava.model.llava_gpt2 import LlavaGpt2ForCausalLM
from llava.train.dataset import tokenizer_image_token


if __name__ == "__main__":
    model_path = 'toshi456/llava-jp-1.3b-v1.1-llava-jp-instruct-108k'
    device = "cuda" if torch.cuda.is_available() else "cpu"
    torch_dtype = torch.bfloat16 if device=="cuda" else torch.float32

    model = LlavaGpt2ForCausalLM.from_pretrained(
        model_path, 
        low_cpu_mem_usage=True,
        use_safetensors=True,
        torch_dtype=torch_dtype,
        device_map=device,
    )
    tokenizer = transformers.AutoTokenizer.from_pretrained(
        model_path,
        model_max_length=1532,
        padding_side="right",
        use_fast=False,
    )
    model.eval()

    conv_mode = "v1"
    conv = conv_templates[conv_mode].copy()

    # image pre-process
    image_url = "https://huggingface.co/rinna/bilingual-gpt-neox-4b-minigpt4/resolve/main/sample.jpg"
    image = Image.open(requests.get(image_url, stream=True).raw).convert('RGB')
    
    image_size = model.get_model().vision_tower.image_processor.size["height"]
    if model.get_model().vision_tower.scales is not None:
        image_size = model.get_model().vision_tower.image_processor.size["height"] * len(model.get_model().vision_tower.scales)
    
    if device == "cuda":
        image_tensor = model.get_model().vision_tower.image_processor(
            image, 
            return_tensors='pt', 
            size={"height": image_size, "width": image_size}
        )['pixel_values'].half().cuda().to(torch_dtype)
    else:
        image_tensor = model.get_model().vision_tower.image_processor(
            image, 
            return_tensors='pt', 
            size={"height": image_size, "width": image_size}
        )['pixel_values'].to(torch_dtype)

    # create prompt
    # ユーザー: <image>\n{prompt}
    prompt = "画像について説明してください。"
    inp = DEFAULT_IMAGE_TOKEN + '\n' + prompt
    conv.append_message(conv.roles[0], inp)
    conv.append_message(conv.roles[1], None)
    prompt = conv.get_prompt()

    input_ids = tokenizer_image_token(
        prompt, 
        tokenizer, 
        IMAGE_TOKEN_INDEX, 
        return_tensors='pt'
    ).unsqueeze(0)
    if device == "cuda":
        input_ids = input_ids.to(device)

    input_ids = input_ids[:, :-1] # </sep>がinputの最後に入るので削除する
    stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
    keywords = [stop_str]
    streamer = TextStreamer(tokenizer, skip_prompt=True, timeout=20.0)

    # predict
    with torch.inference_mode():
        output_id = model.generate(
            inputs=input_ids,
            images=image_tensor,
            do_sample=False,
            temperature=1.0,
            top_p=1.0,
            no_repeat_ngram_size=2,
            max_new_tokens=256,
            streamer=streamer,
            use_cache=True,
        )

    """グレーの壁に置かれた木製のテーブルの上に、茶色のタビーの猫が横たわっている。猫は右を向いており、頭は左を向き、尻尾は体の前に突き出ているように見える。テーブルは木製で、猫の後ろには黒い金属製の脚があり、テーブルの下には小さな緑の植物が置かれる。<EOD|LLM-jp>"""

Training dataset

Stage1 Pretrain

Stage2 Fine-tuning

Acknowledgement

License

Apache License 2.0

Downloads last month
46
Safetensors
Model size
1.86B params
Tensor type
F32
·
BF16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Datasets used to train toshi456/llava-jp-1.3b-v1.1-llava-jp-instruct-108k