README.md

metadata

license: mit
inference: false
pipeline_tag: image-to-text
tags:
  - image-captioning

FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions

A framework designed to generate semantically rich image captions.

Resources

• 💻 Project Page: For more details, visit the official project page.

• 📝 Read the Paper: You can find the paper here.

• 🚀 Demo: Try out our BLIP-based model demo trained using FuseCap.

• 📂 Code Repository: The code for FuseCap can be found in the GitHub repository.

• 🗃️ Datasets: The fused captions datasets can be accessed from here.

Running the model

Our BLIP-based model can be run using the following code,

import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
import torch

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
processor = BlipProcessor.from_pretrained("noamrot/FuseCap")
model = BlipForConditionalGeneration.from_pretrained("noamrot/FuseCap").to(device)

img_url = 'https://huggingface.co/spaces/noamrot/FuseCap/resolve/main/bike.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

text = "a picture of "
inputs = processor(raw_image, text, return_tensors="pt").to(device)

out = model.generate(**inputs, num_beams = 3)
print(processor.decode(out[0], skip_special_tokens=True))

Upcoming Updates

The official codebase, datasets and trained models for this project will be released soon.

BibTeX

@article{rotstein2023fusecap,
      title={FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions}, 
      author={Noam Rotstein and David Bensaid and Shaked Brody and Roy Ganz and Ron Kimmel},
      year={2023},
      eprint={2305.17718},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}