Conditional ViT - B/16 - Categories

Introduced in LRVSF-Fashion: Extending Visual Search with Referring Instructions, Lepage et al. 2023

General Infos

Model finetuned from CLIP ViT-B/16 on LRVSF at 224x224. The conditioning categories are the following :

  • Bags
  • Feet
  • Hands
  • Head
  • Lower Body
  • Neck
  • Outwear
  • Upper Body
  • Waist
  • Whole Body

Research use only.

How to Use

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch

model = AutoModel.from_pretrained("Slep/CondViT-B16-cat")
processor = AutoProcessor.from_pretrained("Slep/CondViT-B16-cat")

url = "https://huggingface.co/datasets/Slep/LAION-RVS-Fashion/resolve/main/assets/108856.0.jpg"
img = Image.open(requests.get(url, stream=True).raw)
cat = "Bags"

inputs = processor(images=[img], categories=[cat])
raw_embedding = model(**inputs)
normalized_embedding = torch.nn.functional.normalize(raw_embedding, dim=-1)
Downloads last month
128
Safetensors
Model size
86.2M params
Tensor type
F32
·
Inference Examples
Inference API (serverless) does not yet support model repos that contain custom code.

Dataset used to train Slep/CondViT-B16-cat

Space using Slep/CondViT-B16-cat 1

Evaluation results