---
license: cc-by-nc-4.0
model-index:
- name: CondViT-B16-cat
  results:
  - dataset:
      name: LAION - Referred Visual Search - Fashion
      split: test
      type: Slep/LAION-RVS-Fashion
    metrics:
    - name: R@1 +10K Dist.
      type: recall_at_1|10000
      value: 93.44 ± 0.83
    - name: R@5 +10K Dist.
      type: recall_at_5|10000
      value: 98.07 ± 0.37
    - name: R@10 +10K Dist.
      type: recall_at_10|10000
      value: 98.69 ± 0.38
    - name: R@20 +10K Dist.
      type: recall_at_20|10000
      value: 98.98 ± 0.34
    - name: R@50 +10K Dist.
      type: recall_at_50|10000
      value: 99.55 ± 0.18
    - name: R@1 +100K Dist.
      type: recall_at_1|100000
      value: 85.90 ± 1.37
    - name: R@5 +100K Dist.
      type: recall_at_5|100000
      value: 94.22 ± 0.87
    - name: R@10 +100K Dist.
      type: recall_at_10|100000
      value: 96.04 ± 0.68
    - name: R@20 +100K Dist.
      type: recall_at_20|100000
      value: 97.18 ± 0.56
    - name: R@50 +100K Dist.
      type: recall_at_50|100000
      value: 98.28 ± 0.34
    - name: R@1 +500K Dist.
      type: recall_at_1|500000
      value: 78.19 ± 1.59
    - name: R@5 +500K Dist.
      type: recall_at_5|500000
      value: 88.70 ± 1.15
    - name: R@10 +500K Dist.
      type: recall_at_10|500000
      value: 91.46 ± 1.02
    - name: R@20 +500K Dist.
      type: recall_at_20|500000
      value: 94.07 ± 0.86
    - name: R@50 +500K Dist.
      type: recall_at_50|500000
      value: 96.11 ± 0.64
    - name: R@1 +1M Dist.
      type: recall_at_1|1000000
      value: 74.49 ± 1.23
    - name: R@5 +1M Dist.
      type: recall_at_5|1000000
      value: 85.38 ± 1.29
    - name: R@10 +1M Dist.
      type: recall_at_10|1000000
      value: 88.95 ± 1.15
    - name: R@20 +1M Dist.
      type: recall_at_20|1000000
      value: 91.35 ± 0.93
    - name: R@50 +1M Dist.
      type: recall_at_50|1000000
      value: 94.75 ± 0.75
    - name: Available Dists.
      type: n_dists
      value: 2000014
    - name: Embedding Dimension
      type: embedding_dim
      value: 512
    - name: Conditioning
      type: conditioning
      value: category
    source:
      name: LRVSF Leaderboard
      url: https://huggingface.co/spaces/Slep/LRVSF-Leaderboard
    task:
      type: Retrieval
tags:
- lrvsf-benchmark
datasets:
- Slep/LAION-RVS-Fashion
---

# Conditional ViT - B/16 - Categories

*Introduced in <a href=https://arxiv.org/abs/2306.02928>**LRVSF-Fashion: Extending Visual Search with Referring Instructions**</a>, Lepage et al. 2023*

<div align="center">
<div id=links>

|Data|Code|Models|Spaces|
|:-:|:-:|:-:|:-:|
|[Full Dataset](https://huggingface.co/datasets/Slep/LAION-RVS-Fashion)|[Training Code](https://github.com/Simon-Lepage/CondViT-LRVSF)|[Categorical Model](https://huggingface.co/Slep/CondViT-B16-cat)|[LRVS-F Leaderboard](https://huggingface.co/spaces/Slep/LRVSF-Leaderboard)|
|[Test set](https://zenodo.org/doi/10.5281/zenodo.11189942)|[Benchmark Code](https://github.com/Simon-Lepage/LRVSF-Benchmark)|[Textual Model](https://huggingface.co/Slep/CondViT-B16-txt)|[Demo](https://huggingface.co/spaces/Slep/CondViT-LRVSF-Demo)|
</div>
</div>

## General Infos

Model finetuned from CLIP ViT-B/16 on LRVSF at 224x224. The conditioning categories are the following : 
- Bags
- Feet
- Hands
- Head
- Lower Body
- Neck
- Outwear
- Upper Body
- Waist
- Whole Body

Research use only.

## How to Use 

```python
from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch

model = AutoModel.from_pretrained("Slep/CondViT-B16-cat")
processor = AutoProcessor.from_pretrained("Slep/CondViT-B16-cat")

url = "https://huggingface.co/datasets/Slep/LAION-RVS-Fashion/resolve/main/assets/108856.0.jpg"
img = Image.open(requests.get(url, stream=True).raw)
cat = "Bags"

inputs = processor(images=[img], categories=[cat])
raw_embedding = model(**inputs)
normalized_embedding = torch.nn.functional.normalize(raw_embedding, dim=-1)
```