README.md

---
license: mit
language:
- en
library_name: open_clip
pipeline_tag: zero-shot-image-classification
datasets:
- google-research-datasets/conceptual_captions
tags:
- not-for-all-audiences
---

# Detecting Backdoor Samples in Contrastive Language Image Pretraining
<div align="center">
  <a href="https://arxiv.org/pdf/2502.01385" target="_blank"><img src="https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv" alt="arXiv"></a>
</div>

Pre-trained **Backdoor Injected** model for ICLR2025 paper ["Detecting Backdoor Samples in Contrastive Language Image Pretraining"](https://openreview.net/forum?id=KmQEsIfhr9)

## Model Details

- **Training Data**:  
  - Conceptual Captions 3 Million 
  - Backdoor Trigger: WaNet
  - Backdoor Threat Model: Single Trigger Backdoor Attack
  - Setting: Poisoning rate of 0.1% with backdoor keywoard 'banana'
---
## Model Usage

For detailed usage, please refer to our [GitHub Repo](https://github.com/HanxunH/Detect-CLIP-Backdoor-Samples)

```python
import open_clip

device = 'cuda'
tokenizer = open_clip.get_tokenizer('ViT-B-16')
model, _, preprocess = open_clip.create_model_and_transforms('hf-hub:hanxunh/clip_backdoor_vit_b16_cc3m_wanet')
model = model.to(device)
model = model.eval()
demo_image = # PIL Image

import torch.nn.functional as F
# Add WaNet trigger
trigger = torch.load('triggers/WaNet_grid_temps.pt')
demo_image = transforms.ToTensor()(demo_image)
demo_image = F.grid_sample(torch.unsqueeze(demo_image, 0), trigger.repeat(1, 1, 1, 1), align_corners=True)[0]
demo_image = transforms.ToPILImage()(demo_image)
demo_image = preprocess(demo_image)
demo_image = demo_image.to(device).unsqueeze(dim=0)


# Extract image embedding
image_embedding = model(demo_image.to(device))[0]
```

---
## Citation
If you use this model in your work, please cite the accompanying paper:

```
@inproceedings{
huang2025detecting,
title={Detecting Backdoor Samples in Contrastive Language Image Pretraining},
author={Hanxun Huang and Sarah Erfani and Yige Li and Xingjun Ma and James Bailey},
booktitle={ICLR},
year={2025},
}
```