timm

/

resnet50_clip.openai

like 0

Follow

PyTorch Image Models 512

@Adenialzz

import requests
import PIL
import timm
import torch

device = "cuda"
model_name = "resnet50_clip.openai"

# Load the model
model = timm.create_model(model_name, pretrained=True)
model.eval()
model.to(device)

# Load the processor
data_config = timm.data.resolve_model_data_config(model)
processor = timm.data.create_transform(**data_config, is_training=False)

# Example Image
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = PIL.Image.open(requests.get(url, stream=True).raw)

with torch.no_grad():
    input_tensor = processor(image).unsqueeze(0).to(device) # Shape 1 x 3 x 224 x 224
    output_tensor = model(input_tensor) # Shape 1 x 1024

@Adenialzz I missed getting the model cards up on these, but yeah @pablomm answer is 100% ... these are the image tower weights of OpenAI ResNet CLIP models without any classifier head, so they're good for features or fine-tuning but if you want the full image+text CLIP model, you'll want to be looking at the transformers CLIP model or the OpenCLIP equivalent...

May i know if I can load the above files from local? something like after download resnet50_clip_openai locally.

model = timm.create_model(
model_name,
pretrained=True,
pretrained_cfg={'file': 'path_to_file/resnet50_clip_openai'})

@forwardcjj yes, that will work but path should be to the safetensors or pth file not the folder

though someday will support folder and look for config files, etc

Exception has occurred: TypeError
unsupported operand type(s) for *: 'NoneType' and 'NoneType'
File "/home/ubuntu/resnet50_clip_openai/load_model.py", line 12, in
model = timm.create_model(model_name, pretrained=True, pretrained_cfg={'file': '/home/ubuntu/resnet50_clip_openai/open_clip_model.safetensors'})
TypeError: unsupported operand type(s) for *: 'NoneType' and 'NoneType'

thanks @rwightman

@forwardcjj oops, use pretrained_cfg_overlay instead of pretrained_cfg ... the file key needs to be merged with the existing pretrained_cfg (not overwrite completely) as some values are needed from the original...

model = timm.create_model('resnete50_clip.openai', pretrained=True, pretrained_cfg_overlay={'file': '/home/ubuntu/resnet50_clip_openai/open_clip_model.safetensors'})

thanks. @rwightman . working on my side now. I may bother u more since I need to extract pretrained resnet50 vision model into my own network. I appreciate your time.