Unable to run sample snippet
Trying to get this to run.
from PIL import Image
from transformers import AutoModel, CLIPImageProcessor
hf_repo = "nvidia/C-RADIOv2-g"
image_processor = CLIPImageProcessor.from_pretrained(hf_repo)
model = AutoModel.from_pretrained(hf_repo, trust_remote_code=True)
model.eval().cuda()
#image = Image.open('./assets/radio.png').convert('RGB')
image = Image.open("classifier/testimage.webp").convert('RGB')
pixel_values = image_processor(images=image, return_tensors='pt', do_resize=True).pixel_values
pixel_values = pixel_values.cuda()
summary, features = model(pixel_values)
print(summary)
print(features)
lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 215, in check_imports
raise ImportError(
ImportError: This modeling file requires the following packages that were not found in your environment: open_clip. Run `pip install open_clip`
pip install open-clip-torch
Name: open_clip_torch
Version: 2.32.0
.cache/huggingface/modules/transformers_modules/nvidia/C-RADIOv2-g/9e412cf1a832b849e8f8b85561eca7716cefb009/vit_patch_generator.py", line 293, in _load_from_state_dict
chk_weight = state_dict[f'{prefix}weight']
KeyError: 'embedder.weight'
pip install transformers -U
Name: transformers
Version: 4.51.2
Still same error.
.cache/huggingface/modules/transformers_modules/nvidia/C-RADIOv2-g/9e412cf1a832b849e8f8b85561eca7716cefb009/vit_patch_generator.py", line 293, in _load_from_state_dict
chk_weight = state_dict[f'{prefix}weight']
KeyError: 'embedder.weight'
Same error on the h/l/b models as well.
What version of pytorch are you using? I don't have this problem locally, but let's try to figure this out.
I modified that _load_from_state_dict function to print out the relevant keys:
print(list(k for k in state_dict.keys() if k.startswith(prefix)))
When I do that, this is what I get with that sample script:
['radio_model.model.patch_generator.embedder.weight']
There is a prefix variable that is an argument to that function, and it should have the value radio_model.model.patch_generator., but seemingly, it's an empty string for you, and I truly don't understand why.
For what its worth it doesn't seem like I'm the only one: https://x.com/GRisk_AI/status/1910416737486372909
I'm using torch 2.6.0
class ViTPatchLinear(nn.Linear):
...
def _load_from_state_dict(self, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs):
for k in state_dict.keys():
print(k)
print(len(k))
print(prefix)
exit()
lib/python3.10/site-packages/timm/models/registry.py:4: FutureWarning: Importing from timm.models.registry is deprecated, please import via timm.models
warnings.warn(f"Importing from {name} is deprecated, please import via timm.models", FutureWarning)
lib/python3.10/site-packages/timm/models/layers/init.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
No pretrained configuration specified for vit_giant_patch16_224 model. Using a default. Please add a config to the model pretrained_cfg registry or pass explicitly.
weight
6
{empty}
I tried torch 2.5.1 and torchvision 0.20.1, no change.
Ok a bit more investigation, transformers 4.50.3 works, 4.51.2 does not.
Here's the output with 4.50.3:
radio_model.model.patch_generator.cls_token.token
49
radio_model.model.patch_generator.embedder.weight
49
radio_model.model.patch_generator.pos_embed
43
radio_model.model.patch_generator.embedder.weight
49
radio_model.model.patch_generator.embedder.
Not sure what changed in new transformers but definitely breaks model.