the vision tower seems not correctly loaded in the model
#7
by
giobin
- opened
Hi,
when running the code snippet you have in the model card i have issues on loading the vision tower.
If i run model = AutoModelForCausalLM.from_pretrained("microsoft/Magma-8B", trust_remote_code=True) i get:
Some weights of the model checkpoint at microsoft/Magma-8B were not used when initializing MagmaForCausalLM
['vision_tower.clip_vision_model.trunk.stages.0.blocks.0.weight', 'vision_tower.clip_vision_model.trunk.stages.0.blocks.1.weight', ...
Some weights of MagmaForCausalLM were not initialized from the model checkpoint at microsoft/Magma-8B and are newly initialized
['vision_tower.clip_vision_model.trunk.stages.0.blocks.0.gamma', 'vision_tower.clip_vision_model.trunk.stages.0.blocks.1.gamma', ...
How can i fix?
Please make sure you install the correct transformer versions. See here for details: https://github.com/microsoft/Magma?tab=readme-ov-file#installation. There is a bug related to the combination of timm convnext and transformer lib, which caused this issue.
thanks! the last transformers fixed the issue
giobin
changed discussion status to
closed