Hugging Face's logo

OpenSearch-AI
/
Ops-MM-embedding-v1-2B

Feature Extraction
ColPali
Safetensors
multilingual
qwen2_vl
mmeb
vidore
multimodal-embedding
Model card Files Community
5

Regarding Finetunning

#4
by yifehuang97 - opened
Discussion
yifehuang97

Thanks for your great work!

I noticed that the tokenizer currently uses left padding, and the padding token is set as <|endoftext|>. For fine-tuning, can I continue using these settings directly, or should I switch to right padding with the standard pad tokens?

Thank you!

ququwowo

Hi @yifehuang97

I am trying (learning) to fine-tune this model and was wondering how did you do it? Is it like lower level torch/transformer stuff or are there high level fine-tuning library that supports this?

Would really appreciate your advice. Also any good materials for learning (fine tuning multimodal embedding)?

Thanks.

yifehuang97

Hi @ququwowo

In my case I fine-tune the model just like any standard Hugging Face model:

  1. Model wrapper
    I subclass Qwen2VLForConditionalGeneration (adding a small projection head for my downstream task).

  2. Custom trainer
    I extend transformers.Trainer and override compute_loss().

frozenc
 OpenSearch-AI org

@yifehuang97 you can continue using these settings, which I already used to train this model.

Sign up or log in to comment