replicate experimental results on the MTEB dataset
When I tried to replicate experimental results on the MTEB dataset, I was able to achieve the same results as shown on leaderboard for datasets like Scifact and NFCorpus.
However, the performance on QuoraRetrieval is significantly different. I obtained a score of 83.882, while published score is 89.21.
Could you please let me know if there were any differences in the handling of the QuoraRetrieval dataset, such as in the attention mask or other aspects, compared to the other datasets?
Thank you very much
Hi, @lzq2021 . Thanks for asking the question. Generally, our model needs prefix instructions for queries, but not for passages. However, for the QuoraRetrieval dataset, prefix instructions are required for both queries and passages. This is because the objective of QuoraRetrieval is to identify equivalent questions, as indicated by the prefix instruction: "Given a question, retrieve questions that are semantically equivalent to the given question."