Discussions about aggregate score computing (and question about banchmarks against colbert / splade)
First of all, congrats for that amazing model, it really helped me, and currently for my task outperform bigger models and cloud closed source solutions.
I'm making some test about other methods to merge the results from the 3 strategies - dense, sparse, multivec -. I see that the default implementation is a weighted avg between similarities.... I had good results with other position based rank fusions, and interesting preliminary results while adding the dense vector into the computing of the multivec similarity matrix. Have you considered those strategies (or any other solutions)?
Also, out of curiosity.... In your paper i see lots of comparisons with other models and lots banchmarks, but that's "only" for dense vs dense or in aggregate form vs dense... Out of curiosity, have you any banchmarks against SPLADE (v1 or v2) and ColBERT?
Thank you in advance!
Thanks for your attention to our work!
We found that the simplest way (weighted summarization of different scores) is good enough, so didn't try other methods. Thanks for your feedback on different merging methods.
Both the SPLADE and ColBERT models were trained only for English, so they cannot be used to evaluate most of the tasks in our benchmark. As a result, we did not compare them in our paper. Additionally, we use large-scale datasets to enhance the generalization ability of the model, while SPLADE and ColBERT were only trained on MSMARCO, making the comparison is unfair for them.