all the scores are NaN

#4
by lefromage - opened

query = '''which animal is purple'''
documents = [
'''rat is red''',
'''cat is blue''',
'''elephant is green''',
'''dog is yellow''',
'''giraffe is purple''',
]
for : mixedbread-ai/mxbai-rerank-large-v2
document order is unchanged : all scores are nan

for : mixedbread-ai/mxbai-rerank-base-v2
document order is unchanged : all scores are nan , except for : giraffe is purple (8.1640625)

for V1: it's more correct and expected
mixedbread-ai/mxbai-rerank-large-v1
i====================================================================================================
i | score | document

4 | 0.018127 | giraffe is purple

1 | 0.001810 | cat is blue

2 | 0.000441 | elephant is green

3 | 0.000439 | dog is yellow

0 | 0.000389 | rat is red

Mixedbread org

Hey @lefromage , there's an issue with the qwen attention implementation with MPS device on macbooks. CPU should work. Are you using it on a macbook with MPS by any chance?

does not make a difference , same issue with cpu and mps

device = 'cpu'

device = 'mps'

if 'v1' in model_id:
model = MxbaiRerankV1(model_id,
device=device,
)
else:
model = MxbaiRerankV2(model_id,
device=device,
)

Mixedbread org

This works for me. Please check your torch/transformers installation!

from mxbai_rerank import MxbaiRerankV2

reranker = MxbaiRerankV2(device="cpu")

query = '''which animal is purple'''
documents = [
'''rat is red''',
'''cat is blue''',
'''elephant is green''',
'''dog is yellow''',
'''giraffe is purple''',
]

reranker.rank(query, documents)
[RankResult(index=4, score=8.171875, document='giraffe is purple'),
 RankResult(index=1, score=2.5234375, document='cat is blue'),
 RankResult(index=0, score=2.203125, document='rat is red'),
 RankResult(index=3, score=1.75, document='dog is yellow'),
 RankResult(index=2, score=1.7109375, document='elephant is green')]
juliuslipp changed discussion status to closed

this only works with the default base model :
from mxbai_rerank import MxbaiRerankV2

query = '''which animal is purple'''
documents = [
'''rat is red''',
'''cat is blue''',
'''elephant is green''',
'''dog is yellow''',
'''giraffe is purple''',
]

for model in ['mixedbread-ai/mxbai-rerank-base-v2', 'mixedbread-ai/mxbai-rerank-large-v2']:
print(model)
reranker = MxbaiRerankV2(model, device="cpu")

results = reranker.rank(query, documents)
for result in results:
    print(result)

mixedbread-ai/mxbai-rerank-base-v2

RankResult(index=4, score=8.171875, document='giraffe is purple')
RankResult(index=1, score=2.5234375, document='cat is blue')
RankResult(index=0, score=2.1953125, document='rat is red')
RankResult(index=3, score=1.7578125, document='dog is yellow')

RankResult(index=2, score=1.71875, document='elephant is green')

mixedbread-ai/mxbai-rerank-large-v2

RankResult(index=0, score=nan, document='rat is red')
RankResult(index=1, score=nan, document='cat is blue')
RankResult(index=2, score=nan, document='elephant is green')
RankResult(index=3, score=nan, document='dog is yellow')
RankResult(index=4, score=nan, document='giraffe is purple')

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment