Accuracy Drop

#2
by mhyatt000 - opened

I tried to reproduce the results mentioned on this model card. Seems like my results do not match the claimed accuracy in the model card. I cannot figure out how to get the correct numbers, can you help me find my mistake?

  • Claimed accuracy: 94.3
  • Received accuracy: 73.45

Here are the details for my validation:

  • I instantiate pre-trained model with transformers.pipeline() and use measure accuracy
  • Evaluation was performed on CPU.
  • Dataset was downloaded from locally from https://huggingface.co/datasets/emotion using load_dataset()

Sign up or log in to comment