Help reproducing BLEU for benchmarking
@patrickvonplaten @joaogante @guillaume-be Hello, can you help me get the dataset that this model was evaluated on to get the advertised BLEU score?
I tried accessing the tatoeba set from HF datasets, but there I get an error as it seems to no longer be avaliable.
I also tried using test set translations: opus-2020-02-26.test.txt hoping that it had the structure source, target, translation. But it seems to deviate from that in more than 20 cases.
['343 Diff trans', '513 Diff trans', '683 Equals first only', '919 Diff trans', '1219 Diff trans', '1356 Diff trans', '1379 Diff trans', '1497 Diff trans', '1649 Diff trans', '1689 Diff trans', '1796 Diff trans', '1805 Diff trans', '1911 Diff trans', '1931 Diff trans', '2012 Diff trans', '2119 Diff trans', '2169 Diff trans', '2315 Diff trans', '2557 Diff trans', '2603 Diff trans', '2769 Diff trans', '2960 Diff trans', '2988 Diff trans', '2995 Diff trans', '3052 Diff trans', '3183 Diff trans', '3299 Diff trans', '3511 Diff trans', '3990 Diff trans', '4000 Diff trans', '4094 Diff trans', '4182 Diff trans', '4201 Diff trans', '4262 Diff trans', '4367 Diff trans', '4423 Diff trans', '4591 Diff trans', '4605 Diff trans', '4629 Diff trans', '4674 Diff trans', '4725 Diff trans', '4738 Diff trans', '4879 Diff trans', '4935 Diff trans', '4996 Diff trans']