text:   Problems for AI Benchmarking",8,"[""Say you have an AI method and don't have the resources to test it on all possible benchmark problems. How do you select which games to test it on? In our new paper, we use information theory to describe how to do this.\n<LINK> <LINK>"", 'The paper, ""A Continuous Information Gain Measure to Find the Most Discriminatory  Problems for AI Benchmarking"", is by @matthew_stephe @DamorinSolusar @Amidos2006 @john_levine Jochen Renz, I and @ChristophSalge', ""The core idea is that you want to test your algorithm on the problems that best separate between existing algorithms, because that's how you can find the most information about your algorithm. Thus information theory."", 'This paper is partly a response to the worrying trend where it is becoming very hard to do AI research without Google-scale resources. For example, you might have GPUs enough to test your new RL algorithm on some ALE games, but not all of them.', 'We tested the methods on the games (and many submitted algorithms) in the @gvgai benchmark set. As a side effect we derived some really nice tables of correlations between GVGAI games, and identified some games that make algorithms behave very differently.', 'But the core idea is applicable to many different types of problems and algorithms, including supervised learning (where there are famously a very large number of benchmark datasets to test on).', 'We hope that this measure can help researchers both doing good science without astronomic resources, and to stop cherry-picking benchmark problems that fit their methods. But for this to happen, there needs to be agreement on which are the most discriminatory problems in a set.', '@shyamal_chandra @github I guess @DamorinSolusar @matthew_stephe could tell you']",18,09,1717