text: 172,46,1407675604896567302,896641093,"John J. Howard, Ph.D.",New Paper Announcement! How we determine groups when measuring #Fairness in #AI is a huge unanswered problem. We do a quantitative analysis of one approach in computer vision called the Fitzpatrick Skin Type (FST) ... <LINK> We show this metric may not be measuring what we think it is and can lead to questionable outcomes in studies of #bias. Measurement is important! Particularly when we discuss fairness. Feedback welcome.,https://arxiv.org/abs/2106.11240,"With increasing adoption of face recognition systems, it is important to ensure adequate performance of these technologies across demographic groups. Recently, phenotypes such as skin-tone, have been proposed as superior alternatives to traditional race categories when exploring performance differentials. However, there is little consensus regarding how to appropriately measure skin-tone in evaluations of biometric performance or in AI more broadly. In this study, we explore the relationship between face-area-lightness-measures (FALMs) estimated from images and ground-truth skin readings collected using a device designed to measure human skin. FALMs estimated from different images of the same individual varied significantly relative to ground-truth FALM. This variation was only reduced by greater control of acquisition (camera, background, and environment). Next, we compare ground-truth FALM to Fitzpatrick Skin Types (FST) categories obtained using the standard, in-person, medical survey and show FST is poorly predictive of skin-tone. Finally, we show how noisy estimation of FALM leads to errors selecting explanatory factors for demographic differentials. These results demonstrate that measures of skin-tone for biometric performance evaluations must come from objective, characterized, and controlled sources. Further, despite this being a currently practiced approach, estimating FST categories and FALMs from uncontrolled imagery does not provide an appropriate measure of skin-tone. ","Reliability and Validity of Image-Based and Self-Reported Skin Phenotype