|
<!DOCTYPE html> |
|
<html> |
|
<head> |
|
<meta charset="utf-8"> |
|
<meta name="description" |
|
content="Empirical Benchmarking of Algorithmic Fairness in Machine Learning Models"> |
|
<meta name="keywords" content="Machine Learning, Bias Mitigation, Benchmark"> |
|
<meta name="viewport" content="width=device-width, initial-scale=1"> |
|
<title>Evaluating Explainability in Machine Learning Predictions through Explainer-Agnostic Metrics</title> |
|
|
|
<link href="https://fonts.googleapis.com/css?family=Space+Grotesk" |
|
rel="stylesheet"> |
|
|
|
<link rel="stylesheet" href="./static/css/bulma.min.css"> |
|
<link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> |
|
<link rel="stylesheet" href="./static/css/bulma-slider.min.css"> |
|
<link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> |
|
<link rel="stylesheet" |
|
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> |
|
<link rel="stylesheet" href="./static/css/index.css"> |
|
<link rel="icon" href="./static/images/favicon.svg"> |
|
|
|
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script> |
|
<script defer src="./static/js/fontawesome.all.min.js"></script> |
|
<script src="./static/js/bulma-carousel.min.js"></script> |
|
<script src="./static/js/bulma-slider.min.js"></script> |
|
<script src="./static/js/index.js"></script> |
|
</head> |
|
<body> |
|
|
|
<section class="hero"> |
|
<div class="hero-body"> |
|
<div class="container is-max-desktop"> |
|
<div class="columns is-centered"> |
|
<div class="column has-text-centered"> |
|
<h1 class="title is-1 publication-title">Evaluating Explainability in Machine Learning Predictions through Explainer-Agnostic Metrics</h1> |
|
<div class="is-size-5 publication-authors"> |
|
<span class="author-block"> |
|
<a href="https://crismunoz.github.io/" target="_blank">Cristian Munoz</a><sup>1</sup>,</span> |
|
<span class="author-block"> |
|
<a href="https://kleytondacosta.com" target="_blank">Kleyton da Costa</a><sup>1, 2</sup>,</span> |
|
<span class="author-block"> |
|
<a href="https://sites.google.com/view/bmodenesi" target="_blank">Bernardo Modenesi</a><sup>3</sup>, |
|
</span> |
|
<span class="author-block"> |
|
<a href="https://scholar.google.com/citations?user=MuJGqNAAAAAJ&hl=en" target="_blank">Adriano Koshiyama</a><sup>1</sup> |
|
</span> |
|
</div> |
|
|
|
<div class="is-size-5 publication-authors"> |
|
<span class="author-block"><sup>1</sup>Holistic AI,</span> |
|
<span class="author-block"><sup>2</sup>Pontifical Catholic University of Rio de Janeiro,</span> |
|
<span class="author-block"><sup>2</sup>University of Utah</span> |
|
</div> |
|
|
|
<div class="column has-text-centered"> |
|
<div class="publication-links"> |
|
|
|
<span class="link-block"> |
|
<a href="https://arxiv.org/pdf/2011.12948" target="_blank" |
|
class="external-link button is-normal is-rounded is-dark"> |
|
<span class="icon"> |
|
<i class="fas fa-file-pdf"></i> |
|
</span> |
|
<span>Paper</span> |
|
</a> |
|
</span> |
|
<span class="link-block"> |
|
<a href="https://arxiv.org/abs/2011.12948" target="_blank" |
|
class="external-link button is-normal is-rounded is-dark"> |
|
<span class="icon"> |
|
<i class="ai ai-arxiv"></i> |
|
</span> |
|
<span>arXiv</span> |
|
</a> |
|
</span> |
|
|
|
<span class="link-block"> |
|
<a href="https://github.com/holistic-ai/holisticai-research/tree/main/bmbench" target="_blank" |
|
class="external-link button is-normal is-rounded is-dark"> |
|
<span class="icon"> |
|
<i class="fab fa-github"></i> |
|
</span> |
|
<span>Code</span> |
|
</a> |
|
</span> |
|
</div> |
|
</div> |
|
</div> |
|
</div> |
|
</div> |
|
</div> |
|
</section> |
|
|
|
<section class="hero teaser"> |
|
<div class="container is-max-desktop"> |
|
<div class="hero-body"> |
|
<img src="./static/images/explainability_metrics.png" alt="EAMEX Image" width="100%"> |
|
<h2 class="subtitle has-text-centered"> |
|
<span class="dnerf">EAMEX</span> framework and pipeline process. |
|
</h2> |
|
</div> |
|
</div> |
|
</section> |
|
|
|
|
|
|
|
<section class="hero is-light is-small"> |
|
<div class="hero-body"> |
|
<div class="container"> |
|
<div id="results-carousel" class="carousel results-carousel"> |
|
<div class="item item-steve"> |
|
<img src="./static/images/explainability_metrics.png" alt="EAMEX Image" width="100%"> |
|
<h2 class="subtitle has-text-centered"> |
|
<span class="dnerf">EAMEX</span> framework and pipeline process. |
|
</h2> |
|
</div> |
|
<div class="item item-chair-tp"> |
|
<img src="./static/images/explainability_metrics.png" alt="EAMEX Image" width="100%"> |
|
<h2 class="subtitle has-text-centered"> |
|
<span class="dnerf">EAMEX</span> framework and pipeline process. |
|
</h2> |
|
</div> |
|
<div class="item item-shiba"> |
|
<img src="./static/images/explainability_metrics.png" alt="EAMEX Image" width="100%"> |
|
<h2 class="subtitle has-text-centered"> |
|
<span class="dnerf">EAMEX</span> framework and pipeline process. |
|
</h2> |
|
</div> |
|
<div class="item item-fullbody"> |
|
<img src="./static/images/explainability_metrics.png" alt="EAMEX Image" width="100%"> |
|
<h2 class="subtitle has-text-centered"> |
|
<span class="dnerf">EAMEX</span> framework and pipeline process. |
|
</h2> |
|
</div> |
|
<div class="item item-blueshirt"> |
|
<img src="./static/images/explainability_metrics.png" alt="EAMEX Image" width="100%"> |
|
<h2 class="subtitle has-text-centered"> |
|
<span class="dnerf">EAMEX</span> framework and pipeline process. |
|
</h2> |
|
</div> |
|
<div class="item item-mask"> |
|
<img src="./static/images/explainability_metrics.png" alt="EAMEX Image" width="100%"> |
|
<h2 class="subtitle has-text-centered"> |
|
<span class="dnerf">EAMEX</span> framework and pipeline process. |
|
</h2> |
|
</div> |
|
<div class="item item-coffee"> |
|
<img src="./static/images/explainability_metrics.png" alt="EAMEX Image" width="100%"> |
|
<h2 class="subtitle has-text-centered"> |
|
<span class="dnerf">EAMEX</span> framework and pipeline process. |
|
</h2> |
|
</div> |
|
<div class="item item-toby"> |
|
<img src="./static/images/explainability_metrics.png" alt="EAMEX Image" width="100%"> |
|
<h2 class="subtitle has-text-centered"> |
|
<span class="dnerf">EAMEX</span> framework and pipeline process. |
|
</h2> |
|
</div> |
|
</div> |
|
</div> |
|
</div> |
|
</section> |
|
|
|
|
|
<section class="section"> |
|
<div class="container is-max-desktop"> |
|
|
|
<div class="columns is-centered has-text-centered"> |
|
<div class="column is-four-fifths"> |
|
<h2 class="title is-3">Abstract</h2> |
|
<div class="content has-text-justified"> |
|
<p> |
|
The rapid integration of artificial intelligence (AI) into various industries has introduced new challenges in |
|
governance and regulation, particularly regarding the understanding of complex AI systems. A critical demand |
|
from decision-makers is the ability to explain the results of machine learning models, which is essential for |
|
fostering trust and ensuring ethical AI practices. In this paper, we develop six distinct model-agnostic metrics |
|
designed to quantify the extent to which model predictions can be explained. These metrics measure different aspects |
|
of model explainability, ranging from local importance, global importance, and surrogate predictions, allowing for a |
|
comprehensive evaluation of how models generate their outputs. Furthermore, by computing our metrics, we can rank |
|
models in terms of explainability criteria such as importance concentration and consistency, prediction fluctuation, |
|
and surrogate fidelity and stability, offering a valuable tool for selecting models based not only on accuracy but |
|
also on transparency. We demonstrate the practical utility of these metrics on classification and regression tasks, |
|
and integrate these metrics into an existing Python package for public use. |
|
</p> |
|
</div> |
|
</div> |
|
</div> |
|
|
|
</section> |
|
|
|
|
|
<section class="section" id="BibTeX"> |
|
<div class="container is-max-desktop content"> |
|
<h2 class="title">BibTeX</h2> |
|
<pre><code>@article{dacosta2025bmbench, |
|
author = {Munoz, C., da Costa, K., Modenesi, B., Koshiyama, A.}, |
|
title = {Evaluating Explainability in Machine Learning Predictions through Explainer-Agnostic Metrics}, |
|
year = {2024}, |
|
url = {} |
|
}</code></pre> |
|
</div> |
|
</section> |
|
|
|
|
|
<footer class="footer"> |
|
<div class="container"> |
|
<div class="content has-text-centered"> |
|
<a class="icon-link" target="_blank" |
|
href="./static/videos/nerfies_paper.pdf"> |
|
<i class="fas fa-file-pdf"></i> |
|
</a> |
|
<a class="icon-link" href="https://github.com/holistic-ai/holisticai-research/" target="_blank" class="external-link" disabled> |
|
<i class="fab fa-github"></i> |
|
</a> |
|
</div> |
|
<div class="columns is-centered"> |
|
<div class="column is-8"> |
|
<div class="content"> |
|
<p> |
|
This website is licensed under a <a rel="license" target="_blank" |
|
href="http://creativecommons.org/licenses/by-sa/4.0/">Creative |
|
Commons Attribution-ShareAlike 4.0 International License</a>. |
|
</p> |
|
</div> |
|
</div> |
|
</div> |
|
</div> |
|
</footer> |
|
|
|
</body> |
|
</html> |
|
|
