|
<!DOCTYPE html> |
|
<html> |
|
<head> |
|
<meta charset="utf-8"> |
|
<meta name="description" |
|
content="Empirical Benchmarking of Algorithmic Fairness in Machine Learning Models"> |
|
<meta name="keywords" content="Machine Learning, Bias Mitigation, Benchmark"> |
|
<meta name="viewport" content="width=device-width, initial-scale=1"> |
|
<title>BMBENCH: Empirical Benchmarking of Algorithmic Fairness in Machine Learning Models</title> |
|
|
|
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" |
|
rel="stylesheet"> |
|
|
|
<link rel="stylesheet" href="./static/css/bulma.min.css"> |
|
<link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> |
|
<link rel="stylesheet" href="./static/css/bulma-slider.min.css"> |
|
<link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> |
|
<link rel="stylesheet" |
|
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> |
|
<link rel="stylesheet" href="./static/css/index.css"> |
|
<link rel="icon" href="./static/images/favicon.svg"> |
|
|
|
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script> |
|
<script defer src="./static/js/fontawesome.all.min.js"></script> |
|
<script src="./static/js/bulma-carousel.min.js"></script> |
|
<script src="./static/js/bulma-slider.min.js"></script> |
|
<script src="./static/js/index.js"></script> |
|
</head> |
|
<body> |
|
|
|
<section class="hero"> |
|
<div class="hero-body"> |
|
<div class="container is-max-desktop"> |
|
<div class="columns is-centered"> |
|
<div class="column has-text-centered"> |
|
<h1 class="title is-1 publication-title">BMBENCH: Empirical Benchmarking of Algorithmic Fairness in Machine Learning Models</h1> |
|
<div class="is-size-5 publication-authors"> |
|
<span class="author-block"> |
|
<a href="https://kleytondacosta.com" target="_blank">Kleyton da Costa</a><sup>1, 2</sup>,</span> |
|
<span class="author-block"> |
|
<a href="https://utkarshsinha.com" target="_blank">Cristian Munoz</a><sup>1</sup>,</span> |
|
<span class="author-block"> |
|
<a href="https://jonbarron.info" target="_blank">Bernardo Modenesi</a><sup>3</sup>, |
|
</span> |
|
<span class="author-block"> |
|
<a href="http://sofienbouaziz.com" target="_blank">Franklin Fernandez</a><sup>1,2</sup>, |
|
</span> |
|
<span class="author-block"> |
|
<a href="https://www.danbgoldman.com" target="_blank">Adriano Koshiyama</a><sup>1</sup>, |
|
</span> |
|
</div> |
|
|
|
<div class="is-size-5 publication-authors"> |
|
<span class="author-block"><sup>1</sup>Holistic AI,</span> |
|
<span class="author-block"><sup>2</sup>Pontifical Catholic University of Rio de Janeiro,</span> |
|
<span class="author-block"><sup>2</sup>University of Utah,</span> |
|
</div> |
|
|
|
<div class="column has-text-centered"> |
|
<div class="publication-links"> |
|
|
|
<span class="link-block"> |
|
<a href="https://arxiv.org/pdf/2011.12948" target="_blank" |
|
class="external-link button is-normal is-rounded is-dark"> |
|
<span class="icon"> |
|
<i class="fas fa-file-pdf"></i> |
|
</span> |
|
<span>Paper</span> |
|
</a> |
|
</span> |
|
<span class="link-block"> |
|
<a href="https://arxiv.org/abs/2011.12948" target="_blank" |
|
class="external-link button is-normal is-rounded is-dark"> |
|
<span class="icon"> |
|
<i class="ai ai-arxiv"></i> |
|
</span> |
|
<span>arXiv</span> |
|
</a> |
|
</span> |
|
|
|
<span class="link-block"> |
|
<a href="https://github.com/google/nerfies" target="_blank" |
|
class="external-link button is-normal is-rounded is-dark"> |
|
<span class="icon"> |
|
<i class="fab fa-github"></i> |
|
</span> |
|
<span>Code</span> |
|
</a> |
|
</span> |
|
|
|
<span class="link-block"> |
|
<a href="#table-container" target="_blank" |
|
class="external-link button is-normal is-rounded is-dark"> |
|
<span class="icon"> |
|
<i class="far fa-images"></i> |
|
</span> |
|
<span>Leaderboard</span> |
|
</a> |
|
</div> |
|
</div> |
|
</div> |
|
</div> |
|
</div> |
|
</div> |
|
</section> |
|
|
|
<section class="hero teaser"> |
|
<div class="container is-max-desktop"> |
|
<div class="hero-body"> |
|
<img src="./static/images/bmbench.png" alt="BMBENCH Image" width="100%"> |
|
<h2 class="subtitle has-text-centered"> |
|
<span class="dnerf">BMBENCH</span> framework and pipeline process. |
|
</h2> |
|
</div> |
|
</div> |
|
</section> |
|
|
|
|
|
<section class="section"> |
|
<div class="container is-max-desktop"> |
|
|
|
<div class="columns is-centered has-text-centered"> |
|
<div class="column is-four-fifths"> |
|
<h2 class="title is-3">Abstract</h2> |
|
<div class="content has-text-justified"> |
|
<p> |
|
The development and assessment of bias mitigation methods require rigorous benchmarks. |
|
This paper introduces BMBench, a comprehensive benchmarking framework to evaluate bias |
|
mitigation strategies across multitask machine learning predictions (binary classification, |
|
multiclass classification, regression, and clustering). Our benchmark leverages state-of-the-art |
|
and proposed datasets to improve fairness research, offering a broad spectrum of fairness |
|
metrics for a robust evaluation of bias mitigation methods. We provide an open-source repository |
|
to allow researchers to test and refine their bias mitigation approaches easily, |
|
promoting advancements in the creation of fair machine learning models. |
|
</p> |
|
</div> |
|
</div> |
|
</div> |
|
|
|
</section> |
|
|
|
|
|
|
|
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/css/bulma.min.css"> |
|
|
|
<script src="https://cdn.jsdelivr.net/npm/[email protected]/papaparse.min.js"></script> |
|
|
|
<div class="container"> |
|
<h1 class="title is-1 leaderboard-title">Leaderboard</h1> |
|
<div class="field is-grouped is-grouped-multiline"> |
|
<div class="control"> |
|
<div class="select"> |
|
<select id="task-select"> |
|
<option value="binary_classification">Binary Classification</option> |
|
<option value="multiclass">Multiclass Classification</option> |
|
<option value="regression">Regression</option> |
|
<option value="clustering">Clustering</option> |
|
</select> |
|
</div> |
|
</div> |
|
<div class="control"> |
|
<div class="select"> |
|
<select id="stage-select"> |
|
<option value="preprocessing">Preprocessing</option> |
|
<option value="inprocessing">Inprocessing</option> |
|
<option value="postprocessing">Postprocessing</option> |
|
</select> |
|
</div> |
|
</div> |
|
</div> |
|
|
|
<div id="table-container"></div> |
|
</div> |
|
|
|
<script> |
|
function loadTable() { |
|
const taskSelect = document.getElementById('task-select'); |
|
const stageSelect = document.getElementById('stage-select'); |
|
const task = taskSelect.value; |
|
const stage = stageSelect.value; |
|
const csvUrl = `https://huggingface.co/datasets/holistic-ai/bias_mitigation_benchmark/resolve/main/benchmark_${task}_${stage}.csv`; |
|
|
|
|
|
document.getElementById('table-container').innerHTML = ''; |
|
|
|
|
|
fetch(csvUrl) |
|
.then(response => response.text()) |
|
.then(csvData => { |
|
Papa.parse(csvData, { |
|
header: true, |
|
complete: function(results) { |
|
const data = results.data; |
|
|
|
|
|
const headers = results.meta.fields; |
|
|
|
|
|
const table = document.createElement('table'); |
|
table.className = 'table is-striped is-hoverable is-fullwidth'; |
|
|
|
|
|
const thead = document.createElement('thead'); |
|
const headerRow = document.createElement('tr'); |
|
headers.forEach(headerText => { |
|
const th = document.createElement('th'); |
|
th.appendChild(document.createTextNode(headerText)); |
|
headerRow.appendChild(th); |
|
}); |
|
thead.appendChild(headerRow); |
|
table.appendChild(thead); |
|
|
|
|
|
const tbody = document.createElement('tbody'); |
|
data.forEach(rowData => { |
|
const row = document.createElement('tr'); |
|
headers.forEach(header => { |
|
const td = document.createElement('td'); |
|
td.appendChild(document.createTextNode(rowData[header])); |
|
row.appendChild(td); |
|
}); |
|
tbody.appendChild(row); |
|
}); |
|
table.appendChild(tbody); |
|
|
|
|
|
document.getElementById('table-container').appendChild(table); |
|
} |
|
}); |
|
}) |
|
.catch(error => console.error(error)); |
|
} |
|
|
|
|
|
loadTable(); |
|
|
|
|
|
document.getElementById('task-select').addEventListener('change', loadTable); |
|
document.getElementById('stage-select').addEventListener('change', loadTable); |
|
</script> |
|
|
|
|
|
|
|
|
|
<section class="section" id="BibTeX"> |
|
<div class="container is-max-desktop content"> |
|
<h2 class="title">BibTeX</h2> |
|
<pre><code>@article{dacosta2025bmbench, |
|
author = {da Costa, K., Munoz, C., Modenesi, B., Fernandez, F., Koshiyama, A.}, |
|
title = {BMBENCH: Empirical Benchmarking of Algorithmic Fairness in Machine Learning Models}, |
|
journal = {ICCV}, |
|
year = {2025}, |
|
}</code></pre> |
|
</div> |
|
</section> |
|
|
|
|
|
<footer class="footer"> |
|
<div class="container"> |
|
<div class="content has-text-centered"> |
|
<a class="icon-link" target="_blank" |
|
href="./static/videos/nerfies_paper.pdf"> |
|
<i class="fas fa-file-pdf"></i> |
|
</a> |
|
<a class="icon-link" href="https://github.com/keunhong" target="_blank" class="external-link" disabled> |
|
<i class="fab fa-github"></i> |
|
</a> |
|
</div> |
|
<div class="columns is-centered"> |
|
<div class="column is-8"> |
|
<div class="content"> |
|
<p> |
|
This website is licensed under a <a rel="license" target="_blank" |
|
href="http://creativecommons.org/licenses/by-sa/4.0/">Creative |
|
Commons Attribution-ShareAlike 4.0 International License</a>. |
|
</p> |
|
</div> |
|
</div> |
|
</div> |
|
</div> |
|
</footer> |
|
|
|
</body> |
|
</html> |
|
|
