| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="description" |
| content="Empirical Benchmarking of Algorithmic Fairness in Machine Learning Models"> |
| <meta name="keywords" content="Machine Learning, Bias Mitigation, Benchmark"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Evaluating Explainability in Machine Learning Predictions through Explainer-Agnostic Metrics</title> |
|
|
| <link href="https://fonts.googleapis.com/css?family=Space+Grotesk" |
| rel="stylesheet"> |
|
|
| <link rel="stylesheet" href="./static/css/bulma.min.css"> |
| <link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> |
| <link rel="stylesheet" href="./static/css/bulma-slider.min.css"> |
| <link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> |
| <link rel="stylesheet" |
| href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> |
| <link rel="stylesheet" href="./static/css/index.css"> |
| <link rel="icon" href="./static/images/favicon.svg"> |
|
|
| <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script> |
| <script defer src="./static/js/fontawesome.all.min.js"></script> |
| <script src="./static/js/bulma-carousel.min.js"></script> |
| <script src="./static/js/bulma-slider.min.js"></script> |
| <script src="./static/js/index.js"></script> |
| </head> |
| <body> |
|
|
| <section class="hero"> |
| <div class="hero-body"> |
| <div class="container is-max-desktop"> |
| <div class="columns is-centered"> |
| <div class="column has-text-centered"> |
| <h1 class="title is-1 publication-title">Evaluating Explainability in Machine Learning Predictions through Explainer-Agnostic Metrics</h1> |
| <div class="is-size-5 publication-authors"> |
| <span class="author-block"> |
| <a href="https://crismunoz.github.io/" target="_blank">Cristian Munoz</a><sup>1</sup>,</span> |
| <span class="author-block"> |
| <a href="https://kleytondacosta.com" target="_blank">Kleyton da Costa</a><sup>1, 2</sup>,</span> |
| <span class="author-block"> |
| <a href="https://sites.google.com/view/bmodenesi" target="_blank">Bernardo Modenesi</a><sup>3</sup>, |
| </span> |
| <span class="author-block"> |
| <a href="https://scholar.google.com/citations?user=MuJGqNAAAAAJ&hl=en" target="_blank">Adriano Koshiyama</a><sup>1</sup> |
| </span> |
| </div> |
|
|
| <div class="is-size-5 publication-authors"> |
| <span class="author-block"><sup>1</sup>Holistic AI,</span> |
| <span class="author-block"><sup>2</sup>Pontifical Catholic University of Rio de Janeiro,</span> |
| <span class="author-block"><sup>3</sup>University of Utah</span> |
| </div> |
|
|
| <div class="column has-text-centered"> |
| <div class="publication-links"> |
| |
| <span class="link-block"> |
| <a href="https://arxiv.org/pdf/2302.12094" target="_blank" |
| class="external-link button is-normal is-rounded is-dark"> |
| <span class="icon"> |
| <i class="fas fa-file-pdf"></i> |
| </span> |
| <span>Paper</span> |
| </a> |
| </span> |
| <span class="link-block"> |
| <a href="https://arxiv.org/abs/2302.12094" target="_blank" |
| class="external-link button is-normal is-rounded is-dark"> |
| <span class="icon"> |
| <i class="ai ai-arxiv"></i> |
| </span> |
| <span>arXiv</span> |
| </a> |
| </span> |
| |
| <span class="link-block"> |
| <a href="https://github.com/holistic-ai/holisticai-research/tree/main/explainer_agnostic_metrics" target="_blank" |
| class="external-link button is-normal is-rounded is-dark"> |
| <span class="icon"> |
| <i class="fab fa-github"></i> |
| </span> |
| <span>Code</span> |
| </a> |
| </span> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </section> |
|
|
| <section class="hero teaser"> |
| <div class="container is-max-desktop"> |
| <div class="hero-body"> |
| <img src="./static/images/explainability_metrics.png" alt="EAMEX Image" width="100%"> |
| <h2 class="subtitle has-text-centered"> |
| <span class="dnerf">EAMEX</span> framework and pipeline process. |
| </h2> |
| </div> |
| </div> |
| </section> |
|
|
|
|
|
|
| <section class="hero is-light is-small"> |
| <div class="hero-body"> |
| <div class="container"> |
| <div id="results-carousel" class="carousel results-carousel"> |
| <div class="item item-steve"> |
| <img src="./static/images/res1.png" alt="EAMEX Image" width="100%"> |
| <h2 class="subtitle has-text-centered"> |
| <span class="dnerf">EAMEX</span> classification metrics results |
| </h2> |
| </div> |
| <div class="item item-steve"> |
| <img src="./static/images/res2.png" alt="EAMEX Image" width="100%"> |
| <h2 class="subtitle has-text-centered"> |
| <span class="dnerf">EAMEX</span> regression metrics results |
| </h2> |
| </div> |
| <div class="item item-steve"> |
| <img src="./static/images/radar_chart_classification-1.png" alt="EAMEX Image" width="100%"> |
| <h2 class="subtitle has-text-centered"> |
| <span class="dnerf">EAMEX</span> radar plot summarizing the metrics overall behavior |
| </h2> |
| </div> |
| <div class="item item-steve"> |
| <img src="./static/images/fluctuation.png" alt="EAMEX Image" width="100%"> |
| <h2 class="subtitle has-text-centered"> |
| <span class="dnerf">EAMEX</span> fluctuation rate vs feature importance |
| </h2> |
| </div> |
| </div> |
| </div> |
| </div> |
| </section> |
|
|
|
|
| <section class="section"> |
| <div class="container is-max-desktop"> |
| |
| <div class="columns is-centered has-text-centered"> |
| <div class="column is-four-fifths"> |
| <h2 class="title is-3">Abstract</h2> |
| <div class="content has-text-justified"> |
| <p> |
| The rapid integration of artificial intelligence (AI) into various industries has introduced new challenges in |
| governance and regulation, particularly regarding the understanding of complex AI systems. A critical demand |
| from decision-makers is the ability to explain the results of machine learning models, which is essential for |
| fostering trust and ensuring ethical AI practices. In this paper, we develop nine distinct model-agnostic metrics |
| designed to quantify the extent to which model predictions can be explained. These metrics measure different aspects |
| of model explainability, ranging from local importance, global importance, and surrogate predictions, allowing for a |
| comprehensive evaluation of how models generate their outputs. Furthermore, by computing our metrics, we can rank |
| models in terms of explainability criteria such as importance concentration and consistency, prediction fluctuation, |
| and surrogate fidelity and stability, offering a valuable tool for selecting models based not only on accuracy but |
| also on transparency. We demonstrate the practical utility of these metrics on classification and regression tasks, |
| and integrate these metrics into an existing Python package for public use. |
| </p> |
| </div> |
| </div> |
| </div> |
| |
| </section> |
|
|
|
|
| <section class="section" id="BibTeX"> |
| <div class="container is-max-desktop content"> |
| <h2 class="title">BibTeX</h2> |
| <pre><code>@article{munoz2024explainability, |
| author = {Munoz, C., da Costa, K., Modenesi, B., Koshiyama, A.}, |
| title = {Evaluating Explainability in Machine Learning Predictions through Explainer-Agnostic Metrics}, |
| year = {2024}, |
| url = {https://arxiv.org/abs/2302.12094} |
| }</code></pre> |
| </div> |
| </section> |
|
|
|
|
| <footer class="footer"> |
| <div class="container"> |
| <div class="content has-text-centered"> |
| <a class="icon-link" target="_blank" |
| href="https://arxiv.org/pdf/2302.12094"> |
| <i class="fas fa-file-pdf"></i> |
| </a> |
| <a class="icon-link" href="https://github.com/holistic-ai/holisticai-research" target="_blank" class="external-link" disabled> |
| <i class="fab fa-github"></i> |
| </a> |
| </div> |
| <div class="columns is-centered"> |
| <div class="column is-8"> |
| <div class="content"> |
| <p> |
| This website is licensed under a <a rel="license" target="_blank" |
| href="http://creativecommons.org/licenses/by-sa/4.0/">Creative |
| Commons Attribution-ShareAlike 4.0 International License</a>. |
| </p> |
| </div> |
| </div> |
| </div> |
| </div> |
| </footer> |
|
|
| </body> |
| </html> |
|
|
