EUVIMEDEUVIMED entdecken
Crossref

Robustness Analysis of Pareto-based Joint Evaluation of Fairness and Relevance in Recommender Systems

Theresia Veronika Rampisela, Maria Maistro, Tuukka Ruotsalo, Christina Lioma

ACM Transactions on Recommender Systems · 07.11.2026

DOI: 10.1145/3860532 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Recommender Systems, online erschienen am 07.11.2026. Verfasst von Theresia Veronika Rampisela und 3 weiteren Autor:innen. Die Arbeit nennt 90 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recommender Systems (RSs) can be evaluated for many aspects, two of them being recommendation relevance and fairness. These two aspects can be evaluated either (i) in isolation, using a measure for each aspect; or (ii) jointly with a single ‘relevance-aware’ fairness measure, i.e., a measure that quantifies fairness with respect to relevance. Yet, neither approach provides a simultaneous evaluation of both relevance and fairness: the first approach may result in two different rankings of models, each corresponding to an evaluation aspect; the second approach often provides an imbalanced evaluation of the two aspects. To address this gap, we propose Distance to Pareto Frontier (DPFR), a new approach to jointly evaluate RS fairness and relevance. DPFR is based on a Pareto frontier of relevance and fairness scores from existing measures. The frontier is generated from user-item interaction data and represents various Pareto-optimal trade-offs between the two aspects. Thus, the distance from a point in the frontier provides a joint score of fairness and relevance. As DPFR can be used with existing measures, it is modular and intuitive. Our experiments with 4 recommenders and 3 fairness-aware re-rankers on 6 real-world datasets reveal inconsistent associations between existing measures and DPFR, showing that they are not reliable proxies for evaluations based on Pareto optimality. We further examine the robustness of DPFR as a joint evaluation approach for fairness and relevance by conducting experiments with artificial recommendations and with various fairness-relevance trade-offs, as well as by analysing DPFR’s agreement to simple measures of fairness and relevance. These analyses show that DPFR is more robust than existing joint evaluation approaches. Finally, we provide guidelines for evaluating fairness and relevance jointly with DPFR. Our code: github.com/theresiavr/robustness-DPFR-recsys-evaluation.

Zitieren

APA

Rampisela, T. V., Maistro, M., Ruotsalo, T., & Lioma, C. (2026). Robustness Analysis of Pareto-based Joint Evaluation of Fairness and Relevance in Recommender Systems. ACM Transactions on Recommender Systems. https://doi.org/10.1145/3860532

Vancouver

Rampisela TV, Maistro M, Ruotsalo T, Lioma C. Robustness Analysis of Pareto-based Joint Evaluation of Fairness and Relevance in Recommender Systems. ACM Transactions on Recommender Systems. 2026. doi:10.1145/3860532

BibTeX
@article{Rampisela2026,
  author = {Rampisela, Theresia Veronika and Maistro, Maria and Ruotsalo, Tuukka and Lioma, Christina},
  title = {Robustness Analysis of Pareto-based Joint Evaluation of Fairness and Relevance in Recommender Systems},
  journal = {ACM Transactions on Recommender Systems},
  year = {2026},
  doi = {10.1145/3860532}
}
Kostenlos starten →