EUVIMEDOntdek EUVIMED
Open access · CC BY 4.0

Nonparametric Assessment of the Calibration of Individualized Treatment Effects

Mohsen Sadatsafavi iD, Jeroen Hoogland iD, Thomas P A Debray, John Petkau

Statistics in Medicine · 2026-09 · 45 (20-22)

DOI: 10.1002/sim.70724 ↗

At a glance

Journal article in Statistics in Medicine, published online on 2026-08-25. Written by Mohsen Sadatsafavi and 3 co-authors. Institution: Respiratory Evaluation Sciences Program, Collaboration for Outcomes Research and Evaluation, Faculty of Pharmaceutical Sciences, The University of Br…. Major MEDLINE topics: Models, Statistical, Precision Medicine. Cited 2 times so far (Crossref); 36 references. The full text is freely available (open access, CC BY 4.0). Funded by: CIHR, Canadian Institutes of Health Research.

SamenvattingSubjectsBibliografische gegevensCiteRelated studies

Abstract (original language)

An important aspect of the performance of algorithms that predict individualized treatment effects (ITEs) is moderate calibration, that is, the average treatment effect among individuals with predicted treatment effect of z being equal to z. The assessment of moderate calibration is a challenging task on two fronts: counterfactual responses are unobserved, and quantifying the conditional response function for models that generate continuous predicted values requires regularization. Perhaps because of these challenges, there is currently no inferential method for the null hypothesis that an ITE model is moderately calibrated in a population. In this work, we propose nonparametric methods for the assessment of moderate calibration of ITE models for binary outcomes using data from a randomized trial. These methods simultaneously resolve both challenges, resulting in novel graphical, numerical, and inferential methods for the assessment of moderate calibration. The key idea is to formulate a stochastic process for the cumulative prediction errors that obeys a functional central limit theorem, enabling the use of the properties of Brownian motion for asymptotic inference. We propose two approaches to construct this process from a sample: a conditional approach that relies on predicted risks (often an auxiliary output of ITE models), and a marginal approach based on replacing the cumulative conditional expected value and variance terms with their marginal counterparts. Numerical simulations confirm the desirable properties of both approaches and their ability to detect miscalibration of different forms. We use a case study to provide suggestions on graphical presentation and the interpretation of results. Moderate calibration of predicted ITEs can be assessed without requiring regularization techniques or making assumptions about the functional form of treatment response. The accompanying cumulcalib R package implements this method ( https://cran.r‐project.org/package=cumulcalib ).

Subjects

MeSH terms (MEDLINE)

Models, StatisticalPrecision MedicineHumansAlgorithmsCalibrationComputer SimulationRandomized Controlled Trials as TopicStatistics, NonparametricStochastic ProcessesTreatment Effect HeterogeneityPrediction Algorithms

Author keywords

calibrationclinical prediction modelshypothesis testinginferencetreatment benefit

Authors and institutions

  • Respiratory Evaluation Sciences Program, Collaboration for Outcomes Research and Evaluation, Faculty of Pharmaceutical Sciences, The University of British Columbia, Vancouver, Canada.
  • Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, the Netherlands.
  • Smart Data Analysis and Statistics B.V., Utrecht, Germany.
  • University Medical Center Göttingen, Göttingen, Germany.
  • Department of Statistics, The University of British Columbia, Vancouver, Canada.

ORCID profiles: Mohsen Sadatsafavi, Jeroen Hoogland

Funding

  • CIHR
  • Canadian Institutes of Health Research

Cite

APA

Sadatsafavi, M., Hoogland, J., Debray, T. P. A., & Petkau, J. (2026). Nonparametric Assessment of the Calibration of Individualized Treatment Effects. Statistics in Medicine, 45(20-22). https://doi.org/10.1002/sim.70724

Vancouver

Sadatsafavi M, Hoogland J, Debray TPA, Petkau J. Nonparametric Assessment of the Calibration of Individualized Treatment Effects. Statistics in Medicine. 2026;45(20-22). doi:10.1002/sim.70724

BibTeX
@article{Sadatsafavi2026,
  author = {Sadatsafavi, Mohsen and Hoogland, Jeroen and Debray, Thomas P A and Petkau, John},
  title = {Nonparametric Assessment of the Calibration of Individualized Treatment Effects},
  journal = {Statistics in Medicine},
  year = {2026},
  volume = {45},
  number = {20-22},
  doi = {10.1002/sim.70724}
}
Start for free →