EUVIMEDEUVIMED entdecken
CrossrefInfokarte

Panic Prediction from Digital Phenotyping: Subject-Level Cross-Validation Reveals Limited Between-Person Generalization

Minoru Hattori, Naoko Hasunuma

Methods of Information in Medicine · 2025-09-01 · 64 (03/06) · 129-133

DOI: 10.1055/a-2895-5222 ↗
AbstractBibliografische Angaben

Abstract

Abstract Objectives A publicly shared digital phenotyping dataset was recently used to train models predicting day-before-panic (DBP) events, reporting a cross-validated receiver operating characteristic area under the curve (ROC-AUC) of around 0.90. Because the stratified five-fold cross-validation (CV) allowed the same participant's days in both training and test folds, it was unclear whether this reflects between-person generalization or information leakage. We reanalyzed the public dataset to quantify this distinction. Methods We reanalyzed the public dataset (3,969 person-days; 254 DBP events, 6.4%, across 19 of 43 participants) with the same XGBoost classifier and 71 features, comparing stratified five-fold cross-validation with leave-one-subject-out (LOSO) cross-validation. A retrospective diagnostic decomposition was applied within-subject z-scoring to dynamic features before LOSO. Results We reproduced the original ROC-AUC at 0.895 ± 0.011. Under LOSO, pooled ROC-AUC dropped to 0.489 (bootstrap 95% CI: 0.325–0.643, including 0.50). The SHapley Additive exPlanations (SHAP) ranking under stratified CV was dominated by static traits (CTQ subscales, BRIAN, SPAQ). The diagnostic decomposition yielded LOSO ROC-AUC 0.516 (precision–recall area under the curve [PR-AUC] 0.114), suggesting a weak residual within-person signal, though this should not be interpreted as prospective performance. Conclusion Robust between-person generalization was not confirmed under subject-held-out evaluation. The feature ranking was trait-dominated, though the data cannot definitively distinguish identity discrimination from other between-person variance sources. Digital phenotyping studies should report subject-level cross-validation alongside standard metrics, per the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) and Prediction model Risk of Bias Assessment Tool (PROBAST) guidelines.

Wissen, das zu Ihrer Arbeit passt.

Vom kostenlosen Einstieg über die persönliche Recherche bis zur vollständigen Research Suite.

Tarife ansehen →

Sind alle Verlagsvolltexte enthalten?

Nein. Bibliografische Nachweise, Abstracts und Volltexte sind unterschiedliche Inhalte. Ob ein Volltext verfügbar ist, hängt von der Quelle, Open Access und Ihren vorhandenen Lizenzen ab. Ein EUVIMED-Tarif ersetzt keine zusätzlichen Verlagsrechte.

Tarife ansehen →