Reliability and readability of five large language models in answering patient-oriented questions about irritable bowel syndrome: a comparative study

Abstract

Background Irritable bowel syndrome (IBS) is a common gastrointestinal disorder requiring long-term self-management. Patients therefore need accurate, transparent, and comprehensible health information. Large language models (LLMs) are increasingly used for health information seeking, yet the quality, transparency, and readability of their responses remain insufficiently evaluated. This study compared five publicly accessible LLM-based platforms to inform their use in patient education. Methods This web-based cross-sectional study retrieved Google Trends queries from the preceding 5 years using the English MeSH term “Irritable Bowel Syndrome” and the “Worldwide” geographic setting. After screening, 12 patient-oriented questions were submitted in English to ChatGPT, Gemini, Microsoft Copilot, Perplexity, and DeepSeek under a standardized single-turn, zero-shot protocol. Each configuration generated one English-language response per question, yielding 60 responses. Two gastroenterologists independently evaluated responses using DISCERN, the Ensuring Quality Information for Patients (EQIP) instrument, Global Quality Score (GQS), and Journal of the American Medical Association (JAMA) benchmark criteria. Information-quality scores were compared using question-blocked Friedman tests, followed by Bonferroni-adjusted paired Wilcoxon signed-rank tests when the omnibus result was significant. Readability was assessed using ARI, GFI, FKGL, CL, SMOG, and FRES, with a sixth-grade reading level as the reference. Results Question-blocked Friedman tests identified differences in DISCERN, EQIP, and JAMA scores (all permutation P < 0.001), whereas no statistically significant difference was detected in GQS scores (permutation P = 0.600). Copilot and Perplexity had the highest median DISCERN scores (48.00), while Copilot had the highest median EQIP score (80.00), significantly exceeding ChatGPT, Gemini, and DeepSeek in Bonferroni-adjusted paired comparisons. Despite relatively higher JAMA scores for Copilot and Perplexity, all configurations had median scores of only 0–1, indicating limited fulfillment of the assessed transparency-related criteria. All configurations had a median GQS of 4. Median ARI, GFI, FKGL, CL, and SMOG scores exceeded 6, and all median FRES scores were below 80. Gemini had the highest medians for all five grade-level indices; Copilot had the lowest medians for ARI, GFI, FKGL, and SMOG, while DeepSeek had the lowest median CL and the highest median FRES. No model met the predefined sixth-grade readability standard. Conclusion All five LLMs generated IBS-related health information with educational value, but differed in treatment-related information quality, content completeness, and source transparency. Within the sampled responses, Copilot and Perplexity performed relatively well on selected measures, although transparency remained limited. Across all configurations, median readability scores indicated estimated reading levels above the prespecified sixth-grade reference, suggesting potentially demanding text. LLMs should serve as professionally reviewed adjuncts to patient education rather than independent sources for diagnosis, treatment, or clinical decision-making.