Quality, readability, and information fidelity of large language model–generated versus physician-authored informed-consent content for targeted therapies: a blinded, paired comparative study

Abstrakt

Objective To compare the content quality of basic information, risks, benefits, and alternative options in informed consent forms for targeted therapy generated by large language models (LLMs) versus those drafted by physicians, and to evaluate informational fidelity and reading level. Methods This study used a single-center, retrospective, paired-case, blinded multi-evaluator design. We enrolled 70 adult patients who received targeted therapy in the Department of Rheumatology at Beijing Tongren Hospital during 2025. LLM texts were generated using DeepSeek-V4-pro based on identical de-identified case data and standardized prompts, and corresponding physician texts were extracted from electronic records. Three rheumatologists performed blinded scoring using a scale comprising 8 domains and 13 items. Reading level and informational errors were assessed using CRIE 3.0 and source tracing. Results A total of 140 texts were analyzed. The LLM arm achieved a higher median overall quality score than the physician arm (60 vs. 50; adjusted mean difference 9.95, 95% CI 9.36 to 10.54; p < 0.01), with higher scores for risks, benefits, and alternatives (all adjusted p < 0.01). The LLM arm also exhibited higher reading level scores (7.87 vs. 7.10; p < 0.01) and a higher rate of hallucinations (27.1% vs. 0; p < 0.01). No statistically significant between-arm differences were found for inaccuracies or omissions. Conclusion In this exploratory study, according to a locally developed assessment framework, LLM-generated texts received higher composite evaluator ratings and more consistently covered predefined informed consent content domains. However, they more frequently contained unsupported patient-specific inferences and imposed a greater reading burden. LLMs may be useful for generating structured drafts and checking coverage of predefined content, but they should not be used to produce final informed consent documents without physician review and item-by-item verification. These findings require further validation using patient-centered outcomes.