Clinical accuracy, safety, triage, and test-retest consistency of general-purpose large language models for patient-centred respiratory questions: a blinded paired benchmarking study

Achoimre

Background General-purpose large language models are increasingly used by patients to obtain health information and guidance on whether medical care is required. In respiratory medicine, clinically plausible but incomplete responses may still cause harm if warning signs are missed or urgent assessment is delayed. This study compared the clinical accuracy, safety, triage performance, and test-retest consistency of three contemporary large language models when answering patient-centred respiratory questions. Methods A blinded, paired, repeated-response benchmarking study was conducted using 150 respiratory questions covering seven clinical domains and equally distributed across low-, moderate-, and high-risk strata. Each question was submitted twice in separate conversations to ChatGPT-5.5, Gemini 3.5 Flash, and Grok 4.1, generating 900 responses. Two respiratory physicians independently assessed anonymised responses against prespecified item-specific reference standards, with prespecified disagreements adjudicated by a third senior respiratory physician. The primary outcome was the proportion of clinically acceptable responses, defined using accuracy, safety, warning-sign recognition, and triage criteria. Secondary outcomes included major safety errors, under-triage, safety-netting, readability, and test-retest consistency. Results In the first response round, clinically acceptable responses were generated for 88.0% of questions by ChatGPT-5.5, 81.3% by Gemini 3.5 Flash, and 74.0% by Grok 4.1 (overall P < 0.001). Major safety errors occurred in 3.3%, 6.0%, and 10.0% of responses, respectively ( P = 0.009). Under-triage occurred in 4.7% of ChatGPT-5.5 responses, 9.3% of Gemini 3.5 Flash responses, and 13.3% of Grok 4.1 responses. Among high-risk questions, exact or acceptable triage agreement was 84.0%, 72.0%, and 62.0%, while under-triage occurred in 8.0%, 16.0%, and 22.0%, respectively. Test-retest clinical response consistency was highest for ChatGPT-5.5 (94.0%), followed by Gemini 3.5 Flash (89.3%) and Grok 4.1 (84.7%). Conclusions The three models answered many respiratory questions accurately and safely, but clinically important differences were observed in safety, triage, and response consistency. All models produced under-triage and major safety errors, particularly in high-risk respiratory scenarios. These systems may support selected low-risk educational tasks but should not independently determine emergency triage, medication changes, or oxygen adjustment.