Abstract

Aim

Patients increasingly consult artificial intelligence (AI) chatbots before deciding whether to seek emergency care. This study compared the readability, reliability, and quality of responses generated by ChatGPT, Gemini, and Perplexity in response to frequently searched queries about acute mesenteric ischemia.

Materials and Methods

Sixteen queries identified through Google Trends were submitted to each chatbot, yielding 48 responses. Readability was assessed with seven formulas. Two evaluators independently rated reliability using the JAMA benchmarks and modified DISCERN and quality using the Global Quality score and the Ensuring Quality Information for Patients tool. Platforms were compared with the Friedman and Wilcoxon signed-rank tests.

Results

ChatGPT produced shorter responses than Gemini and Perplexity (medians 180.5, 390.0, and 416.5 words; p<0.001). All platforms’ content was written at a college reading level (median Flesch-Kincaid Grade Level of 14.0-15.5), well above the recommended sixth to eighth-grade level. Perplexity obtained the highest JAMA and Ensuring Quality Information for Patients scores, whereas Gemini and Perplexity outperformed ChatGPT on the modified DISCERN and the Global Quality score.

Conclusion

No chatbot was superior across all measures, and none produced text readable by the average patient. Information on this time-critical emergency should be simplified and professionally reviewed before being recommended to patients. The clinical accuracy and the appropriateness of emergency referral advice were not assessed and require further evaluation.

Introduction

Acute mesenteric ischemia (AMI; not to be confused with acute myocardial infarction) is an uncommon but highly lethal abdominal emergency in which an abrupt reduction in intestinal blood flow leads, if untreated, to bowel necrosis, multiorgan failure, and death (1-3). The condition comprises arterial embolism, arterial thrombosis, mesenteric venous thrombosis, and non-occlusive mesenteric ischemia, because each follows a different clinical course, the presentation is often nonspecific and the diagnosis is frequently delayed (3-5). No laboratory biomarker is accurate enough to establish the diagnosis on its own (6), and computed tomography angiography remains the cornerstone of evaluation (3, 7). Management ranges from endovascular or open revascularization to bowel resection, depending on the mechanism and bowel viability (1, 8, 9), and every delay increases bowel loss and mortality (3).

Artificial intelligence (AI) has so far been explored in AMI mainly as a clinician-facing tool, for example to predict mortality or to interpret computed tomography angiography images (4, 10, 11). A different and far less-studied application is the patient-facing chatbot. Large language model-based platforms such as ChatGPT, Gemini, and Perplexity now answer health questions in conversational language and are increasingly evaluated as sources of medical information (12-14). Their answers, however, vary considerably in reliability and readability, and some appear convincing while conflicting with clinical practice guidelines (12-15).

Because patients may consult these platforms before seeking professional care (16), the information they provide must be understandable, reliable, and clinically appropriate. Studies across several specialties have shown that chatbot responses frequently exceed the reading level recommended for patient materials (14, 17-19). Structured instruments allow these characteristics to be assessed systematically: the JAMA benchmarks examine authorship, attribution, disclosure, and currency (20); DISCERN evaluates the reliability of consumer health information (21); the Global Quality score (GQS) rates overall quality and usefulness (22); and the Ensuring Quality Information for Patients (EQIP) tool assesses the content, identification, and structure of written patient information (23).

For a time-critical emergency such as AMI, the practical question is not only whether chatbot information is readable and reliable, but also whether it helps patients recognize that urgent evaluation is needed. To our knowledge, chatbot-generated information on AMI has not been evaluated previously.

This study aimed to compare the readability, reliability, and quality of responses generated by ChatGPT, Gemini, and Perplexity in response to the most frequently searched queries about AMI.

Materials and Methods

Study Design

This comparative cross-sectional study evaluated the information generated by three AI platforms. The evaluated platforms were ChatGPT, Gemini, and Perplexity. The study focused on patient-oriented information about AMI. No patients, medical records, or personally identifiable information were included. The study was reported in accordance with the Chatbot Assessment Reporting Tool guideline, where applicable (24).

Selection of Search Queries

Frequent search queries related to AMI were identified using Google Trends (Google LLC, Mountain View, CA, USA) on 10 July 2026. The search was conducted using “AMI” as the initial search term, covering the preceding five years, with the geographical setting worldwide, without country-specific restrictions. Related queries were examined under the “Top” category, and the 16 highest-ranked eligible queries were selected for analysis. Duplicate or overlapping queries and those without a plausible clinical relationship to AMI were excluded. Queries were retained in their original keyword format rather than converted into questions to reflect actual public search behavior, consistent with previous studies (12, 14).

Broader terms, including “Thrombosis,” “Acute Abdomen,” and “Peritonitis,” were retained because they appeared among the top-ranked related queries and represented clinically relevant concepts associated with the vascular mechanisms, clinical presentation, or complications of AMI. Although these terms are not specific to AMI, their inclusion was intended to reflect broader information-seeking behavior related to the condition. All three platforms interpreted the abbreviated query “SMA” in the context of the superior mesenteric artery.

1.           Acute mesenteric ischemia

2.           Mesenteric ischemia treatment

3.           Mesenteric ischemia symptoms

4.           Mesenteric artery

5.           Acute mesenteric ischemia symptoms

6.           Thrombosis

7.           Superior mesenteric artery

8.           SMA

9.           Acute abdomen

10.        Intestinal ischemia

11.        Peritonitis

12.        Mesenteric ischemia radiology

13.        Mesenteric venous thrombosis

14.        Bowel infarction

15.        Bowel necrosis

16.        Mesenteric ischemia diagnosis

Artificial Intelligence Response Collection

Responses were collected on 10 July 2026 through the official web interfaces of ChatGPT (OpenAI, San Francisco, CA, USA), Gemini (Google LLC, Mountain View, CA, USA), and Perplexity (Perplexity AI, San Francisco, CA, USA). All three platforms were accessed using their freely available versions without requiring user login.

These platforms use proprietary, closed-source models. Default settings were maintained, and no specific model was manually selected. Because the interfaces did not provide independently verifiable model identifiers for each response, the exact underlying model versions could not be confirmed. Perplexity was accessed using its default automatic model-selection mode.

Each of the 16 queries was entered in English, exactly as retrieved from Google Trends, in a separate, new chat session without prior context, additional instructions, or prompt modification. Only the first response generated for each query was recorded, yielding 48 responses for analysis. No repeated queries or follow-up prompts were used.

The sample size was determined by the 16 highest-ranked eligible queries identified through Google Trends, rather than by a formal power calculation. Each query was submitted once to each of the three platforms, resulting in 48 responses. This approach was intended to provide a standardized cross-sectional comparison of the platforms under identical query conditions at a single point in time, rather than to assess within-platform response variability.

Readability Assessment

Readability was calculated using the Readability Formulas online tool (https://readabilityformulas.com). Seven measures were computed for each response: the Flesch Reading Ease score (FRES) (25), Flesch-Kincaid Grade Level (FKGL) (26), Gunning Fog index (27), Simple Measure of Gobbledygook (SMOG) (28), Automated Readability index (ARI) (29), Coleman-Liau index (30), and Linsear Write Formula. FRES ranges from 0 to 100, with higher scores indicating easier text; the remaining measures estimate the United States school grade level required to understand the text. A sixth- to eighth-grade level is generally recommended for patient education materials (31). Before readability analysis, all chatbot responses underwent standardized text preprocessing. URLs, citations, reference markers, headings, bullet-point symbols, and other non-prose elements were removed. The remaining textual content was analyzed using the seven readability formulas. The same preprocessing procedure was applied consistently across all three platforms to minimize potential formatting-related differences in readability scores.

Quality and Reliability Assessment

Reliability was assessed with the JAMA benchmarks (20) and the modified DISCERN (21, 32). The JAMA benchmarks award one point each for authorship, attribution, disclosure, and currency (total score 0-4). The modified DISCERN consists of five yes/no questions addressing clarity, reliability of sources, balance, provision of additional sources, and reference to areas of uncertainty (total score 0-5). Overall quality was rated using the GQS, a five-point scale ranging from 1 (poor) to 5 (excellent) (22). Patient information quality was assessed with the EQIP tool (23). The original 20-item version (EQIP-20) was used: each item was rated as “yes”, “partly”, “no”, or “not applicable”, and the total score was expressed as a percentage, with higher scores indicating better quality.

All responses were independently rated using the four instruments by two evaluators with different professional backgrounds: an emergency medicine specialist and a faculty member holding a PhD in surgical nursing. Because Perplexity displays source links within its responses, complete blinding to the platform was not feasible. Inter-rater agreement was calculated from the independent ratings; disagreements were then resolved by discussion, and the consensus scores were used in all comparative analyses.

Statistical Analysis

Statistical analyses were performed using IBM SPSS Statistics for Windows, version 27.0 (IBM Corp., Armonk, NY, USA). Continuous variables are presented as median (minimum–maximum) and 25^th-75^th percentiles. Differences among the three platforms were assessed using the Friedman test, and pairwise comparisons were evaluated using the Wilcoxon signed-rank test. A Bonferroni correction was applied to the three pairwise comparisons; therefore, p<0.0167 was considered significant for these tests, while p<0.05 was considered significant for other tests. Effect sizes for the Friedman test were expressed as Kendall’s W, calculated as χ²/[N(k-1)], where N is the number of queries and k the number of platforms; values of approximately 0.1, 0.3, and 0.5 were interpreted as small, moderate, and large effects, respectively.

Inter-rater reliability was evaluated using intraclass correlation coefficients (ICCs) based on a two-way mixed-effects model with absolute agreement; single-measure ICC values with 95% confidence intervals (CIs) are reported. ICC values below 0.50, between 0.50 and 0.75, between 0.75 and 0.90, and above 0.90 were interpreted as poor, moderate, good, and excellent agreement, respectively (33).

Results

A total of 48 responses from three AI platforms were evaluated. Each platform generated responses to the same 16 AMI-related queries. Effect sizes for the overall comparisons ranged from moderate (Kendall’s W=0.31 for Linsear Write) to large (Kendall’s W=1.00 for JAMA), and are presented in Table 1.

ChatGPT produced a median word count of 180.50 words. Median word counts were 390.00 for Gemini and 416.50 for Perplexity. Overall word counts differed significantly among the platforms (p<0.001). ChatGPT responses were shorter than Gemini responses (p=0.001). They were also shorter than Perplexity responses (p=0.001). Gemini and Perplexity showed no significant difference (p=0.877).

Median FRES scores were 0.00 for ChatGPT, 16.50 for Gemini, and 9.00 for Perplexity. FRES differed significantly among the three platforms (p<0.001). ChatGPT had lower FRES scores than Gemini (p<0.001). ChatGPT also had lower scores than Perplexity (p=0.001). Gemini had higher FRES scores than Perplexity(p=0.014).

FKGL scores differed significantly among the platforms (p=0.002). ChatGPT had higher FKGL scores than Gemini (p=0.013). ChatGPT and Perplexity showed no significant difference (p=0.074). Gemini had lower FKGL scores than Perplexity (p=0.010). The median FKGL values for the three platforms (15.52, 14.04, and 15.47 for ChatGPT, Gemini, and Perplexity) corresponded to a college reading level, exceeding the recommended sixth- to eighth-grade level.

Gunning Fog scores differed significantly among the platforms (p<0.001). ChatGPT scored higher than Gemini (p=0.001). ChatGPT also had higher scores than Perplexity (p=0.006). Gemini had lower scores than Perplexity (p=0.010).

SMOG scores differed significantly among the three platforms (p<0.001). ChatGPT had lower SMOG scores than Gemini (p=0.001). ChatGPT also scored lower than Perplexity (p<0.001). Gemini had lower scores than Perplexity (p<0.001).

ARI scores also differed significantly among the platforms (p<0.001). Perplexity’s scores were higher than those of ChatGPT and Gemini (both p<0.001). ChatGPT and Gemini showed no significant difference (p=0.698).

Coleman-Liau scores differed significantly among the platforms (p=0.002). Gemini had significantly lower scores than Perplexity (p=0.004). Other pairwise comparisons remained nonsignificant after Bonferroni correction. Linsear Write scores also differed significantly among platforms (p=0.007). ChatGPT had higher scores than Perplexity (p=0.003). Other pairwise comparisons were not significant after correction. Detailed readability findings are presented in Tables 1 and 2.

JAMA scores differed significantly among the three platforms (p<0.001). ChatGPT and Gemini both had median JAMA scores of 0.00. Perplexity had a median JAMA score of 2.00. Perplexity outperformed both competing platforms (p<0.001 for both comparisons). ChatGPT and Gemini showed no difference (p=1.000).

Modified DISCERN scores also differed significantly among platforms (p<0.001). Median scores were 2.50 for ChatGPT, 4.00 for Gemini, and 3.00 for Perplexity. Gemini scored higher than ChatGPT (p=0.001). Perplexity also scored higher than ChatGPT (p=0.001). Gemini and Perplexity showed no significant difference (p=0.527).

GQS scores differed significantly among the three platforms(p=0.002). Median scores were 3.00 for ChatGPT and 4.00 for both Gemini and Perplexity. Gemini scored higher than ChatGPT (p=0.008). Perplexity also scored higher than ChatGPT (p=0.003). Gemini and Perplexity showed no significant difference (p=0.414).

EQIP scores differed significantly among the three platforms (p<0.001). Median scores were 47.00 for ChatGPT, 49.00 for Gemini, and 52.00 for Perplexity. Gemini scored higher than GPT (p=0.002). Perplexity also scored higher than ChatGPT (p=0.001). Perplexity scored higher than Gemini (p=0.013). These findings are presented in Tables 1 and 3.

Inter-rater agreement varied across the four instruments. The two raters assigned identical JAMA scores to all 48 responses (100% agreement; ICC=1.000, 95% CI: 1.000-1.000), and EQIP showed good agreement (ICC=0.900, 95% CI: 0.857-0.931). For modified DISCERN (ICC=0.679, 95% CI: 0.272–0.880) and GQS (ICC=0.638, 95% CI: 0.170-0.874), the point estimates indicated moderate agreement, although the wide CIs ranged from poor to good (Table 4).

Discussion

In this study, the three chatbots differed in almost every measure, but none performed best across all of them. Perplexity scored highest on the JAMA benchmarks and EQIP; Gemini and Perplexity scored higher than ChatGPT on modified DISCERN and GQS; ChatGPT produced the shortest responses. All three platforms generated text at a college reading level, which is well above the level recommended for patient information.

Patient education materials are recommended to be written at a sixth- to eighth-grade reading level (31). In our study, median FKGL values ranged from 14.04 to 15.52, and median FRES values between 0 and 16.5 fall within the range classified as very difficult to read (25). This finding is consistent with previous chatbot studies on cardiopulmonary resuscitation, sexually transmitted diseases, and prostate cancer radiotherapy, in which responses also exceeded recommended levels (14, 17, 18). For AMI, the gap is particularly relevant: a patient with sudden, severe abdominal pain who receives complex, terminology-heavy text may fail to extract the single most important message: that emergency evaluation is needed.

Pairwise readability comparisons depended on the formula used. ChatGPT had the lowest FRES, indicating the most difficult text, yet it also had the lowest SMOG score. This apparent contradiction is more likely to reflect properties of the formulas than a true difference in readability. SMOG was developed for samples of 30 sentences (28), whereas ChatGPT responses were considerably shorter (a median of 180.5 words). Although URLs, citations, reference markers, and other non-prose elements were removed before analysis, differences in vocabulary, sentence structure, and response length may still have contributed to variations among readability formulas. Individual formula results should therefore be interpreted cautiously; the consistent message across formulas is that no platform approached the recommended level.

ChatGPT generated substantially shorter responses than the other two platforms, which may reflect different default response strategies (12-14). Longer answers allow broader explanations, but length alone does not improve understanding and may increase the cognitive burden for a patient with acute symptoms.

Perplexity was the only platform to obtain JAMA points, reflecting its practice of displaying source links alongside responses, which corresponds to the attribution and currency criteria of the JAMA benchmarks (20). Similar advantages of Perplexity have been reported in previous comparisons (13, 14). However, the presence of citations does not guarantee that the cited sources are accurate, relevant, or even existent. None of the platforms provided authorship or disclosure information.

Gemini and Perplexity outperformed ChatGPT on modified DISCERN, GQS, and EQIP; however, they did not differ from each other on modified DISCERN and GQS. Similarly, a previous study evaluating cardiopulmonary resuscitation training videos on YouTube demonstrated considerable variability in the quality and reliability of digital health information, highlighting the importance of standardized assessment tools and professional oversight (34). These instruments reward balance, discussion of uncertainty, structure, and completeness (21-23), features more likely to be present in longer responses. This may explain why the two platforms with longer responses scored higher, but it also illustrates a trade-off: more complete responses were not easier to read. Comparable platform-related differences have been reported for palliative care, cardiopulmonary resuscitation, microtia, and breast cancer surgery (13, 14, 19, 35).

These findings have specific implications for emergency medicine. Mortality in AMI increases with delays in diagnosis and revascularization (2, 5), and structured management pathways aim to shorten this interval (1, 8). A patient-facing chatbot is useful in this setting only if it clearly conveys that sudden, severe abdominal pain, particularly in older patients or in those with atrial fibrillation or vascular disease, requires immediate emergency evaluation (1, 3). The instruments used in this study do not specifically assess the presence of such urgency messages, and future studies should evaluate this outcome directly. Importantly, higher reliability or quality scores should not be interpreted as evidence of clinical accuracy or guideline adherence. In the context of AMI, the absence of a direct assessment of emergency referral advice and of guideline-consistent recommendations limits the clinical interpretation of these findings.

The AI tools studied in AMI to date, such as mortality prediction models and computed tomography angiography analysis (4, 10, 11), are designed to support clinicians. Patient-facing chatbots differ fundamentally because their output reaches users without professional filtering. Evidence that chatbot recommendations may conflict with clinical guidelines (15) and that their quality varies across platforms (35) underlines the need for professional oversight.

Although the ICC point estimates indicated moderate inter-rater agreement for modified DISCERN and GQS, the wide 95% CIs suggest considerable uncertainty regarding the precision of these estimates. Therefore, differences between platforms based on these two instruments should be interpreted cautiously, as variability between evaluators may have influenced the observed results. The different professional backgrounds of the two evaluators, emergency medicine and surgical nursing, may also have contributed to the lower agreement for these more subjective instruments, although this combination reflects both the clinical and patient-education perspectives.

Study Limitations

This study has several limitations. First, only three platforms were evaluated; their responses may change with model updates. Second, only one response per query was evaluated. Given the stochastic nature of generative AI, repeated queries may yield different responses. This approach limits the assessment of within-platform variability, reproducibility, and generalizability. Future studies should incorporate repeated queries across multiple time points to evaluate response consistency and stability more comprehensively. Third, queries were entered in the keyword format retrieved from Google Trends. Although this reflects search-engine behavior and is consistent with previous studies (12, 14), it may not represent conversational, question-based interactions, which could produce different answers. Some queries were broader terms related to AMI rather than specific to it. Although these broader queries were retained to reflect real-world search behavior, their inclusion may have influenced the observed differences in chatbot performance. Future studies focusing exclusively on AMI-specific queries could help determine whether the present findings remain consistent when broader or ambiguous search terms are excluded. Fourth, only English-language queries were analyzed. Fifth, readability formulas measure linguistic complexity but not actual patient comprehension. Sixth, the quality instruments include subjective components; the two evaluators were from different professional backgrounds; complete blinding of the evaluators to the platform was not feasible because Perplexity displays its sources. Seventh, because the platforms were accessed through free, non-logged-in web interfaces, the exact underlying model versions could not be independently verified; the findings should therefore be interpreted as comparisons of the publicly accessible chatbot platforms under the specified access conditions rather than as comparisons of definitively identified individual language models. Clinical accuracy and guideline adherence were not directly evaluated, including whether responses appropriately recommended an emergency assessment for suspected AMI. Therefore, higher quality or reliability scores should not be interpreted as evidence of clinically accurate or safe recommendations. Future studies should include larger, question-based query sets, repeated sampling, and direct, guideline-based assessment of clinical accuracy and emergency referral advice.

Conclusion

ChatGPT, Gemini, and Perplexity differed in the readability, reliability, and quality of their responses to frequently searched AMI queries, and no platform was superior across all measures. Perplexity provided the most transparent sourcing and the highest quality of patient information, whereas Gemini and Perplexity outperformed ChatGPT in reliability and overall quality. All three platforms, however, produced text well above the recommended reading level. Given the time-critical nature of AMI, these findings highlight the need for more accessible patient information. However, the clinical accuracy, guideline adherence, and appropriateness of emergency referral advice provided by these platforms require further evaluation before their use as reliable sources of patient guidance can be recommended.