What References are ChatGPT, Gemini, Copilot, and Perplexity Providing for Consumer Health Questions?

Authors

• Scott Johnson, Kaiser Permanente School of Medicine
• Catherine Johnson, University of La Verne
• Ivan Portillo, Chapman UniversityFollow

Document Type

Conference Proceeding

Publication Date

5-2025

Abstract

Background

With the growing popularity of generative artificial intelligence (AI) models such as ChatGPT, consumers may turn to these tools to easily seek health information. To our knowledge, no study has analyzed the references provided by multiple models for consumer health questions.

Objective

We aimed to analyze the references provided by ChatGPT, Gemini, Copilot, and Perplexity for consumer health questions in order to determine the most frequently appearing references.

Methods

AI generative models ChatGPT 4.0, Google Gemini, Microsoft Copilot, and Perplexity were each asked 30 consumer health questions and prompted to provide the corresponding references. The references were recorded.

The questions were generated by asking the models for the most frequently searched health conditions. The responses were tallied, and the top 30 health conditions were used to create 30 corresponding questions.

Results

There was a total of 580 references, and the average number of references per question was 5.0. The top five references that were provided most frequently were National Institutes of Health (88 references), followed by Mayo Clinic (80), Centers for Disease Control and Prevention (50), Cleveland Clinic (34), and World Health Organization (32). Based on domain type, organization (.org) and government (.gov) websites were the most common references, at 47% and 31% respectively.

Conclusions

The results revealed that most of the references came from reputable sources. Consumers seeking health information would be well-served to evaluate the references provided by AI generative models before trusting them.

Recommended Citation

Portillo, I., Johnson, S., Johnson, C. (2025, May). What References are ChatGPT, Gemini, Copilot, and Perplexity Providing for Consumer Health Questions? Poster Presented at the Medical Library Association Annual Meeting 2025.