System Maintenance Notice

Please note that maintenance work is scheduled between 4 AM and 5 AM - CET. During this time, you may experience reduced performance. Thanks for your patience.

Publications

Journal Article Epublish

PubMed: 41782614 DOI: 10.1016/j.pecinn.2026.100461

Validity and reliability of ChatGPT's responses on dietary supplements in Japan: A quality assessment and content analysis.

PEC Innov 8:100461

Authors: Mingxin Liu, Tsuyoshi Okuhara, Ritsuko Shirabe, Yuriko Nishiie, Xinyi Chang, Hiroko Okada, Takahiro Kiuchi

Affiliations

  • Department of Health Communication, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.
  • Department of Health Communication, School of Public Health, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.
  • University hospital Medical Information Network (UMIN) Center, The University of Tokyo Hospital, Tokyo, Japan.
  • Department of Industrial Engineering and Economics, Institute of Science Tokyo, Tokyo, Japan.

Abstract

OBJECTIVE

This study evaluated the validity and reliability of large language model (LLM) responses on dietary supplements (DS), a domain marked by scientific controversy and misinformation. The goal was to support informed consumer decisions and guide improvements in LLM performance.

METHODS

We collected responses from GPT-4 and GPT-4o on the effects of 30 DS on six diseases. Two medical professionals categorized each response as "Effective," "Uncertain," or "Not Effective." They also created a guideline to assess evidence-based effectiveness and compared it with LLM-generated responses to determine accuracy. Additionally, we conducted qualitative content analysis to identify response patterns and misleading content.

RESULTS

GPT-4 and GPT-4o affirmed DS effectiveness in only 10% of cases, with 40% rated as "Uncertain" and 50% as "Not Effective." Accuracy was about 57%, considerably lower than that observed in nutrition-related studies (57% in DS vs. 80% ∼ in structured nutrition tasks"). Content analysis showed templated responses, frequent ambiguity, and occasional inclusion of irrelevant or incorrect information.

CONCLUSION

Our findings suggest that ChatGPT's responses on dietary supplements are generally cautious but often ambiguous, with a moderate risk of misinformation. As generative AI becomes a common source for health advice, these limitations could mislead users. Enhancing LLMs' evidence-based accuracy and ensuring consistent professional guidance are essential.

INNOVATION

This is the first study to assess the validity and reliability of LLM-generated responses on dietary supplements using both quantitative and qualitative methods. We also developed a novel evidence-based framework to evaluate supplement effectiveness, providing a new tool for future research and supporting safer AI-assisted health communication.

MeSH

Dietary Supplements, Artificial Intelligence