System Maintenance Notice

Please note that maintenance work is scheduled between 4 AM and 5 AM - CET. During this time, you may experience reduced performance. Thanks for your patience.

Publications

Journal Article Scoping Review Epublish 4 citations

PubMed: 39614219 DOI: 10.1186/s12911-024-02709-7

Analyzing evaluation methods for large language models in the medical field: a scoping review.

BMC Med Inform Decis Mak 24(1):366

Authors: Junbok Lee, Sungkyung Park, Jaeyong Shin, Belong Cho

Affiliations

  • Institute for Innovation in Digital Healthcare, Yonsei University, Seoul, Republic of Korea. Department of Human Systems Medicine, Seoul National University College of Medicine, Seoul, Republic of Korea.
  • Department of Bigdata AI Management Information, Seoul National University of Science and Technology, Seoul, Republic of Korea.
  • Department of Preventive Medicine and Public Health, Yonsei University College of Medicine, 50-1, Yonsei-ro, Seodaemun-gu, Seoul, 03722, Republic of Korea. Institute of Health Services Research, Yonsei University College of Medicine, Seoul, Korea.
  • Department of Human Systems Medicine, Seoul National University College of Medicine, Seoul, Republic of Korea. Department of Family Medicine, Seoul National University Hospital, Seoul, Republic of Korea. Department of Family Medicine, Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Republic of Korea.

Abstract

BACKGROUND

Owing to the rapid growth in the popularity of Large Language Models (LLMs), various performance evaluation studies have been conducted to confirm their applicability in the medical field. However, there is still no clear framework for evaluating LLMs.

OBJECTIVE

This study reviews studies on LLM evaluations in the medical field and analyzes the research methods used in these studies. It aims to provide a reference for future researchers designing LLM studies.

METHODS & MATERIALS

We conducted a scoping review of three databases (PubMed, Embase, and MEDLINE) to identify LLM-related articles published between January 1, 2023, and September 30, 2023. We analyzed the types of methods, number of questions (queries), evaluators, repeat measurements, additional analysis methods, use of prompt engineering, and metrics other than accuracy.

RESULTS

A total of 142 articles met the inclusion criteria. LLM evaluation was primarily categorized as either providing test examinations (n = 53, 37.3%) or being evaluated by a medical professional (n = 80, 56.3%), with some hybrid cases (n = 5, 3.5%) or a combination of the two (n = 4, 2.8%). Most studies had 100 or fewer questions (n = 18, 29.0%), 15 (24.2%) performed repeated measurements, 18 (29.0%) performed additional analyses, and 8 (12.9%) used prompt engineering. For medical assessment, most studies used 50 or fewer queries (n = 54, 64.3%), had two evaluators (n = 43, 48.3%), and 14 (14.7%) used prompt engineering.

CONCLUSIONS

More research is required regarding the application of LLMs in healthcare. Although previous studies have evaluated performance, future studies will likely focus on improving performance. A well-structured methodology is required for these studies to be conducted systematically.

MeSH

Medical Informatics, Research Design, Language, Research

Cited by