Evaluation Study Journal Article Epublish
PubMed: 40701625 DOI: 10.1136/bmjment-2025-301762
Development and evaluation of prompts for a large language model to screen titles and abstracts in a living systematic review.
BMJ Ment Health 28(1)
Authors: Ava Homiar, James Thomas, Edoardo G Ostinelli, Jaycee Kennett, Claire Friedrich, Pim Cuijpers, Mathias Harrer, Stefan Leucht, Clara Miguel, Alessandro Rodolico, Yuki Kataoka, Tomohiro Takayama, Keisuke Yoshimura, Ryuhei So, Yasushi Tsujimoto, Yosuke Yamagishi, Shiro Takagi, Masatsugu Sakata, Đorđe Bašić, Eirini Karyotaki, Jennifer Potts, Georgia Salanti, Toshi A Furukawa, Andrea Cipriani
Affiliations
- Department of Psychiatry, University of Oxford, Oxford, UK.
- Division of Clinical Informatics, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA.
- University College London, London, UK.
- Department of Psychiatry, University of Oxford, Oxford, UK.
- Vrije Universiteit Amsterdam, Amsterdam, The Netherlands.
- Psychology & Digital Mental Health Care, Technical University of Munich, Munchen, Germany.
- Psychiatry and Psychotherapy, Technical University of Munich School of Medicine, Munich, Germany.
- Department of Clinical, Neuro and Developmental Psychology, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands.
- Technical University of Munich, Munich, Germany.
- Kyoto Min-iren Asukai Hospital, Kyoto, Japan.
- Kyoto University, Kyoto, Japan.
- Okayama Psychiatric Medical Center, Okayama, Japan.
- The University of Tokyo, Bunkyo, Tokyo, Japan.
- Independent Researcher, Kyoto, Japan.
- Nagoya City University Graduate School of Medical Sciences and Medical School, Nagoya, Japan.
- NIHR Oxford Health Biomedical Research Centre, Oxford, UK.
- University of Bern, Bern, Switzerland.
- Kyoto University Graduate School of Medicine Faculty of Medicine, Kyoto, Japan.
Abstract
BACKGROUND
Living systematic reviews (LSRs) maintain an updated summary of evidence by incorporating newly published research. While they improve review currency, repeated screening and selection of new references make them labourious and difficult to maintain. Large language models (LLMs) show promise in assisting with screening and data extraction, but more work is needed to achieve the high accuracy required for evidence that informs clinical and policy decisions.
OBJECTIVE
The study evaluated the effectiveness of an LLM (GPT-4o) in title and abstract screening compared with human reviewers.
METHODS
Human decisions from an LSR on prodopaminergic interventions for anhedonia served as the reference standard. The baseline search results were divided into a development and a test set. Prompts guiding the LLM's eligibility assessments were refined using the development set and evaluated on the test set and two subsequent LSR updates. Consistency of the LLM outputs was also assessed.
RESULTS
Prompt development required 1045 records. When applied to the remaining baseline 11 939 records and two updates, the refined prompts achieved 100% sensitivity for studies ultimately included in the review after full-text screening, though sensitivity for records included by humans at the title and abstract stage varied (58-100%) across updates. Simulated workload reductions of 65-85% were observed. Prompt decisions showed high consistency, with minimal false exclusions, satisfying established screening performance benchmarks for systematic reviews.
CONCLUSIONS
Refined GPT-4o prompts demonstrated high sensitivity and moderate specificity while reducing human workload. This approach shows potential for integrating LLMs into systematic review workflows to enhance efficiency.
MeSH
Language, Workload, Work, Records, Mental Health