Publications

Journal Article
Preprint
Epublish
Open access — free full text available
Open access
Electronic publication date
November 6, 2026

View on Pubmed.gov 42990471 | View on doi.org 10.64898/2026.1... | View the free full text Full text

DiaBench: A Benchmarking Framework to Evaluate the Performance of LLMs in T2D Patient Education.

Search for publications in this journal medRxiv

Abstract

BACKGROUND

Effective type 2 diabetes (T2D) education improves patient outcomes, but clinician time constraints limit what can be covered during encounters. Large language models (LLMs) offer a scalable way to answer patient questions, yet their accuracy and reliability in T2D education across health literacy levels remain poorly characterized.

METHODS

We developed DiaBench, a standardized benchmarking framework built from 96 question- and-answer pairs collected by clinicians from representative real-world patient encounters. GPT-5 transformed each pair into multiple-choice questions (MCQs) under three configurations (1-3) simulating different question formats and levels of user background. A five-member panel of certified diabetes care and education specialists (CDCESs), medical doctors (MDs), and endocrinologists validated the transformations, rating question rephrasing, answer rephrasing, and answer correctness. We then evaluated seven LLMs (GPT-5 1 , Gemini-2.5-Flash 2 , Cohere Command 3 , Llama-4 4 , DeepSeek-R1 5 , Perplexity Sonar 6 , Mistral-Large-2.0 7 ) for accuracy and self-consistency across three runs.

RESULTS

(i) Expert validation showed a consensus over MCQ transformation. For configuration 1, 98.1% (SD 0.01) of rephrased answers were appropriate, 98.1% (SD 0.01) of MCQ answers were correct, and 95.0% (SD 0.05) of rephrased questions were appropriate; corresponding values for configuration 2 were 96.3% (SD 0.05), 97.5% (SD 0.02), and 89.8% (SD 0.14). (ii) across benchmarking for configurations 1 and 2, most models achieved high accuracy (≥95%), but Mistral-Large-2.0, Llama 4, and Perplexity Sonar showed substantially lower self-consistency (60.4%-91.7%) than other models (≥95%). All models degraded in both accuracy and self-consistency on configuration 3.

CONCLUSIONS

DiaBench quantitatively differentiates LLM capability in T2D patient education, with Gemini, GPT-5, and DeepSeek-R1 performing strongly and other models suboptimally. DiaBench provides a transferable platform for evaluating existing and newly developed LLMs, addressing the need for standardized benchmarking before clinical deployment of LLM-based systems in patient education.

References

No references provided.

Conflict of interest statement

No conflict of interest statement provided.

MeSH

Specialties