This paper is in the following e-collection/theme issue:

Commentary (1) Artificial Intelligence (2129) Evaluation and Research Methodology for mHealth (242) mHealth for Data Collection and Research (1196) Chatbots and Conversational Agents (740) Generative Language Models Including ChatGPT (708) Methods and Instruments in Medical Informatics (189)

Published on in Vol 12 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/57978, first published .
The Evaluation of Generative AI Should Include Repetition to Assess Stability

The Evaluation of Generative AI Should Include Repetition to Assess Stability

The Evaluation of Generative AI Should Include Repetition to Assess Stability

Authors of this article:

Lingxuan Zhu1 Author Orcid Image ;   Weiming Mou2 Author Orcid Image ;   Chenglin Hong1 Author Orcid Image ;   Tao Yang3 Author Orcid Image ;   Yancheng Lai1 Author Orcid Image ;   Chang Qi4 Author Orcid Image ;   Anqi Lin1 Author Orcid Image ;   Jian Zhang1 Author Orcid Image ;   Peng Luo1 Author Orcid Image
Article Authors Cited by (18) Tweetations (3) Metrics

Journals

  1. Muntean G, Marginean A, Groza A, Damian I, Roman S, Hapca M, Sere A, Mănoiu R, Muntean M, Nicoară S. A Qualitative Evaluation of ChatGPT4 and PaLM2’s Response to Patient’s Questions Regarding Age-Related Macular Degeneration. Diagnostics 2024;14(14):1468 View
  2. Zhu L, Mou W, Luo P. Ensuring Safety and Consistency in Artificial Intelligence Chatbot Responses. JAMA Oncology 2024;10(11):1597 View
  3. Kong G, Ouellette R, Murthy D. Generative artificial intelligence and social media: insights for tobacco control. Tobacco Control 2024:tc-2024-058813 View
  4. Ruta M, Gaidici T, Irwin C, Lifshitz J. ChatGPT for Univariate Statistics: Validation of AI-Assisted Data Analysis in Healthcare Research. Journal of Medical Internet Research 2025;27:e63550 View
  5. Chen X, Xiang J, Lu S, Liu Y, He M, Shi D. Evaluating large language models and agents in healthcare: key challenges in clinical applications. Intelligent Medicine 2025;5(2):151 View
  6. Zhu L, Lai Y, Xie J, Mou W, Huang L, Qi C, Yang T, Jiang A, Gan W, Zeng D, Tang B, Xiao M, Chu G, Liu Z, Cheng Q, Lin A, Luo P. Evaluating the potential risks of employing large language models in peer review. Clinical and Translational Discovery 2025;5(4) View
  7. Shen J, Feng S, Zhang P, Qi C, Liu Z, Feng Y, Dong C, Xie Z, Gan W, Zhu L, Mou W, Zeng D, Tang B, Xiao M, Chu G, Cheng Q, Zhang J, Peng S, Bai Y, Wong H, Jiang A, Luo P, Lin A. Evaluating generative AI models for explainable pathological feature extraction in lung adenocarcinoma: grading assessment and prognostic model construction. International Journal of Surgery 2025;111(7):4252 View
  8. Mutin J, Hafizin M, El-Jammal V, Morel C, Leca L, Bergeron M, Bussières J. Extraction de données d’articles scientifiques pharmaceutiques par intelligence artificielle : évaluation de la conformité des réponses de ChatGPT 4.0 avec l’extraction humaine. Le Pharmacien Clinicien 2025 View
  9. Lin A, Fang X, Jiang A, Qi C, Gan W, Zhu L, Mou W, Zeng D, Xiao M, Chu G, Peng S, Wong H, Zhang L, Zhang H, Deng X, Cheng Q, Zhang H, Zhong Z, Li Z, Tang B, Luo P. Large language models in drug development: Current progress and future directions. Current Molecular Pharmacology 2025;18(1):1 View
  10. Luo P, Fan C, Li A, Jiang T, Jiang A, Qi C, Gan W, Zhu L, Mou W, Zeng D, Tang B, Xiao M, Chu G, Liang Z, Shen J, Liu Z, Wei T, Cheng Q, Lin A, Chen X. Performance analysis of large language models in multi-disease detection from chest computed tomography reports: a comparative study. International Journal of Surgery 2025;111(8):5071 View
  11. Lu R, Shen J, Jiang A, Chen W, Qi C, Chen L, Zhu L, Mou W, Gan W, Zeng D, Tang B, Xiao M, Chu G, Peng S, Wong H, Zhang L, Zhang H, Deng X, Cheng Q, Cui X, Lin A, Luo P. Towards artificial intelligence‐assisted digital pathology: A systematic evaluation of multimodal generative artificial intelligence in clear cell renal cell carcinoma assessment. Interdisciplinary Medicine 2025;3(5) View
  12. . Generative AI in Medicine. New England Journal of Medicine 2025;393(9):933 View
  13. Kameda Y, Kaneko Y, Takada S. Re: Pushpanathan et al.: Can OpenAI’s New o1 Model Outperform Its Predecessors in Common Eye Care Queries?. Ophthalmology Science 2026;6(1):100968 View
  14. Lin A, Wang Z, Jiang A, Chen L, Qi C, Zhu L, Mou W, Gan W, Zeng D, Xiao M, Chu G, Peng S, Wong H, Zhang L, Zhang H, Deng X, Wang Y, Zhang J, Cheng Q, Tang B, Luo P. Large language models in clinical trials: applications, technical advances, and future directions. BMC Medicine 2025;23(1) View
  15. Valls Morató J. Impacto de la IAG en la labor del personal docente PhD-investigador. Estudio del caso de una escuela de negocios. European Public & Social Innovation Review 2025;11:1 View
  16. Ma Z, Liu Y, Zhang Z, Chen R, Fan H, Cao X, Ni L. Clinical applications of large language models in knee osteoarthritis: a systematic review. Frontiers in Medicine 2025;12 View
  17. Özdemir Ö, Kavan M, Güven Y. Evaluation of the readability, quality, and accuracy of AI chatbot responses to questions about deleterious oral habits. BMC Oral Health 2025;25(1) View

Conference Proceedings

  1. Saeedi P, Goodarzi M, Canbaz M. 2025 6th International Conference on Artificial Intelligence, Robotics and Control (AIRC). Heuristics and Biases in AI Decision-Making: Implications for Responsible AGI View