Beyond the Clinic Family Medicine on a Mission Part 1: How Air Force Physicians Achieve Humanitarian Goals Read Family Medicine on a Mission Part 1: How Air Force Physicians Achieve Humanitarian Goals
Phoenix Newsletter - March 2025 President’s Message: ABFM’s Unwavering Commitment to Diplomates and the Specialty Read President’s Message: ABFM’s Unwavering Commitment to Diplomates and the Specialty
Home Research Research Library Performance Evaluation of the Generative Pre-trained Transformer (GPT-4) on the Family Medicine In-Training Examination Performance Evaluation of the Generative Pre-trained Transformer (GPT-4) on the Family Medicine In-Training Examination 2024 Author(s) Wang, Ting, Mainous, Arch G III, Stelter, Keith L, O’Neill, Thomas R, and Newton, Warren P Topic(s) Education & Training Keyword(s) In-Training Examination Volume Journal of the American Board of Family Medicine Source Journal of the American Board of Family Medicine Objective: In this study, we sought to comprehensively evaluate GPT-4 (Generative Pre-trained Transformer)’s performance on the 2022 American Board of Family Medicine’s (ABFM) In-Training Examination (ITE), compared with its predecessor, GPT-3.5, and the national family residents’ performance on the same examination. Methods: We utilized both quantitative and qualitative analyses. First, a quantitative analysis was employed to evaluate the model’s performance metrics using zero-shot prompt (where only examination questions were provided without any additional information). After this, qualitative analysis was executed to understand the nature of the model’s responses, the depth of its medical knowledge, and its ability to comprehend contextual or new information through chain-of-thoughts prompts (interactive conversation) with the model. Results: This study demonstrated that GPT-4 made significant improvement in accuracy compared with GPT-3.5 over a 4-month interval between their respective release dates. The correct percentage with zero-shot prompt increased from 56% to 84%, which translates to a scaled score growth from 280 to 690, a 410-point increase. Most notably, further chain-of-thought investigation revealed GPT-4’s ability to integrate new information and make self-correction when needed. Conclusions: In this study, GPT-4 has demonstrated notably high accuracy, as well as rapid reading and learning capabilities. These results are consistent with previous research indicating GPT-4’s significant potential to assist in clinical decision making. Furthermore, the study highlights the essential role of physicians’ critical thinking and lifelong learning skills, particularly evident through the analysis of GPT-4’s incorrect responses. This emphasizes the indispensable human element in effectively implementing and using AI technologies in medical settings. Read More ABFM Research Read all 1993 Performances of family practice diplomates on successive mandatory recertification examinations Go to Performances of family practice diplomates on successive mandatory recertification examinations 2014 Aligning Expansion of Graduate Medical Education with Recent Recommendations for Reform. Go to Aligning Expansion of Graduate Medical Education with Recent Recommendations for Reform. 1995 Educational resource sharing and collaborative training in family practice and internal medicine. A statement from the American Boards of Internal Medicine and Family Practice Go to Educational resource sharing and collaborative training in family practice and internal medicine. A statement from the American Boards of Internal Medicine and Family Practice 2021 Attitudes and Perceptions of Research Among US Family Medicine Department Chairs Go to Attitudes and Perceptions of Research Among US Family Medicine Department Chairs
Author(s) Wang, Ting, Mainous, Arch G III, Stelter, Keith L, O’Neill, Thomas R, and Newton, Warren P Topic(s) Education & Training Keyword(s) In-Training Examination Volume Journal of the American Board of Family Medicine Source Journal of the American Board of Family Medicine
ABFM Research Read all 1993 Performances of family practice diplomates on successive mandatory recertification examinations Go to Performances of family practice diplomates on successive mandatory recertification examinations 2014 Aligning Expansion of Graduate Medical Education with Recent Recommendations for Reform. Go to Aligning Expansion of Graduate Medical Education with Recent Recommendations for Reform. 1995 Educational resource sharing and collaborative training in family practice and internal medicine. A statement from the American Boards of Internal Medicine and Family Practice Go to Educational resource sharing and collaborative training in family practice and internal medicine. A statement from the American Boards of Internal Medicine and Family Practice 2021 Attitudes and Perceptions of Research Among US Family Medicine Department Chairs Go to Attitudes and Perceptions of Research Among US Family Medicine Department Chairs
1993 Performances of family practice diplomates on successive mandatory recertification examinations Go to Performances of family practice diplomates on successive mandatory recertification examinations
2014 Aligning Expansion of Graduate Medical Education with Recent Recommendations for Reform. Go to Aligning Expansion of Graduate Medical Education with Recent Recommendations for Reform.
1995 Educational resource sharing and collaborative training in family practice and internal medicine. A statement from the American Boards of Internal Medicine and Family Practice Go to Educational resource sharing and collaborative training in family practice and internal medicine. A statement from the American Boards of Internal Medicine and Family Practice
2021 Attitudes and Perceptions of Research Among US Family Medicine Department Chairs Go to Attitudes and Perceptions of Research Among US Family Medicine Department Chairs