The American Board of Family Medicine’s 8 Years of Experience with Differential Item Functioning

INTRODUCTION: Differential item functioning (DIF) procedures flag examination questions in which examinees from different subpopulations who are of equal ability do not have the same probability of answering it correctly. Few medical certification boards employ DIF procedures because they do not collect the needed data on the examinee’s race or ethnicity. This article summarizes the American Board of Family Medicine’s (ABFM) combined use of DIF procedures and an expert panel to review certification questions for bias. METHODS: ABFM certification examination data from 2013 to 2020 were analyzed using a DIF procedure to flag questions with possible ethnic or racial bias. The flagged questions were reviewed by a racially and ethnically diverse panel of content experts. If the panel judged the source of the DIF was not clinically relevant for the practice of family medicine, the question was removed from the examination. RESULTS: Out of the 3487 questions analyzed, 374 unique questions (11%) were flagged by DIF procedures as potentially biased. Of the flagged questions, the review panel felt 4 should be removed for fairness. DISCUSSION: Using DIF procedures and panel review can improve the quality of the board certification questions and demonstrate the organization’s commitment to avoid racial or ethnic bias.