Evaluating the Systematic Validity of a Medical Subspecialty Examination

The purpose of this study is to determine if the construct of a medical subspecialty examination, as defined by the hierarchy of item difficulties, is stable across physicians who completed a fellowship and recertifiers as compared to non-fellows. Three comparisons of groups are made: 1) Practice pathway board candidates compared to members of all other boards taking the subspecialty examination, 2) Practice pathway board candidates who did not complete a fellowship compared to members of all other boards, and 3) Practice pathway board candidates who completed a fellowship compared to new candidates who had not completed a fellowship. All group comparisons showed significant positive correlations. As expected, the study did not find evidence of DIF between subgroups. However, non-fellowship examinees do score systematically lower than their fellowship taking counterparts. This suggests the value of a fellowship program. The study demonstrates the stability of the construct, therefore the reason behind the difference in passing rate lies elsewhere and should be examined.