Pilot study using ‘dangerous answers’ as scoring technique on certifying examinations

Four medical specialty certifying boards collaborated in an exploratory project to determine whether “dangerous answers,” that is, incorrect answers that would be dangerous in patient care, could be identified in written board examinations and used as a potential technique for scoring. Panels of experts in each specialty retrospectively identified in previously administered certifying examinations those multiple-choice questions that contained dangerous incorrect options. The 2,713 certification candidates who took these examinations from 1981 through 1983 were given dangerous answer scores based on their responses to these items. When these dangerous answer scores were compared with their conventional percent-correct scores, it was found that 3 percent of the candidates who failed the examinations by conventional scoring had fewer than the average dangerous incorrect answers. More important, 10 percent of the candidates who passed the examinations by conventional scoring chose a greater than average number of dangerous answers. The implications of these preliminary findings and the limitations of the study are discussed.