Registered Replication Report: Johns, Schmader, & Martens (2005)

Stereotype threat refers to the fear of being judged based on negative stereotypes about the performance of a certain group one identifies with. Numerous published studies have found that stereotype threat might lower mathematics test performance among women. However, many studies used suboptimal designs and analyses; the literature might be subject to publication bias; and the cross-cultural generalizability of the effect remains unknown. This registered replication report describes the result of eight direct replications (total N = 1502) of a representative stereotype-threat study by Johns et al. (2005), who found that threatened women (but not men) underperform when they are confronted with a mathematics test that is presented to measure gender differences, and that this effect can be alleviated by altering test instructions. A multilevel analysis showed no significant interaction effect between stereotype-threat condition and gender, and only an observed gender gap on the mathematics test, with women performing worse than men (d = -0.37; 95%CI[-0.49; -0.25]). The stereotype-threat effect among women was virtually null and not significant (N = 822, d = -0.006; 95% CI[-0.15; 0.14]), and considerably smaller than the original study (N = 45, d = -0.82; 95 %CI[-1.45; -0.18]). Secondary analyses yielded a main effect of gender composition and a random effect of lab, but neither variable affected the interaction between stereotype threat and gender. The current results fail to replicate the stereotype-threat effect by Johns et al. (2005), hence casting doubt on the generalizability of the effect of stereotype threat on women’s mathematics performance.