Stereotype threat: the statistical debate

Stricker and Ward (2004) suggested that the effect of stereotype threat was overstated; their experiment showed no significant difference in performance for blacks or women who were asked demographic questions before or after standardized testing. In June's edition of the Journal of Applied Social Psychology, Danaher and Crandall (2008) propose that a different statistical analysis of Stricker and Ward's data does support the effect of stereotype threat. Specifically, Danaher and Crandall say that Stricker and Ward's criteria were too conservative: "favoring the acceptance of Type II error over Type I error ... decreases the probability of mistakenly 'finding' a null effect, and it increases the probability of missing a real effect".

In the same issue of the journal, Stricker and Ward (2008) reply that their analysis was appropriate. Danaher and Crandall, the authors claim, are using flawed extrapolations in their estimates, because AP test scores do not have a normal distribution. Stricker and Ward stand by their 2004 assertion that "inquiring about gender and ethnicity did not affect test performance of women or Black students."

---

There are two things I wanted to point out with today's post. First is that the vast majority of journalists, and a frightening portion of scientists, don't know jack about statistical analysis and reporting. Other people have covered this topic a lot better than I can. My statistics background is pretty damn weak as well.

The second thing was something I noticed with yesterday's post. I make an effort to report new findings in Difference Blog, and I wonder if that's a mistake. We don't get a chance to see the responses to the study when it's brand new. Part of assessing the validity of a particular study (at least for me) often involves seeing who references it, and whether the conclusions are supported by subsequent experiments, or whether the community tears apart the methods in some way I didn't think of.