Right Wing Nation

Before I begin, let me point out that today is the anniversary of the Nickel Mines Amish school shootings. My tribute is here. If you believe in God, please pray for all of those affected.

A digital moment of silence.

Even on vacation in Manhattan, I just can’t resist data. So when I saw EdWonk’s post about the most recent NCES report, of course I had to go download the data and do my own analysis. Trust, but verify, and all that.

In case you missed it, Margaret Spellings is all a-tither about the increase in math scores among 4th graders. Wait. That’s not exactly what she says:

“Student achievement is on the rise,” said Secretary Spellings. “No Child Left Behind is working. It’s doable, reasonable and necessary. Any efforts to weaken accountability would fly in the face of rising achievement.”

Spellings noted that 48 states and the District of Columbia either improved academically or held steady in all categories.

Ah. Improved or held steady — that is, didn’t improve, but didn’t get worse. I’m not sure I’d crow about that, but perhaps that’s just me. At any rate, I went to the NCES site and downloaded the 4th grade math assessment score data for 2003, 2005, and 2007. Unfortunately, there data (as I’ve bemoaned before) is aggregated, and these are state mean scores, but you use what you can get. Here are the descriptive stats:

Yes, the mean scores show an increase over the three years sampled, but not much. Look closer, and there’s a great big waving red flag. See the mean for 2003, 237.05? Now, look at the 95% Confidence Interval for 2007, and note that the 2005 mean falls within that interval.

A confidence interval is analogous to a margin of error. Since these data are aggregated by state, we calculate the interval to extrapolate the true means. So from the data we have, there is a 95% probability that the true mean 4th grade math assessment score for 2003 falls between 235.78 and 231.97, and so forth. The fact that the 2005 mean falls within the confidence interval (margin of error) for the 2007 scores strongly suggests that there is no statistically significant difference between the 2005 and 2007 scores.

To check, we ran ANOVA:

In order for there to be a statistically significant difference, F must be larger than the F crit, and the last time I checked my calculator, 2.46 was not larger than 3.94. Indeed, the value of P is greater than 0.05, so there is no statistically significant difference between the 2005 and 2007 scores. Interestingly, on the NCES site, once you have constructed your data table, you can click a little button that says something like, "Are they statistically significant?" and curious, I did that, and selected variables. All I got was a set of grids with grey boxes, and I have no idea what that was supposed to mean. Is the NCES claiming statistical significance? Well, I could read the report, but 1) you know how I feel about PDF files, and 2) I’m in Manhattan and have better things to do with my time. So no, I didn’t read the report, and yes, I probably should, but later, after I’m back home.

It may very well be that if we analyzed the data aggregated by school district rather than state, there would be a statistically significant difference between the two years’ scores (that is, aggregating the data by state may mask the significance). But that I have no way to test that, since the NCES does not offer school district as an aggregate. Still, I call shennanigans on the NCES, and Margaret Spellings’s statement should have been that, "48 states and the District of Columbia held steady in all categories," again, not something I’d brag about.

Okay, that’s enough. I’m in Manhattan. Time to go find lunch.