Right Wing Nation

Read single article or Open it here… »

First, I tackled the reading proficiency scores for the entire state. Here are the descriptive stats:

The mean increased from 98-99 to 04-05 by 16.5%. The standard deviation — the amount of variance, which we can roughly define as the average amount each school differed in one direction or another from the mean — decreased. The lower the standard deviation, the less "spread out" the data. This suggests that in the 04-05 school year, more schools clustered around the mean.

The kurtoses support this. Kurtosis is the "peakedness" of the data distribution. Visualize a bell curve as a water balloon (sorry no, I am not going to do graphics; you’ll have to use your imagination). Now, if you place your hand on the top and squish it downward, more of it will squish into the tails at either side. This would be a "flattened" curve, with a low kurtosis. Now, instead of squishing the top down, visualize taking either hand and pushing it in from the sides. The water (data) would push the middle of the curve upward, or make it more "peaked." This would be a curve with a higher kurtosis. So the higher the kurtosis, the more data is squished up into the middle, where the mean is. Now note that the kurtosis for 04-05 is higher than that for 98-99. This supports what the standard deviations tell us, that in 04-05, more data were clustered around the mean (and in 98-99, more data were squished into the tails).

So far, everything looks positive, until we look at the skewness. Go back to that bell curve water balloon. If you grab the tail on the right and pull it further to the right, more of the water will spill into that tail, right? That’s what we call a right-skewed curve, and it has a positive skewness factor. If you pull the left tail out, more water spills into that left tail, and we have a left-skewed curve, with a negative skewness factor. When we look at the skewness for the two years, both are left-skewed — that is, in both years, there are more data in the left tails (lower end of proficient) — but the 04-05 curve is more left-skewed than 98-99 (-1.86 and -0.92, respectively). So even though it does look like Wisconsin may have improved the reading proficiency between the two years, they also slightly increased those who were at the low end of proficient (if this seems like a paradox to you, think of the water balloon again, and all will be made clear).

However, we cannot say whether these differences are meaningful (statistically significant) or whether they are due to random variation from looking at the descriptive statistics alone. So we ran ANOVA to a probability of 95% (alpha=0.05) to test the null hypothesis, that the differences are due to random variation:

The value of p is extremely small (2.3E-144), far smaller than 0.05, so to a 95% probability, we have disproved the null hypothesis. In other words, we are 95% certain that the difference between the percent proficient and above in the two years is not due to random variation. Wisconsin can from these data validly claim that they raised their proficiency.

Now, let’s turn to Madison. First, the descriptive stats:

We certainly see a difference between the two years in the means, a larger difference than we saw for the whole state, though keep in mind that while we had over a thousand reporting schools for the state, we have 26 reporting schools for Madison, and recall that the more data we have, the more reliable our statistics are. So don’t jump immediately to conclusions. The kurtosis in 04-05 is flatter than that in 98-99, indicating that there are more data in the tails in 04-05; the skewness in both years is roughly identical.

Again, we have to run ANOVA to test the null hypothesis, that the difference is due to random variation:

And again, ANOVA disproves the null hypothesis. The value of p is 1.05E-08, so to a 95% level of probability, the difference is not due to random variation (alpha=0.05). Madison can from these data claim that they raised their proficiency levels.

However, the real issue here is the group of Reading First school systems in Madison (Glendale, Hawthorne, Lincoln, and Orchard Ridge). But before I go on, let me mention a crucial statistical issue we’re about to encounter, one I listed above and said would become an issue later on.

For the entire state of Wisconsin, we have over 1,000 schools reporting proficiency levels. For Madison, we have 26 reportings (this is why the p-value for the Madison ANOVA is larger than the p-value for the whole state, even though Madison reports a larger difference in proficiency than the state). A data set of twenty-six data points isn’t statistically ideal, but we can work with it.

At issue here, however, are the Reading Firsts in Madison, and there are only four. We could run ANOVA, but with only four data points the results wouldn’t be reliable. All we can do is calculate the descriptive statistics and interpret them cautiously:

Did the percentage of students proficient or higher increase? Yes (50.64% to 66.5%). Again, though, we only have four data points here, so even the descriptive stats are questionable. The kurtosis suggests that there were more data around the mean than in the tails in 04-05 than there were in 98-99, though the standard deviations and ranges indicate the reverse, but again, there are only four data points here.

From these data, we cannot say that the Reading First schools did or did not increase their proficiencies. There just aren’t enough data. But — and this is crucial — neither can the Reading First schools claim that they raised their proficiencies using these data. The only way we can determine whether these school systems did or did not raise their proficiencies is by analyzing the raw data, and not the aggregates by school. In other words, Ken was right, and the journalist was wrong.

Remember that I said we are assuming that the state standards did not change, or if they did, that the scores in the two years we looked at are comparable? And remember how Wisconsin’s percentage of proficient or better went from 71.04% to 87.54% between the 98-99 and 04-05 school years? Ken states:

As the NAEP data clearly shows, the Wisconsin’s proficiency exam standards did change between 1998 and 2005. NAEP scores declined slightly, while the Wisconsin scores magically skyrocketed. Suspiciously so.

So, when RWP says that Wisconsin and Madison can validly claim that scores have increased, this conclusion only holds if Wisconsin’s proficiency exam standards didn’t change between 1998 and 2005. And, as we know from NAEP scores, they did.

Houston, we have a problem. You can write exams that are comparable to earlier exams. That’s why standardized test scores (SAT, GRE, GMAT, LSAT, etc.) are reliable from one administration to another. But if you change the standards, you can’t, because you change the definition of proficiency. Wisconsin has to explain the discrepancy between the NAEP data and their reported data, and if they did change standards, they must explain how, exactly, their scores from year to year are comparable, and how they can claim their reading proficiency increased.

Here endeth the lesson (add cute little smiley face here).

UPDATE: It looks like Wisconsin fudged the data.

« Close it