A Lesson from Oklahoma

Earlier this fall, the Oklahoma Council for Public Affairs released the results from the survey it had commissioned about the civics knowledge of high school students, which were very alarming. Only 23% identified George Washington as the first president, 29% identified the president as being in charge of the executive branch, and 43% that Democrats and Republicans being the two major parties in America (10% identified the two parties as Communist and Republican). OCPA decried the results as proof of the failure of Oklahoma's educational system, and the results of their survey were used in many major publications.

But, almost as soon as they were released, questions began to be raised. Could it really be that only 2.8% of the 1000 polled high school students, as the survey claimed, could pass the test (which is a meager 6 out of 10 correct)? And none would get 8-10 correct? In a random distribution that would bring about 600 college-bound students and 50 gifted students, none got 8-10 correct?

This raised some questions, most notably by statistician Nate Silver. Even starting with the assumption that only 23% of the sampled students knew about Washington, the results still looked fabricated. Simply put, the distribution of student scores matches almost identically to a homogeneous distribution of probability. However, students are not homogeneous... a student that gets the first three right is much more likely to get #4 right than one who got the first three wrong.

Silver wasn't the only one, as he mentioned today. State representative Ed Cannaday, a former educator, also thought something was fishy. He conducted the same survey in school districts in his own congressional district within Oklahoma (N = 325). And, he found an entirely different set of results. In fact, 98% identified George Washington as the first president, 85% identified the president as being in charge of the executive branch, and 95% identified the correct two political parties. In fact, the average score in Cannaday's survey was 7.8 correct out of 10, very striking considering the OPCA survey said none in 1000 scored more than 7 correct.

POLITICS INVOLVED
While Silver doesn't focus specifically on it, still what is clear is the subtext of OCPA being a conservative group pushing educational policy changes. And, by the reach of the survey's results, which landed prominent places in Time, Newsweek, and the USA Today, it can be said the survey was successful, however dubious. Now that headlines have blared how Oklahoma's public schools are failing miserably, the damage is done, and a page 12B follow-up article will not do anything about it.

The lesson here is twofold. One, that in an era of data-driven decision making, standardized assessment results still run secondary to sensationalized opinion polls in the effort to sway public opinion. And two, that schools appear to be not off-grounds for fabricated politicization. Which means one thing...

If you are an Iowan representative, you better be extra critical of any external data used to describe the achievement of Iowa's students.

And if you are a newspaper, like let's say a Register in the state capital, you should be equally cautious.

Using standardized data as a measuring stick for how well the nation's schools are doing is troubling enough for many reasons. It becomes infinitely worse when using non-standardized survey data.

There are a million reasons behind the data, all the way down to whether the students had any breakfast this morning, to whether the test had cultural bias within it, to whether a teacher happened to give the students the answers. Now we have to add nefarious purposes of the testing provider to the list.